10:38 a.m.
Pierrick Bouvier <pierrick.bouvier(a)linaro.org> writes:
Note: This RFC was posted to trigger a discussion around this topic,
and it's
not expected to merge it as it is.
Context
=======
Linaro is working towards heterogeneous emulation, mixing several architectures
in a single QEMU process. The first prerequisite is to be able to build such a
binary, which we commonly name "single-binary" in our various series.
An (incomplete) list of series is available here:
https://patchew.org/search?q=project%3AQEMU+single-binary
We don't expect to change existing command line interface or any observable
behaviour, it should be identical to existing binaries. If anyone notices a
difference, it will be a bug.
Define "notice a difference" :) More on that below.
The first objective we target is to combine qemu-system-arm and
qemu-system-aarch64 in a single binary, showing that we can build and link such
a thing. While being useless from a feature point of view, it allows us to make
good progress towards the goal, and unify two "distinct" architectures, and
gain
experience on problems met.
Makes sense to me.
Our current approach is to remove compilation units duplication to be
able to
link all object files together. One of the concerned subsystem is QAPI.
QAPI
====
QAPI generated files contain conditional clauses to define various structures,
enums, and commands only for specific targets. This forces files to be
compiled for every target.
To be precise: conditionals that use macros restricted to
target-specific code, i.e. the ones poisoned by exec/poison.h. Let's
call them target-specific QAPI conditionals.
The QAPI generator is blissfully unaware of all this.
The build system treats QAPI modules qapi/*-target.json as
target-specific. The .c files generated for them are compiled per
target. See qapi/meson.build.
Only such target-specific modules can can use target-specific QAPI
conditionals. Use in target-independent modules will generate C that
won't compile.
Poisoned macros used in qapi/*-target.json:
CONFIG_KVM
TARGET_ARM
TARGET_I386
TARGET_LOONGARCH64
TARGET_MIPS
TARGET_PPC
TARGET_RISCV
TARGET_S390X
What we try to do here is to build them
only once
instead.
You're trying to eliminate target-specific QAPI conditionals. Correct?
In the past, we identied that the best approach to solve this is to
expose code
for all targets (thus removing all #if clauses), and stub missing
symbols for concerned targets.
This affects QAPI/QMP introspection, i.e. the value of query-qmp-schema.
Management applications can no longer use introspection to find out
whether target-specific things are available.
For instance, query-cpu-definitions is implemented for targets arm,
i386, loongarch, mips, ppc, riscv, and s390x. It initially was for
fewer targets, and more targets followed one by one. Still more may
follow in the future. Right now, management applications can use
introspection to find out whether it is available. That stops working
when you make it available for all targets, stubbed out for the ones
that don't (yet) implement it.
Management applications may have to be adjusted for this.
This is not an attempt to shoot down your approach. I'm merely
demonstrating limitations of your promise "if anyone notices a
difference, it will be a bug."
Now, we could get really fancy and try to keep introspection the same by
applying conditionals dynamically somehow. I.e. have the single binary
return different introspection values depending on the actual guest's
target.
This requires fixing the target before introspection. Unless this is
somehow completely transparent (wrapper scripts, or awful hacks based on
the binary's filename, perhaps), management applications may have to be
adjusted to actually do that.
Applies not just to introspection. Consider query-cpu-definitions
again. It currently returns CPU definitions for *the* target. What
would a single binary's query-cpu-definitions return? The CPU
definitions for *all* its targets? Management applications then receive
CPUs that won't work, which may upset them. To avoid noticable
difference, we again have to fix the target before we look.
Of course, "fixing the target" stops making sense once we move to
heterogeneous machines with multiple targets.
This series build QAPI generated code once, by removing all
TARGET_{arch} and
CONFIG_KVM clauses. What it does *not* at the moment is:
- prevent target specific commands to be visible for all targets
(see TODO comment on patch 2 explaining how to address this)
- nothing was done to hide all this from generated documentation
For better or worse, generated documentation always contains everything.
An argument could be made for stripping out documentation for the stuff
that isn't included in this build.
From what I understood, the only thing that matters is to limit qmp
commands
visible. Exposing enums, structure, or events is not a problem, since they
won't be used/triggered for non concerned targets. Please correct me if this is
wrong, and if there are unexpected consequences for libvirt or other consumers.
I'm not sure what you mean by "to limit qmp commands visible".
QAPI/QMP introspection has all commands and events, and all types
reachable from them. query-qmp-schema returns an array, where each
array element describes one command, event, or type. When a command,
event, or type is conditional in the schema, the element is wrapped in
the #if generated for the condition.
Impact on code size
===================
There is a strong focus on keeping QEMU fast and small. Concerning performance,
there is no impact, as the only thing that would change is to conditionally
check current target to register some commands.
Concerning code size, you can find the impact on various qemu-system binaries
with optimized and stripped build.
upstream:
12588 ./build/qemu-system-s390x
83992 ./build/qemu-system-x86_64
31884 ./build/qemu-system-aarch64
upstream + this series:
12644 ./build/qemu-system-s390x (+56kB, +0.004%)
84076 ./build/qemu-system-x86_64 (+84kB, +0.001%)
31944 ./build/qemu-system-aarch64 (+60kB, +0.001%)
Feedback
========
The goal of this series is to be spark a conversation around following topics:
- Would you be open to such an approach? (expose all code, and restrict commands
registered at runtime only for specific targets)
Yes, if we can find acceptable solutions for the problems that come with
it.
- Are there unexpected consequences for libvirt or other consumers to
expose
more definitions than what we have now?
Maybe.
- Would you recommend another approach instead? I experimented with
having per
target generated files, but we still need to expose quite a lot in headers, so
my opinion is that it's much more complicated for zero benefit. As well, the
code size impact is more than negligible, so the simpler, the better.
Feel free to add anyone I could have missed in CC.
I'm throwing in devel(a)lists.libvirt.org.
Regards,
Pierrick
Pierrick Bouvier (3):
qapi: add weak stubs for target specific commands
qapi: always expose TARGET_* or CONFIG_KVM code
qapi: make all generated files common
qapi/commands-weak-stubs.c | 38 ++++++++++++++++++++++++++++++++++++++
qapi/meson.build | 5 ++++-
scripts/qapi/commands.py | 4 ++++
scripts/qapi/common.py | 4 +++-
4 files changed, 49 insertions(+), 2 deletions(-)
create mode 100644 qapi/commands-weak-stubs.c
4:07 p.m.
New subject: [RFC PATCH 0/3] single-binary: make QAPI generated files common
On 4/25/25 08:38, Markus Armbruster wrote:
Pierrick Bouvier <pierrick.bouvier(a)linaro.org> writes:
> Note: This RFC was posted to trigger a discussion around this topic, and it's
> not expected to merge it as it is.
>
> Context
> =======
>
> Linaro is working towards heterogeneous emulation, mixing several architectures
> in a single QEMU process. The first prerequisite is to be able to build such a
> binary, which we commonly name "single-binary" in our various series.
> An (incomplete) list of series is available here:
> https://patchew.org/search?q=project%3AQEMU+single-binary
>
> We don't expect to change existing command line interface or any observable
> behaviour, it should be identical to existing binaries. If anyone notices a
> difference, it will be a bug.
Define "notice a difference" :) More on that below.
Given a single-binary *named* exactly like an existing qemu-system-X
binary, any user or QEMU management layer should not be able to
distinguish it from the real qemu-system-X binary.
The new potential things will be:
- introduction of an (optional) -target option, which allows to
override/disambiguate default target detected.
- potentially more boards/cpus/devices visible, once we start developing
heterogeneous emulation. See it as a new CONFIG_{new_board} present.
Out of that, once the current target is identified, based on argv[0],
there should be absolutely no difference, whether in the behaviour, UI,
command line, or the monitor interfaces.
Maybe (i.e. probably) one day people will be interested to create a new
shiny command line for heteregenous scenarios, but for now, this is
*not* the goal we pursue. We just want to be able to manually define a
board mixing two different cpu architectures, without reinventing all
the wheels coming with that. Once everything is ready (and not before),
it will be a good time to revisit the command line interface to reflect
this. Definitely a small task compared to all we have left to do now.
Finally, even if we decide to do those changes, I think they should be
reflected on both existing binaries and the new single-binary. It would
be a mistake to create "yet another" way to use QEMU, just because we
have N architectures available instead of one.
> The first objective we target is to combine qemu-system-arm and
> qemu-system-aarch64 in a single binary, showing that we can build and link such
> a thing. While being useless from a feature point of view, it allows us to make
> good progress towards the goal, and unify two "distinct" architectures, and
gain
> experience on problems met.
Makes sense to me.
> Our current approach is to remove compilation units duplication to be able to
> link all object files together. One of the concerned subsystem is QAPI.
>
> QAPI
> ====
>
> QAPI generated files contain conditional clauses to define various structures,
> enums, and commands only for specific targets. This forces files to be
> compiled for every target.
To be precise: conditionals that use macros restricted to
target-specific code, i.e. the ones poisoned by exec/poison.h. Let's
call them target-specific QAPI conditionals.
The QAPI generator is blissfully unaware of all this.
Indeed, the only thing QAPI generaor is aware of is that it's a compile
time definition, since it implements those with #if, compared to a
runtime check.
The build system treats QAPI modules qapi/*-target.json as
target-specific. The .c files generated for them are compiled per
target. See qapi/meson.build.
Only such target-specific modules can can use target-specific QAPI
conditionals. Use in target-independent modules will generate C that
won't compile.
Poisoned macros used in qapi/*-target.json:
CONFIG_KVM
TARGET_ARM
TARGET_I386
TARGET_LOONGARCH64
TARGET_MIPS
TARGET_PPC
TARGET_RISCV
TARGET_S390X
> What we try to do here is to build them only once
> instead.
You're trying to eliminate target-specific QAPI conditionals. Correct?
Yes, but without impacting the list of commands exposed. Thus, it would
be needed to select at runtime to expose/register commands.
> In the past, we identied that the best approach to solve this is
to expose code
> for all targets (thus removing all #if clauses), and stub missing
> symbols for concerned targets.
This affects QAPI/QMP introspection, i.e. the value of query-qmp-schema.
Management applications can no longer use introspection to find out
whether target-specific things are available.
As asked on my previous email answering Daniel, would that be possible
to build the schema dynamically, so we can decide what to expose or not
introspection wise?
For instance, query-cpu-definitions is implemented for targets arm,
i386, loongarch, mips, ppc, riscv, and s390x. It initially was for
fewer targets, and more targets followed one by one. Still more may
follow in the future. Right now, management applications can use
introspection to find out whether it is available. That stops working
when you make it available for all targets, stubbed out for the ones
that don't (yet) implement it.
I will repeat, just to be clear, I don't think exposing all commands is
a good idea.
The current series *does not* do this, simply because I didn't want to
huge work for nothing.
Management applications may have to be adjusted for this.
This is not an attempt to shoot down your approach. I'm merely
demonstrating limitations of your promise "if anyone notices a
difference, it will be a bug."
I stick to this promise :).
Now, we could get really fancy and try to keep introspection the same
by
applying conditionals dynamically somehow. I.e. have the single binary
return different introspection values depending on the actual guest's
target.
This requires fixing the target before introspection. Unless this is
somehow completely transparent (wrapper scripts, or awful hacks based on
the binary's filename, perhaps), management applications may have to be
adjusted to actually do that.
Applies not just to introspection. Consider query-cpu-definitions
again. It currently returns CPU definitions for *the* target. What
would a single binary's query-cpu-definitions return? The CPU
definitions for *all* its targets? Management applications then receive
CPUs that won't work, which may upset them. To avoid noticable
difference, we again have to fix the target before we look.
Of course, "fixing the target" stops making sense once we move to
heterogeneous machines with multiple targets.
At this point, I don't have think about what should be the semantic when
we'll have multiple targets running simultaneously (expose the union,
restrict to the main arch, choose a third way).
> This series build QAPI generated code once, by removing all
TARGET_{arch} and
> CONFIG_KVM clauses. What it does *not* at the moment is:
> - prevent target specific commands to be visible for all targets
> (see TODO comment on patch 2 explaining how to address this)
> - nothing was done to hide all this from generated documentation
For better or worse, generated documentation always contains everything.
Fine for me, it makes sense, as the official documentation published,
which is what people will consume primarily, is for all targets.
An argument could be made for stripping out documentation for the
stuff
that isn't included in this build.
> From what I understood, the only thing that matters is to limit qmp commands
> visible. Exposing enums, structure, or events is not a problem, since they
> won't be used/triggered for non concerned targets. Please correct me if this is
> wrong, and if there are unexpected consequences for libvirt or other consumers.
I'm not sure what you mean by "to limit qmp commands visible".
QAPI/QMP introspection has all commands and events, and all types
reachable from them. query-qmp-schema returns an array, where each
array element describes one command, event, or type. When a command,
event, or type is conditional in the schema, the element is wrapped in
the #if generated for the condition.
After reading and answering to your valuable email, I definitely think
the introspection schema we expose should be adapted, independently of
how we build QAPI code (i.e. using #ifdef TARGET or not).
Is it something technically hard to achieve?
>
> Impact on code size
> ===================
>
> There is a strong focus on keeping QEMU fast and small. Concerning performance,
> there is no impact, as the only thing that would change is to conditionally
> check current target to register some commands.
> Concerning code size, you can find the impact on various qemu-system binaries
> with optimized and stripped build.
>
> upstream:
> 12588 ./build/qemu-system-s390x
> 83992 ./build/qemu-system-x86_64
> 31884 ./build/qemu-system-aarch64
> upstream + this series:
> 12644 ./build/qemu-system-s390x (+56kB, +0.004%)
> 84076 ./build/qemu-system-x86_64 (+84kB, +0.001%)
> 31944 ./build/qemu-system-aarch64 (+60kB, +0.001%)
>
> Feedback
> ========
>
> The goal of this series is to be spark a conversation around following topics:
>
> - Would you be open to such an approach? (expose all code, and restrict commands
> registered at runtime only for specific targets)
Yes, if we can find acceptable solutions for the problems that come with
it.
> - Are there unexpected consequences for libvirt or other consumers to expose
> more definitions than what we have now?
Maybe.
> - Would you recommend another approach instead? I experimented with having per
> target generated files, but we still need to expose quite a lot in headers, so
> my opinion is that it's much more complicated for zero benefit. As well, the
> code size impact is more than negligible, so the simpler, the better.
>
> Feel free to add anyone I could have missed in CC.
I'm throwing in devel(a)lists.libvirt.org.
> Regards,
> Pierrick
>
> Pierrick Bouvier (3):
> qapi: add weak stubs for target specific commands
> qapi: always expose TARGET_* or CONFIG_KVM code
> qapi: make all generated files common
>
> qapi/commands-weak-stubs.c | 38 ++++++++++++++++++++++++++++++++++++++
> qapi/meson.build | 5 ++++-
> scripts/qapi/commands.py | 4 ++++
> scripts/qapi/common.py | 4 +++-
> 4 files changed, 49 insertions(+), 2 deletions(-)
> create mode 100644 qapi/commands-weak-stubs.c
Thanks,
Pierrick
4:13 p.m.
New subject: [RFC PATCH 0/3] single-binary: make QAPI generated files common
On 4/25/25 14:07, Pierrick Bouvier wrote:
> QAPI/QMP introspection has all commands and events, and all
types
> reachable from them. query-qmp-schema returns an array, where each
> array element describes one command, event, or type. When a command,
> event, or type is conditional in the schema, the element is wrapped in
> the #if generated for the condition.
>
After reading and answering to your valuable email, I definitely think
the introspection schema we expose should be adapted, independently of
how we build QAPI code (i.e. using #ifdef TARGET or not).
Is it something technically hard to achieve?
From existing json format, we could imagine to change semantic of "if"
field to mean "expose in the schema, and register associated commands",
instead "introduce ifdefs around this".
QAPI generator would not have to know about what is inside the ifs,
simply calling expression passed as value, and condition command
registering and schema exposition with that.
1:21 a.m.
New subject: [RFC PATCH 0/3] single-binary: make QAPI generated files common
Pierrick Bouvier <pierrick.bouvier(a)linaro.org> writes:
On 4/25/25 08:38, Markus Armbruster wrote:
> Pierrick Bouvier <pierrick.bouvier(a)linaro.org> writes:
>
>> Note: This RFC was posted to trigger a discussion around this topic, and
it's
>> not expected to merge it as it is.
>>
>> Context
>> =======
>>
>> Linaro is working towards heterogeneous emulation, mixing several architectures
>> in a single QEMU process. The first prerequisite is to be able to build such a
>> binary, which we commonly name "single-binary" in our various series.
>> An (incomplete) list of series is available here:
>> https://patchew.org/search?q=project%3AQEMU+single-binary
>>
>> We don't expect to change existing command line interface or any observable
>> behaviour, it should be identical to existing binaries. If anyone notices a
>> difference, it will be a bug.
>
> Define "notice a difference" :) More on that below.
>
Given a single-binary *named* exactly like an existing qemu-system-X
binary, any user or QEMU management layer should not be able to
distinguish it from the real qemu-system-X binary.
The new potential things will be:
- introduction of an (optional) -target option, which allows to
override/disambiguate default target detected.
- potentially more boards/cpus/devices visible, once we start developing
heterogeneous emulation. See it as a new CONFIG_{new_board} present.
Out of that, once the current target is identified, based on argv[0],
there should be absolutely no difference, whether in the behaviour, UI,
command line, or the monitor interfaces.
Letting argv[0] affect behavior is problematic. See prior discussion:
Subject: Re: [RFC PATCH 04/18] qemu: Introduce 'qemu/legacy_binary_info.h'
Message-ID: <87bjttzh3b.fsf(a)pond.sub.org>
https://lore.kernel.org/qemu-devel/87bjttzh3b.fsf@pond.sub.org/
Maybe (i.e. probably) one day people will be interested to create a
new
shiny command line for heteregenous scenarios, but for now, this is
*not* the goal we pursue. We just want to be able to manually define a
board mixing two different cpu architectures, without reinventing all
the wheels coming with that. Once everything is ready (and not before),
it will be a good time to revisit the command line interface to reflect
this. Definitely a small task compared to all we have left to do now.
Finally, even if we decide to do those changes, I think they should be
reflected on both existing binaries and the new single-binary. It would
be a mistake to create "yet another" way to use QEMU, just because we
have N architectures available instead of one.
>> The first objective we target is to combine qemu-system-arm and
>> qemu-system-aarch64 in a single binary, showing that we can build and link such
>> a thing. While being useless from a feature point of view, it allows us to make
>> good progress towards the goal, and unify two "distinct" architectures,
and gain
>> experience on problems met.
>
> Makes sense to me.
>
>> Our current approach is to remove compilation units duplication to be able to
>> link all object files together. One of the concerned subsystem is QAPI.
>>
>> QAPI
>> ====
>>
>> QAPI generated files contain conditional clauses to define various structures,
>> enums, and commands only for specific targets. This forces files to be
>> compiled for every target.
>
> To be precise: conditionals that use macros restricted to
> target-specific code, i.e. the ones poisoned by exec/poison.h. Let's
> call them target-specific QAPI conditionals.
>
> The QAPI generator is blissfully unaware of all this.
>
Indeed, the only thing QAPI generaor is aware of is that it's a compile
time definition, since it implements those with #if, compared to a
runtime check.
QAPI conditionals are designed to provide build-time configuration. In
C, we use #if for that. QAPI conditionals are merely a slight
abstraction of C #if. Used to be even slighter until merge commit
c83fcfaf8a5.
Two kinds of #if conditions in QEMU C code:
1. True build-time configuration, i.e. control what's being built based
on what the host can do and what the user wants, as figured out by
configure. These conditions are the same for the entire build.
2. Specializing target-specific code. These conditions vary per target.
C code using them needs to be compiled per target.
To get a single binary, we want to reduce use of the second kind as much
as practical. This can involve replacing compile-time conditionals by
run-time checks. Remaining target-specific code needs to adjusted so
target-specific code for multiple targets can coexist in the single
binary.
Trouble is some uses of the second kind are in QAPI conditionals. I can
see three options:
(1) Drop these conditionals.
(2) Replace them by run-time checks.
(3) Have target-specific QAPI-generated code for multiple targets
coexist in the single binary.
As far as I can tell, your RFC series is an incomplete attempt at (2).
I gather you considered (3), but you dislike it for its bloat and
possibly other reasons. I sympathize; the QAPI-generated code is plenty
bloated as it is, in good part to early design decisions (not mine).
Your "no noticeable differences" goal precludes (1).
Back to (2). In C, replacing compile-time conditionals by run-time
checks means replacing #if FOO by if (foo). Such a transformation isn't
possible in the QAPI schema. To make it possible, we need to evolve the
QAPI schema language.
docs/devel/qapi-code-gen.rst describes what we have:
COND = STRING
| { 'all: [ COND, ... ] }
| { 'any: [ COND, ... ] }
| { 'not': COND }
[....]
The C code generated for the definition will then be guarded by an #if
preprocessing directive with an operand generated from that condition:
* STRING will generate defined(STRING)
* { 'all': [COND, ...] } will generate (COND && ...)
* { 'any': [COND, ...] } will generate (COND || ...)
* { 'not': COND } will generate !COND
So, conditions are expression trees where the leaves are preprocessor
symbols and the inner nodes are operators.
It's not quite obvious to me how to best evolve this to support run-time
checks.
Whatever we choose should support generating Rust and Go as well. Why?
Rust usage in QEMU is growing, and we'll likely need to generate some
Rust from the QAPI schema. Victor Toso has been working on Go bindings
for use in Go QMP client software.
> The build system treats QAPI modules qapi/*-target.json as
> target-specific. The .c files generated for them are compiled per
> target. See qapi/meson.build.
>
> Only such target-specific modules can can use target-specific QAPI
> conditionals. Use in target-independent modules will generate C that
> won't compile.
>
> Poisoned macros used in qapi/*-target.json:
>
> CONFIG_KVM
> TARGET_ARM
> TARGET_I386
> TARGET_LOONGARCH64
> TARGET_MIPS
> TARGET_PPC
> TARGET_RISCV
> TARGET_S390X
>
>> What we try to do here is to build them only once
>> instead.
>
> You're trying to eliminate target-specific QAPI conditionals. Correct?
>
Yes, but without impacting the list of commands exposed. Thus, it would
be needed to select at runtime to expose/register commands.
Conditionals affect more than just commands.
>> In the past, we identied that the best approach to solve this
is to expose code
>> for all targets (thus removing all #if clauses), and stub missing
>> symbols for concerned targets.
>
> This affects QAPI/QMP introspection, i.e. the value of query-qmp-schema.
>
> Management applications can no longer use introspection to find out
> whether target-specific things are available.
>
As asked on my previous email answering Daniel, would that be possible
to build the schema dynamically, so we can decide what to expose or not
introspection wise?
QAPI was designed to be compile-time static. Revising such fundamental
design assumptions is always fraught. I can't give you a confident
assessment now. All I can offer you is my willingness to explore
solutions. See "really fancy" below.
Fun fact: we used to generate the value of query-qmp-schema as a single
string. We switched to the current, more bloated representation to
support conditionals (commit 7d0f982bfbb).
> For instance, query-cpu-definitions is implemented for targets
arm,
> i386, loongarch, mips, ppc, riscv, and s390x. It initially was for
> fewer targets, and more targets followed one by one. Still more may
> follow in the future. Right now, management applications can use
> introspection to find out whether it is available. That stops working
> when you make it available for all targets, stubbed out for the ones
> that don't (yet) implement it.
>
I will repeat, just to be clear, I don't think exposing all commands is
a good idea.
The current series *does not* do this, simply because I didn't want to
huge work for nothing.
Got it.
> Management applications may have to be adjusted for this.
>
> This is not an attempt to shoot down your approach. I'm merely
> demonstrating limitations of your promise "if anyone notices a
> difference, it will be a bug."
>
I stick to this promise :).
> Now, we could get really fancy and try to keep introspection the same by
> applying conditionals dynamically somehow. I.e. have the single binary
> return different introspection values depending on the actual guest's
> target.
>
> This requires fixing the target before introspection. Unless this is
> somehow completely transparent (wrapper scripts, or awful hacks based on
> the binary's filename, perhaps), management applications may have to be
> adjusted to actually do that.
>
> Applies not just to introspection. Consider query-cpu-definitions
> again. It currently returns CPU definitions for *the* target. What
> would a single binary's query-cpu-definitions return? The CPU
> definitions for *all* its targets? Management applications then receive
> CPUs that won't work, which may upset them. To avoid noticable
> difference, we again have to fix the target before we look.
>
> Of course, "fixing the target" stops making sense once we move to
> heterogeneous machines with multiple targets.
>
At this point, I don't have think about what should be the semantic when
we'll have multiple targets running simultaneously (expose the union,
restrict to the main arch, choose a third way).
We have to unless we make query-cpu-definitions fail or impossible to
send while the target is still undecided.
Making it fail would violate the "no observable differences" goal.
The only path to true "no observable differences" I can see is to fix
the target before the management application interacts with QEMU in any
way. This would make QMP commands (query-cpu-definitions,
query-qmp-schema, ...) impossible to send before the target is fixed.
>> This series build QAPI generated code once, by removing all
TARGET_{arch} and
>> CONFIG_KVM clauses. What it does *not* at the moment is:
>> - prevent target specific commands to be visible for all targets
>> (see TODO comment on patch 2 explaining how to address this)
>> - nothing was done to hide all this from generated documentation
>
> For better or worse, generated documentation always contains everything.
>
Fine for me, it makes sense, as the official documentation published,
which is what people will consume primarily, is for all targets.
> An argument could be made for stripping out documentation for the stuff
> that isn't included in this build.
>
>> From what I understood, the only thing that matters is to limit qmp commands
>> visible. Exposing enums, structure, or events is not a problem, since they
>> won't be used/triggered for non concerned targets. Please correct me if this
is
>> wrong, and if there are unexpected consequences for libvirt or other consumers.
>
> I'm not sure what you mean by "to limit qmp commands visible".
>
> QAPI/QMP introspection has all commands and events, and all types
> reachable from them. query-qmp-schema returns an array, where each
> array element describes one command, event, or type. When a command,
> event, or type is conditional in the schema, the element is wrapped in
> the #if generated for the condition.
>
After reading and answering to your valuable email, I definitely think
Thanks!
the introspection schema we expose should be adapted, independently
of
how we build QAPI code (i.e. using #ifdef TARGET or not).
Is it something technically hard to achieve?
Unclear. See "fundamental design assumptions" and "need to evolve the
QAPI schema language" above.
If you want to learn more about introspection, I'd recommend
docs/devel/qapi-code-gen.rst section "Client JSON Protocol
introspection".
[...]
5:25 a.m.
New subject: [RFC PATCH 0/3] single-binary: make QAPI generated files common
On Fri, Apr 25, 2025 at 14:07:34 -0700, Pierrick Bouvier wrote:
On 4/25/25 08:38, Markus Armbruster wrote:
> Pierrick Bouvier <pierrick.bouvier(a)linaro.org> writes:
>
> > Note: This RFC was posted to trigger a discussion around this topic, and
it's
> > not expected to merge it as it is.
> >
> > Context
> > =======
> >
> > Linaro is working towards heterogeneous emulation, mixing several
architectures
> > in a single QEMU process. The first prerequisite is to be able to build such a
> > binary, which we commonly name "single-binary" in our various
series.
> > An (incomplete) list of series is available here:
> > https://patchew.org/search?q=project%3AQEMU+single-binary
> >
> > We don't expect to change existing command line interface or any
observable
> > behaviour, it should be identical to existing binaries. If anyone notices a
> > difference, it will be a bug.
>
> Define "notice a difference" :) More on that below.
>
Given a single-binary *named* exactly like an existing qemu-system-X binary,
any user or QEMU management layer should not be able to distinguish it from
the real qemu-system-X binary.
The new potential things will be:
- introduction of an (optional) -target option, which allows to
override/disambiguate default target detected.
- potentially more boards/cpus/devices visible, once we start developing
heterogeneous emulation. See it as a new CONFIG_{new_board} present.
Out of that, once the current target is identified, based on argv[0], there
should be absolutely no difference, whether in the behaviour, UI, command
line, or the monitor interfaces.
Okay, so assuming that the correctly named binaries will apply whatever
necessary magic to have them behave identically as they did.
I'll also ignore the distros that rename them assuming they do it in a
way that stays compatible.
The question is how the new unified binary will behave when being
introspected:
- Can the unified binary be introspected without selecting an
architecture?
(by introspection I mean starting with -M none and querying stuff via
QMP)
if no: libvirt will have a chicken&egg problem deciding what to do
- What will be the answer for the platform-specific stuff such as CPU
definitions?
e.g. does/can an architecture need to be instantiated later via QMP?
can it be changed dynamically?
3:55 a.m.
New subject: [RFC PATCH 0/3] single-binary: make QAPI generated files common
On Fri, Apr 25, 2025 at 17:38:44 +0200, Markus Armbruster via Devel wrote:
Pierrick Bouvier <pierrick.bouvier(a)linaro.org> writes:
[...]
To be precise: conditionals that use macros restricted to
target-specific code, i.e. the ones poisoned by exec/poison.h. Let's
call them target-specific QAPI conditionals.
The QAPI generator is blissfully unaware of all this.
The build system treats QAPI modules qapi/*-target.json as
target-specific. The .c files generated for them are compiled per
target. See qapi/meson.build.
Only such target-specific modules can can use target-specific QAPI
conditionals. Use in target-independent modules will generate C that
won't compile.
Poisoned macros used in qapi/*-target.json:
CONFIG_KVM
TARGET_ARM
TARGET_I386
TARGET_LOONGARCH64
TARGET_MIPS
TARGET_PPC
TARGET_RISCV
TARGET_S390X
I've had a look at what bits of the QMP schema are depending on the
above defines which libvirt uses.
In many cases libvirt could restrict the use of given command/property
to only supported architectures. We decided to simply probe the presence
of the command because it's convenient to not have to filter them any
more
- query-gic-capabilities
- libvirt already calls this only for ARM guests based on the
definition
- query-sev and friends
- libvirt uses presence of 'query-sev' to decide if the binary
supports it; patching in a platofrm check is possible although
inconvenient
- query-sgx and friends
- similar to sev
-query-cpu-definitions and friends
- see below
> What we try to do here is to build them only once
instead.
You're trying to eliminate target-specific QAPI conditionals. Correct?
> In the past, we identied that the best approach to solve this is to expose code
> for all targets (thus removing all #if clauses), and stub missing
> symbols for concerned targets.
This affects QAPI/QMP introspection, i.e. the value of query-qmp-schema.
Management applications can no longer use introspection to find out
whether target-specific things are available.
Indeed and libvirt already uses this in few cases as noded above.
For instance, query-cpu-definitions is implemented for targets arm,
i386, loongarch, mips, ppc, riscv, and s390x. It initially was for
fewer targets, and more targets followed one by one. Still more may
follow in the future. Right now, management applications can use
introspection to find out whether it is available. That stops working
when you make it available for all targets, stubbed out for the ones
that don't (yet) implement it.
Management applications may have to be adjusted for this.
This is not an attempt to shoot down your approach. I'm merely
demonstrating limitations of your promise "if anyone notices a
difference, it will be a bug."
Now, we could get really fancy and try to keep introspection the same by
applying conditionals dynamically somehow. I.e. have the single binary
return different introspection values depending on the actual guest's
target.
I wonder how this will work if libvirt is probing a binary. Libvirt does
not look at the filename. It can't because it can be a
user-specified/compiled binary, override script, or a distro that chose
to rename the binary.
The second thing that libvirt does after 'query-version' is
'query-target'.
So what should libvirt do once multiple targets are supported?
How do we query CPUs for each of the supported targets?
Will the result be the same if we query them one at a time or all at
once?
This requires fixing the target before introspection. Unless this
is
somehow completely transparent (wrapper scripts, or awful hacks based on
the binary's filename, perhaps), management applications may have to be
adjusted to actually do that.
As noted filename will not work. Users can specify any filename and
create override scripts or rename the binary.
Applies not just to introspection. Consider query-cpu-definitions
again. It currently returns CPU definitions for *the* target. What
would a single binary's query-cpu-definitions return? The CPU
definitions for *all* its targets? Management applications then receive
CPUs that won't work, which may upset them. To avoid noticable
difference, we again have to fix the target before we look.
Ah I see you had a similar question :D
Of course, "fixing the target" stops making sense once we move to
heterogeneous machines with multiple targets.
> This series build QAPI generated code once, by removing all TARGET_{arch} and
> CONFIG_KVM clauses. What it does *not* at the moment is:
> - prevent target specific commands to be visible for all targets
> (see TODO comment on patch 2 explaining how to address this)
> - nothing was done to hide all this from generated documentation
6:07 a.m.
New subject: [RFC PATCH 0/3] single-binary: make QAPI generated files common
Peter Krempa <pkrempa(a)redhat.com> writes:
On Fri, Apr 25, 2025 at 17:38:44 +0200, Markus Armbruster via Devel
wrote:
> Pierrick Bouvier <pierrick.bouvier(a)linaro.org> writes:
[...]
> To be precise: conditionals that use macros restricted to
> target-specific code, i.e. the ones poisoned by exec/poison.h. Let's
> call them target-specific QAPI conditionals.
>
> The QAPI generator is blissfully unaware of all this.
>
> The build system treats QAPI modules qapi/*-target.json as
> target-specific. The .c files generated for them are compiled per
> target. See qapi/meson.build.
>
> Only such target-specific modules can can use target-specific QAPI
> conditionals. Use in target-independent modules will generate C that
> won't compile.
>
> Poisoned macros used in qapi/*-target.json:
>
> CONFIG_KVM
> TARGET_ARM
> TARGET_I386
> TARGET_LOONGARCH64
> TARGET_MIPS
> TARGET_PPC
> TARGET_RISCV
> TARGET_S390X
Commands and events:
CPU introspection: query-cpu-model-baseline, query-cpu-model-comparison,
query-cpu-model-expansion, query-cpu-definitions
S390 KVM CPU stuff: set-cpu-topology, CPU_POLARIZATION_CHANGE,
query-s390x-cpu-polarization.
GIC: query-gic-capabilities
SEV: query-sev, query-sev-launch-measure, query-sev-capabilities,
sev-inject-launch-secret, query-sev-attestation-report
SGX: query-sgx, query-sgx-capabilities
Xen: xen-event-list, xen-event-inject
An odd duck: rtc-reset-reinjection
I've had a look at what bits of the QMP schema are depending on
the
above defines which libvirt uses.
In many cases libvirt could restrict the use of given command/property
to only supported architectures. We decided to simply probe the presence
of the command because it's convenient to not have to filter them any
more
- query-gic-capabilities
- libvirt already calls this only for ARM guests based on the
definition
- query-sev and friends
- libvirt uses presence of 'query-sev' to decide if the binary
supports it; patching in a platofrm check is possible although
inconvenient
- query-sgx and friends
- similar to sev
-query-cpu-definitions and friends
- see below
Large subset of my list.
> > What we try to do here is to
build them only once
> instead.
>
> You're trying to eliminate target-specific QAPI conditionals. Correct?
>
> > In the past, we identied that the best approach to solve this is to expose code
> > for all targets (thus removing all #if clauses), and stub missing
> > symbols for concerned targets.
>
> This affects QAPI/QMP introspection, i.e. the value of query-qmp-schema.
>
> Management applications can no longer use introspection to find out
> whether target-specific things are available.
Indeed and libvirt already uses this in few cases as noded above.
>
> For instance, query-cpu-definitions is implemented for targets arm,
> i386, loongarch, mips, ppc, riscv, and s390x. It initially was for
> fewer targets, and more targets followed one by one. Still more may
> follow in the future. Right now, management applications can use
> introspection to find out whether it is available. That stops working
> when you make it available for all targets, stubbed out for the ones
> that don't (yet) implement it.
>
> Management applications may have to be adjusted for this.
>
> This is not an attempt to shoot down your approach. I'm merely
> demonstrating limitations of your promise "if anyone notices a
> difference, it will be a bug."
>
> Now, we could get really fancy and try to keep introspection the same by
> applying conditionals dynamically somehow. I.e. have the single binary
> return different introspection values depending on the actual guest's
> target.
I wonder how this will work if libvirt is probing a binary. Libvirt does
not look at the filename. It can't because it can be a
user-specified/compiled binary, override script, or a distro that chose
to rename the binary.
The second thing that libvirt does after 'query-version' is
'query-target'.
So what should libvirt do once multiple targets are supported?
How do we query CPUs for each of the supported targets?
Will the result be the same if we query them one at a time or all at
once?
Pierrick's stated goal is to have no noticable differences between the
single binary and the qemu-system-<target> it covers. This is obviously
impossible if we can interact with the single binary before the target
is fixed.
> This requires fixing the target before introspection. Unless
this is
> somehow completely transparent (wrapper scripts, or awful hacks based on
> the binary's filename, perhaps), management applications may have to be
> adjusted to actually do that.
As noted filename will not work. Users can specify any filename and
create override scripts or rename the binary.
True.
> Applies not just to introspection. Consider
query-cpu-definitions
> again. It currently returns CPU definitions for *the* target. What
> would a single binary's query-cpu-definitions return? The CPU
> definitions for *all* its targets? Management applications then receive
> CPUs that won't work, which may upset them. To avoid noticable
> difference, we again have to fix the target before we look.
Ah I see you had a similar question :D
>
> Of course, "fixing the target" stops making sense once we move to
> heterogeneous machines with multiple targets.
>
> > This series build QAPI generated code once, by removing all TARGET_{arch} and
> > CONFIG_KVM clauses. What it does *not* at the moment is:
> > - prevent target specific commands to be visible for all targets
> > (see TODO comment on patch 2 explaining how to address this)
> > - nothing was done to hide all this from generated documentation
7:48 a.m.
New subject: [RFC PATCH 0/3] single-binary: make QAPI generated files common
On 28/4/25 13:07, Markus Armbruster wrote:
Peter Krempa <pkrempa(a)redhat.com> writes:
> The second thing that libvirt does after 'query-version'
is
> 'query-target'.
>
> So what should libvirt do once multiple targets are supported?
>
> How do we query CPUs for each of the supported targets?
>
> Will the result be the same if we query them one at a time or all at
> once?
Pierrick's stated goal is to have no noticable differences between the
single binary and the qemu-system-<target> it covers. This is obviously
impossible if we can interact with the single binary before the target
is fixed.
My naive impression is "management applications" aims mostly for
virtualization (likely 1 single target). Heterogeneous (more than 1
target) setups imply some kind of emulation.
Are *current* "management applications" interested in managing
heterogeneous machines? If so, we could introduce 'query-targets'
(note the 's' suffix for plural).
Some users (EDA industry) are interested in using some QEMU transport
layer (QMP?) to dynamically create machines (see [*] for example).
IIUC only a subset of current QMP commands is needed for that.
Maybe we can only adapt and enable these required commands for the
heterogeneous binary, and keep the current ones unmodified for the
single-target binary (where you can run a single target at a time,
identically to current qemu-system-FOO binaries).
[*]
https://lore.kernel.org/qemu-devel/20220223090706.4888-1-damien.hedde@gre...
0
days inactive
3
days old
7 comments
4 participants
participants (4)
-
Markus Armbruster -
Peter Krempa -
Philippe Mathieu-Daudé -
Pierrick Bouvier