[PATCH 0/2] Properly handle tri-state for KVM options
Two of the qemu kvm_features (kvm-poll-control and kvm-pv-ipi) were only emitted when on or when off respectively, rather than handling the full tri-state for each (on, off, absent). This meant that config like <poll-control state='off'/> was a no-op, equivalent to not specifying poll-control at all, which fails to prevent QEMU from enabling it based on its own internal defaults. In particular, this meant that having both <cpu mode='host-passthrough'> and <poll-control state='off'/> would cause the latter to be ignored, since libvirt would emit no explicit kvm-poll-control arg, and QEMU defaults to enabling most KVM features when CPU passthrough is enabled. Fix these two cases such that explicit on/off are both respected. The `hidden` and `hint-dedicated` options are left as-is, since these really do only make sense in the `on` direction; QEMU will never automatically enable them. Chris Riches (2): qemu: Emit explicit kvm-poll-control=off when disabled qemu: Emit explicit kvm-pv-ipi=on when enabled src/qemu/qemu_command.c | 14 ++++++++++---- .../kvm-features-off.x86_64-latest.args | 2 +- .../kvm-features.x86_64-latest.args | 2 +- 3 files changed, 12 insertions(+), 6 deletions(-) -- 2.43.0
From: Chris Riches <chris.riches@nutanix.com> Ever since commit b2757b697e (qemu: support kvm-pv-ipi off) introduced support for kvm-pv-ipi, libvirt has only emitted the arg when it is explicitly disabled. However, leaving the arg missing from the output is not equivalent to enabling it, since QEMU may default it to `off` in certain paths (for example, without `-cpu host`). Therefore, if the XML specifies state='on', we should explicitly set that to ensure it is enabled in qemu. Signed-off-by: Chris Riches <chris.riches@nutanix.com> --- src/qemu/qemu_command.c | 7 +++++-- tests/qemuxmlconfdata/kvm-features.x86_64-latest.args | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index dbc4b5ac94..d1e4e39c45 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6753,8 +6753,11 @@ qemuBuildCpuCommandLine(virCommand *cmd, break; case VIR_DOMAIN_KVM_PVIPI: - if (def->kvm_features->features[i] == VIR_TRISTATE_SWITCH_OFF) - virBufferAddLit(&buf, ",kvm-pv-ipi=off"); + if (def->kvm_features->features[i]) { + virBufferAsprintf(&buf, ",kvm-pv-ipi=%s", + def->kvm_features->features[i] == + VIR_TRISTATE_SWITCH_ON ? "on" : "off"); + } break; case VIR_DOMAIN_KVM_DIRTY_RING: diff --git a/tests/qemuxmlconfdata/kvm-features.x86_64-latest.args b/tests/qemuxmlconfdata/kvm-features.x86_64-latest.args index 8221fdb16f..90d3ffba55 100644 --- a/tests/qemuxmlconfdata/kvm-features.x86_64-latest.args +++ b/tests/qemuxmlconfdata/kvm-features.x86_64-latest.args @@ -12,7 +12,7 @@ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain--1-QEMUGuest1/master-key.aes"}' \ -machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=on \ -accel kvm,dirty-ring-size=4096 \ --cpu host,migratable=off,kvm=off,kvm-hint-dedicated=on,kvm-poll-control=on \ +-cpu host,migratable=off,kvm=off,kvm-hint-dedicated=on,kvm-poll-control=on,kvm-pv-ipi=on \ -m size=219136k \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":224395264}' \ -overcommit mem-lock=off \ -- 2.43.0
From: Chris Riches <chris.riches@nutanix.com> Ever since commit 3fc4412c6f (qemu: support kvm-poll-control performance hint) introduced support for kvm-poll-control, libvirt has only emitted the arg when it is explicitly enabled. However, leaving the arg missing from the output is not equivalent to disabling it, since QEMU may default it to `on` in certain paths (for example, with `-cpu host`). Therefore, if the XML specifies state='off', we should explicitly set that to ensure it is disabled in qemu. Signed-off-by: Chris Riches <chris.riches@nutanix.com> --- src/qemu/qemu_command.c | 7 +++++-- tests/qemuxmlconfdata/kvm-features-off.x86_64-latest.args | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index e726dc661c..dbc4b5ac94 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6745,8 +6745,11 @@ qemuBuildCpuCommandLine(virCommand *cmd, break; case VIR_DOMAIN_KVM_POLLCONTROL: - if (def->kvm_features->features[i] == VIR_TRISTATE_SWITCH_ON) - virBufferAddLit(&buf, ",kvm-poll-control=on"); + if (def->kvm_features->features[i]) { + virBufferAsprintf(&buf, ",kvm-poll-control=%s", + def->kvm_features->features[i] == + VIR_TRISTATE_SWITCH_ON ? "on" : "off"); + } break; case VIR_DOMAIN_KVM_PVIPI: diff --git a/tests/qemuxmlconfdata/kvm-features-off.x86_64-latest.args b/tests/qemuxmlconfdata/kvm-features-off.x86_64-latest.args index 70b1fb83e1..8c1e238e2e 100644 --- a/tests/qemuxmlconfdata/kvm-features-off.x86_64-latest.args +++ b/tests/qemuxmlconfdata/kvm-features-off.x86_64-latest.args @@ -12,7 +12,7 @@ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-QEMUGuest1/.config \ -object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain--1-QEMUGuest1/master-key.aes"}' \ -machine pc,usb=off,dump-guest-core=off,memory-backend=pc.ram,acpi=on \ -accel kvm \ --cpu host,migratable=off,kvm-pv-ipi=off \ +-cpu host,migratable=off,kvm-poll-control=off,kvm-pv-ipi=off \ -m size=219136k \ -object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":224395264}' \ -overcommit mem-lock=off \ -- 2.43.0
On Thu, Jun 25, 2026 at 14:12:22 +0100, Chris Riches via Devel wrote:
From: Chris Riches <chris.riches@nutanix.com>
Ever since commit 3fc4412c6f (qemu: support kvm-poll-control performance hint) introduced support for kvm-poll-control, libvirt has only emitted the arg when it is explicitly enabled. However, leaving the arg missing from the output is not equivalent to disabling it, since QEMU may default it to `on` in certain paths (for example, with `-cpu host`).
Therefore, if the XML specifies state='off', we should explicitly set that to ensure it is disabled in qemu.
Signed-off-by: Chris Riches <chris.riches@nutanix.com> --- src/qemu/qemu_command.c | 7 +++++-- tests/qemuxmlconfdata/kvm-features-off.x86_64-latest.args | 2 +- 2 files changed, 6 insertions(+), 3 deletions(-)
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index e726dc661c..dbc4b5ac94 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -6745,8 +6745,11 @@ qemuBuildCpuCommandLine(virCommand *cmd, break;
case VIR_DOMAIN_KVM_POLLCONTROL: - if (def->kvm_features->features[i] == VIR_TRISTATE_SWITCH_ON) - virBufferAddLit(&buf, ",kvm-poll-control=on"); + if (def->kvm_features->features[i]) {
Make this an explicit comparison: def->kvm_features->features[i] != VIR_TRISTATE_SWITCH_ABSENT
+ virBufferAsprintf(&buf, ",kvm-poll-control=%s", + def->kvm_features->features[i] == + VIR_TRISTATE_SWITCH_ON ? "on" : "off");
And this can use virTristateSwitchTypeToString instead of a hardcoded string conversion. One unfortunate thing is that this can break guest ABI. The ABI stability check 'virDomainDefFeaturesCheckABIStability' is correctly rejecting it but if we generate a config from an XML using the '_OFF' variant but the default was _ON the ABI will change. Do you know if there is a possibility to probe the current state from a running VM? If yes we'll likely have to reconcile the state from the running VM so that this doesn't happen once we start to honour the '_OFF' state explicitly. If there is a way to detect it it will need to go somewhere into the reconnection code path.
On 29/06/2026 13:47, Peter Krempa wrote:
case VIR_DOMAIN_KVM_POLLCONTROL: - if (def->kvm_features->features[i] == VIR_TRISTATE_SWITCH_ON) - virBufferAddLit(&buf, ",kvm-poll-control=on"); + if (def->kvm_features->features[i]) {
Make this an explicit comparison:
def->kvm_features->features[i] != VIR_TRISTATE_SWITCH_ABSENT
+ virBufferAsprintf(&buf, ",kvm-poll-control=%s", + def->kvm_features->features[i] == + VIR_TRISTATE_SWITCH_ON ? "on" : "off"); And this can use virTristateSwitchTypeToString instead of a hardcoded string conversion.
Thanks, will make those improvements.
One unfortunate thing is that this can break guest ABI. The ABI stability check 'virDomainDefFeaturesCheckABIStability' is correctly rejecting it but if we generate a config from an XML using the '_OFF' variant but the default was _ON the ABI will change.
Do you know if there is a possibility to probe the current state from a running VM? If yes we'll likely have to reconcile the state from the running VM so that this doesn't happen once we start to honour the '_OFF' state explicitly.
If there is a way to detect it it will need to go somewhere into the reconnection code path.
Ah, I hadn't considered the ABI. Is the following understanding of the problem correct? 1. If the XML had an explicit poll-control=off but QEMU had actually enabled it under the hood, then we upgrade libvirt, then do something like a save+restore, the restore will actually force poll control off and break the ABI. 2. If we are in the same libvirt-qemu mismatch, and do not upgrade libvirt but instead migrate the VM to a remote host that has already upgraded libvirt, then the incoming migration will have the same ABI breakage. The first case sounds solvable by new logic in the reconnect path as you suggest, but the second case sounds much harder to solve, since the source libvirt doesn't know how to tell the destination one about the mismatch. Have I understood that correctly, and do you have any ideas about how to resolve this? - Chris
On 30/06/2026 13:10, Chris Riches wrote:
[...]
Have I understood that correctly, and do you have any ideas about how to resolve this?
Hi Peter, gentle ping on the above thread. Is it possible to resolve this kind of backwards-compatibility mismatch, or are we stuck with the current behaviour forever? Any pointers you have towards how I should best account for this would be appreciated. - Chris
On Tue, Jun 30, 2026 at 13:10:34 +0100, Chris Riches wrote:
On 29/06/2026 13:47, Peter Krempa wrote:
case VIR_DOMAIN_KVM_POLLCONTROL: - if (def->kvm_features->features[i] == VIR_TRISTATE_SWITCH_ON) - virBufferAddLit(&buf, ",kvm-poll-control=on"); + if (def->kvm_features->features[i]) {
Make this an explicit comparison:
def->kvm_features->features[i] != VIR_TRISTATE_SWITCH_ABSENT
+ virBufferAsprintf(&buf, ",kvm-poll-control=%s", + def->kvm_features->features[i] == + VIR_TRISTATE_SWITCH_ON ? "on" : "off"); And this can use virTristateSwitchTypeToString instead of a hardcoded string conversion.
Thanks, will make those improvements.
One unfortunate thing is that this can break guest ABI. The ABI stability check 'virDomainDefFeaturesCheckABIStability' is correctly rejecting it but if we generate a config from an XML using the '_OFF' variant but the default was _ON the ABI will change.
Do you know if there is a possibility to probe the current state from a running VM? If yes we'll likely have to reconcile the state from the running VM so that this doesn't happen once we start to honour the '_OFF' state explicitly.
If there is a way to detect it it will need to go somewhere into the reconnection code path.
Ah, I hadn't considered the ABI. Is the following understanding of the problem correct?
1. If the XML had an explicit poll-control=off but QEMU had actually enabled it under the hood, then we upgrade libvirt, then do something like a save+restore, the restore will actually force poll control off and break the ABI. 2. If we are in the same libvirt-qemu mismatch, and do not upgrade libvirt but instead migrate the VM to a remote host that has already upgraded libvirt, then the incoming migration will have the same ABI breakage.
The first case sounds solvable by new logic in the reconnect path as you suggest, but the second case sounds much harder to solve, since the source libvirt doesn't know how to tell the destination one about the mismatch.
Have I understood that correctly, and do you have any ideas about how to resolve this?
Hmm it is indeed tricky. Especially to do somewhat "cleanly". Additional constraint is that we strive to support backward migration (if you start a VM on older libvirt you should be able to migrate it back to it even if it round-trips through new ones). The only thing that comes into my mind is to: 1) document that 'off' isn't really off 2) introduce a new enum value for 'really-off' (obviously with less terrible name 3) keep the behaviour for 'off' introduce explicit disable for 'really-off' That way new libvirt can see which one is used, and old libvirt will behave the same for 'off' and reject 'really-off'. But it is disgusting in the way that 'off' will not mean exactly the same thing as for others and this will be recorded only in the documentation. Optionally a new 'really-on' value could be added and suggested in the docs. Unfortunately anything else akin to adding another attribute or something will not work because old libvirt will ignore it.
On 31/07/2026 12:58, Peter Krempa wrote:
Hmm it is indeed tricky. Especially to do somewhat "cleanly".
Additional constraint is that we strive to support backward migration (if you start a VM on older libvirt you should be able to migrate it back to it even if it round-trips through new ones).
The only thing that comes into my mind is to:
1) document that 'off' isn't really off 2) introduce a new enum value for 'really-off' (obviously with less terrible name 3) keep the behaviour for 'off' introduce explicit disable for 'really-off'
That way new libvirt can see which one is used, and old libvirt will behave the same for 'off' and reject 'really-off'.
But it is disgusting in the way that 'off' will not mean exactly the same thing as for others and this will be recorded only in the documentation.
Optionally a new 'really-on' value could be added and suggested in the docs.
Yeah, I suppose something like "force-{off,on}" could work. It's a bit unpleasant as you say, though.
Unfortunately anything else akin to adding another attribute or something will not work because old libvirt will ignore it.
Two ideas that came to my mind: 1. For the backward migration case, is there any sort of precheck that could intercept the migration on the newer source side before sending data that would be misinterpreted by the older dest? This would have the same ultimate migration failure result, but frees up the need to create a new state variant. 2. Is there any kind of translation that happens when migrating from old to new? i.e. could the newer libvirt be aware the the source is older, and perform the same domain translation that the reconnect path would? If so, we wouldn't need any new XML at all, just the logic in those reconnect and "translate" paths to drop any "state=off" from the XML and turn it into absent. Although I fear that even in those cases, we wouldn't cover persistent but currently inactive defined domains - we'd have to somehow iterate over those and edit them too which seems also rather unpleasant. Interested in your thoughts. - Chris
participants (2)
-
Chris Riches -
Peter Krempa