[PATCH 1/1] qemu: stop silently narrowing the guest CPU during live migration
qemuDomainMakeCPUMigratable() strips features marked added='yes' (in src/cpu_map/x86_*.xml) from the migration cookie when the source CPU was specified as host-model. The intent was libvirt-protocol compat with older destinations; the cost is guest CPU compat, paid silently on every migration. Every Intel x86 CPU model from Westmere through Sapphire Rapids carries 60+ added='yes' features, including vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any of these models drops those features from the destination's qemu argv. Modern qemu gates the nested VMX capability MSRs on the explicit -cpu list, so the guest's MSR view shifts. Linux distributions that load kvm_intel by default snapshot those MSRs at module-load time and validate every newly online CPU against that snapshot. An automated test that boots a guest, live-migrates it, and hot-plugs additional vCPUs reliably trips kvm_intel: Inconsistent VMCS config on CPU N kvm: enabling virtualization on CPUN failed smpboot: CPU N is now offline the guest agent's online attempt returns -EIO and the hot-plug fails. Live migration must preserve the guest's CPU specification bit for bit -- this is a hard contract, not an optimisation target. Drop the strip. If a destination libvirt does not know a feature in the cookie, its parser rejects the migration with a precise unknown-feature error: operators can upgrade or narrow the source CPU definition. Either is visible; the status quo is not. This effectively reverts 14d3517410 ("qemu: domain: Drop added features from migratable CPU") together with its follow-up aae8a5774b ("qemu: Drop vmx-* from migratable CPU model only when origCPU is set"), and removes the now-unused origCPU plumbing in qemuDomainMakeCPUMigratable() and its callers. Signed-off-by: Denis V. Lunev <den@openvz.org> --- src/qemu/qemu_domain.c | 46 ++------------------------------ src/qemu/qemu_domain.h | 3 +-- src/qemu/qemu_migration_cookie.c | 5 +--- 3 files changed, 4 insertions(+), 50 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 84c8645259..eccd279bbe 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -5307,36 +5307,10 @@ qemuDomainDefCopy(virQEMUDriver *driver, } -typedef struct { - const char * const *added; - GStrv keep; -} qemuDomainDropAddedCPUFeaturesData; - - -static bool -qemuDomainDropAddedCPUFeatures(const char *name, - virCPUFeaturePolicy policy G_GNUC_UNUSED, - void *opaque) -{ - qemuDomainDropAddedCPUFeaturesData *data = opaque; - - if (!g_strv_contains(data->added, name)) - return true; - - if (data->keep && g_strv_contains((const char **) data->keep, name)) - return true; - - return false; -} - - int qemuDomainMakeCPUMigratable(virArch arch, - virCPUDef *cpu, - virCPUDef *origCPU) + virCPUDef *cpu) { - qemuDomainDropAddedCPUFeaturesData data = { 0 }; - if (cpu->mode != VIR_CPU_MODE_CUSTOM || !cpu->model || !ARCH_IS_X86(arch)) @@ -5353,22 +5327,6 @@ qemuDomainMakeCPUMigratable(virArch arch, virCPUDefUpdateFeature(cpu, "pconfig", VIR_CPU_FEATURE_DISABLE); } - if (origCPU) { - if (virCPUx86GetAddedFeatures(cpu->model, &data.added) < 0) - return -1; - - /* Drop features marked as added in a cpu model, but only - * when they are not mentioned in origCPU, i.e., when they were not - * explicitly mentioned by the user. - */ - if (data.added) { - g_auto(GStrv) keep = virCPUDefListExplicitFeatures(origCPU); - data.keep = keep; - - virCPUDefFilterFeatures(cpu, qemuDomainDropAddedCPUFeatures, &data); - } - } - return 0; } @@ -5569,7 +5527,7 @@ qemuDomainDefFormatBufInternal(virQEMUDriver *driver, } if (def->cpu && - qemuDomainMakeCPUMigratable(def->os.arch, def->cpu, origCPU) < 0) + qemuDomainMakeCPUMigratable(def->os.arch, def->cpu) < 0) return -1; /* Old libvirt doesn't understand <audio> elements so diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index b321a64e96..0600af8b7c 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -1066,8 +1066,7 @@ qemuDomainValidateActualNetDef(const virDomainNetDef *net, int qemuDomainMakeCPUMigratable(virArch arch, - virCPUDef *cpu, - virCPUDef *origCPU); + virCPUDef *cpu); int qemuDomainInitializePflashStorageSource(virDomainObj *vm, diff --git a/src/qemu/qemu_migration_cookie.c b/src/qemu/qemu_migration_cookie.c index 7311a8294b..ebc2072ab0 100644 --- a/src/qemu/qemu_migration_cookie.c +++ b/src/qemu/qemu_migration_cookie.c @@ -535,15 +535,12 @@ static int qemuMigrationCookieAddCPU(qemuMigrationCookie *mig, virDomainObj *vm) { - qemuDomainObjPrivate *priv = vm->privateData; - if (mig->cpu) return 0; mig->cpu = virCPUDefCopy(vm->def->cpu); - if (qemuDomainMakeCPUMigratable(vm->def->os.arch, mig->cpu, - priv->origCPU) < 0) + if (qemuDomainMakeCPUMigratable(vm->def->os.arch, mig->cpu) < 0) return -1; mig->flags |= QEMU_MIGRATION_COOKIE_CPU; -- 2.51.0
On Tue, May 26, 2026 at 17:17:10 +0200, Denis V. Lunev wrote:
qemuDomainMakeCPUMigratable() strips features marked added='yes' (in src/cpu_map/x86_*.xml) from the migration cookie when the source CPU was specified as host-model. The intent was libvirt-protocol compat with older destinations; the cost is guest CPU compat, paid silently on every migration.
Sigh, yeah there's still (at least) one thing missing around vmx migration...
Every Intel x86 CPU model from Westmere through Sapphire Rapids carries 60+ added='yes' features, including vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any of these models drops those features from the destination's qemu argv.
Yeah, unless those vmx features are explicitly specified in the domain XML, libvirt adds them according to what QEMU enabled based on the -cpu command line and then removes them during migration. The assumption is that if QEMU automatically added them on the source host, it will also add them on the destination host.
Modern qemu gates the nested VMX capability MSRs on the explicit -cpu list, so the guest's MSR view shifts.
What exactly "modern qemu" means? Do you have an exact version? Anyway, this sounds like a QEMU bug to me. If a combination of -cpu command line and a machine type enabled the features on the source, the same combination should enable them on the source as well. The machine type does not change during migration and treating vmx features shouldn't change either.
Drop the strip. If a destination libvirt does not know a feature in the cookie, its parser rejects the migration with a precise unknown-feature error: operators can upgrade or narrow the source CPU definition. Either is visible; the status quo is not.
The problem is old libvirt was not tracking vmx features at all and thus any domain started on new libvirt would fail to migrate to an older libvirt. If a user explicitly required a vmx feature in domain XML, old libvirt would correctly refuse incoming migration because of an unknown CPU feature. But it's new libvirt suddenly recognizing features QEMU always enabled and adding them to domain XML. We always deal with such situation by dropping the automatically added XML elements for migration compatibility.
This effectively reverts 14d3517410 ("qemu: domain: Drop added features from migratable CPU") together with its follow-up aae8a5774b ("qemu: Drop vmx-* from migratable CPU model only when origCPU is set"), and removes the now-unused origCPU plumbing in qemuDomainMakeCPUMigratable() and its callers.
Unfortunately this is not the correct solution. We don't have a policy for backward migration compatibility with old libvirt so we should keep the compatibility as long as the domain XML does not contain anything unknown to the old libvirt. That said, we're missing a code that would transfer all the removed vmx flags in a migration cookie to make sure new libvirt can see the exact CPU definition and thus can explicitly request all the vmx features the source QEMU added. Jirka
(sorry for later answer - I was on vacation last week). On 6/1/26 15:44, Jiří Denemark wrote:
On Tue, May 26, 2026 at 17:17:10 +0200, Denis V. Lunev wrote:
qemuDomainMakeCPUMigratable() strips features marked added='yes' (in src/cpu_map/x86_*.xml) from the migration cookie when the source CPU was specified as host-model. The intent was libvirt-protocol compat with older destinations; the cost is guest CPU compat, paid silently on every migration. Sigh, yeah there's still (at least) one thing missing around vmx migration...
Every Intel x86 CPU model from Westmere through Sapphire Rapids carries 60+ added='yes' features, including vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any of these models drops those features from the destination's qemu argv. Yeah, unless those vmx features are explicitly specified in the domain XML, libvirt adds them according to what QEMU enabled based on the -cpu command line and then removes them during migration. The assumption is that if QEMU automatically added them on the source host, it will also add them on the destination host.
I agree with this assumption, but the problem is subtle. Libvirt specifies these features using command line on source and misses them on target. This is the bug and QEMU is innocent here. It works specifically as wanted - if there are options, QEMU enables features and without them does not provide options to the guest. My statement here is very simple. We are able to catch user visible problem with this. VMX features are noticeably inconsistent and lucky enough to get this triggered not in complex nested behavior but with very simple CPU hotplug, which has really checked these features.
Modern qemu gates the nested VMX capability MSRs on the explicit -cpu list, so the guest's MSR view shifts. What exactly "modern qemu" means? Do you have an exact version? Anyway, this sounds like a QEMU bug to me. If a combination of -cpu command line and a machine type enabled the features on the source, the same combination should enable them on the source as well. The machine type does not change during migration and treating vmx features shouldn't change either. The problem has been reported for QEMU 10.0 but there is no difference with upstream. I have not checked specifically, but I expect no difference. As written above - QEMU is dumb and should be dumb. CPU options specified via command line should be same on source and on target.
Drop the strip. If a destination libvirt does not know a feature in the cookie, its parser rejects the migration with a precise unknown-feature error: operators can upgrade or narrow the source CPU definition. Either is visible; the status quo is not. The problem is old libvirt was not tracking vmx features at all and thus any domain started on new libvirt would fail to migrate to an older libvirt. If a user explicitly required a vmx feature in domain XML, old libvirt would correctly refuse incoming migration because of an unknown CPU feature. But it's new libvirt suddenly recognizing features QEMU always enabled and adding them to domain XML. We always deal with such situation by dropping the automatically added XML elements for migration compatibility. Are you supposing that we should pass all options in migration cookie? That would be sane. But the problem if both way problematic. Features should not gone and should not appear.
This effectively reverts 14d3517410 ("qemu: domain: Drop added features from migratable CPU") together with its follow-up aae8a5774b ("qemu: Drop vmx-* from migratable CPU model only when origCPU is set"), and removes the now-unused origCPU plumbing in qemuDomainMakeCPUMigratable() and its callers. Unfortunately this is not the correct solution. We don't have a policy for backward migration compatibility with old libvirt so we should keep the compatibility as long as the domain XML does not contain anything unknown to the old libvirt.
That said, we're missing a code that would transfer all the removed vmx flags in a migration cookie to make sure new libvirt can see the exact CPU definition and thus can explicitly request all the vmx features the source QEMU added.
Jirka
Let us discuss what can we do. The problem is real and subtle. This is definitely Libvirt side problem. Thank you in advance, Den
On Mon, Jun 08, 2026 at 14:39:00 +0200, Denis V. Lunev wrote:
(sorry for later answer - I was on vacation last week).
On 6/1/26 15:44, Jiří Denemark wrote:
On Tue, May 26, 2026 at 17:17:10 +0200, Denis V. Lunev wrote:
qemuDomainMakeCPUMigratable() strips features marked added='yes' (in src/cpu_map/x86_*.xml) from the migration cookie when the source CPU was specified as host-model. The intent was libvirt-protocol compat with older destinations; the cost is guest CPU compat, paid silently on every migration. Sigh, yeah there's still (at least) one thing missing around vmx migration...
Every Intel x86 CPU model from Westmere through Sapphire Rapids carries 60+ added='yes' features, including vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any of these models drops those features from the destination's qemu argv. Yeah, unless those vmx features are explicitly specified in the domain XML, libvirt adds them according to what QEMU enabled based on the -cpu command line and then removes them during migration. The assumption is that if QEMU automatically added them on the source host, it will also add them on the destination host.
I agree with this assumption, but the problem is subtle. Libvirt specifies these features using command line on source and misses them on target.
Hmm, somehow I managed to miss you were talking about host-model CPU. That changes the situation quite a bit. You're right we explicitly pass all those vmx features to QEMU in -cpu rather than adding them to the XML according to what QEMU enabled without asking. In my reply I was thinking about a custom CPU mode and we must not break that while fixing host-model. So instead of reverting the compatibility code, We should consider all features we eventually pass to -cpu on the source as explicitly requested. That is, even those that come from host-model CPU. This should fix the bug as the vmx features would not be dropped from the CPU definition during migration. The custom CPU mode would keep working as it is now and host-model has never been guaranteed to be migratable to older libvirt releases so we're not breaking anything. Jirka
On 6/17/26 11:27, Jiří Denemark wrote:
(sorry for later answer - I was on vacation last week).
On 6/1/26 15:44, Jiří Denemark wrote:
On Tue, May 26, 2026 at 17:17:10 +0200, Denis V. Lunev wrote:
qemuDomainMakeCPUMigratable() strips features marked added='yes' (in src/cpu_map/x86_*.xml) from the migration cookie when the source CPU was specified as host-model. The intent was libvirt-protocol compat with older destinations; the cost is guest CPU compat, paid silently on every migration. Sigh, yeah there's still (at least) one thing missing around vmx migration...
Every Intel x86 CPU model from Westmere through Sapphire Rapids carries 60+ added='yes' features, including vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any of these models drops those features from the destination's qemu argv. Yeah, unless those vmx features are explicitly specified in the domain XML, libvirt adds them according to what QEMU enabled based on the -cpu command line and then removes them during migration. The assumption is that if QEMU automatically added them on the source host, it will also add them on the destination host. I agree with this assumption, but the problem is subtle. Libvirt specifies these features using command line on source and misses them on target. Hmm, somehow I managed to miss you were talking about host-model CPU. That changes the situation quite a bit. You're right we explicitly pass all those vmx features to QEMU in -cpu rather than adding them to the XML according to what QEMU enabled without asking. In my reply I was
On Mon, Jun 08, 2026 at 14:39:00 +0200, Denis V. Lunev wrote: thinking about a custom CPU mode and we must not break that while fixing host-model.
So instead of reverting the compatibility code, We should consider all features we eventually pass to -cpu on the source as explicitly requested. That is, even those that come from host-model CPU. This should fix the bug as the vmx features would not be dropped from the CPU definition during migration. The custom CPU mode would keep working as it is now and host-model has never been guaranteed to be migratable to older libvirt releases so we're not breaking anything.
Jirka
yes. this approach is much better :-) The patch effectively becomes 1-liner. Sent v2. Thank you for review, Den
BTW, could you please stop added people on the Cc list? At least in my case it causes your emails to go to spam. I don't really know why our spam filter marks your emails as spam, but I have a rule that checks for List-Id and moves the email to the right mailbox anyway. However, if you Cc me directly, the direct email usually comes first and because it lacks List-Id, it goes to spam. The second copy that goes via the mailing list is discarded by gmail as a duplicate so it never reaches my filters. Thanks, Jirka
On 6/17/26 11:35, Jiří Denemark wrote:
BTW, could you please stop added people on the Cc list? At least in my case it causes your emails to go to spam. I don't really know why our spam filter marks your emails as spam, but I have a rule that checks for List-Id and moves the email to the right mailbox anyway. However, if you Cc me directly, the direct email usually comes first and because it lacks List-Id, it goes to spam. The second copy that goes via the mailing list is discarded by gmail as a duplicate so it never reaches my filters.
Thanks, Jirka
ok. The problem is that it is unclear who should be primary reviewer. Den
On Wed, Jun 17, 2026 at 15:32:10 +0200, Denis V. Lunev wrote:
On 6/17/26 11:35, Jiří Denemark wrote:
BTW, could you please stop added people on the Cc list? At least in my case it causes your emails to go to spam. I don't really know why our spam filter marks your emails as spam, but I have a rule that checks for List-Id and moves the email to the right mailbox anyway. However, if you Cc me directly, the direct email usually comes first and because it lacks List-Id, it goes to spam. The second copy that goes via the mailing list is discarded by gmail as a duplicate so it never reaches my filters.
Thanks, Jirka
ok.
The problem is that it is unclear who should be primary reviewer.
"don't contact individual maintainers/developers directly with your patches; reviewers are subscribed to the mailing list" https://www.libvirt.org/submitting-patches.html#review-process
participants (4)
-
Denis V. Lunev -
Denis V. Lunev -
Jiří Denemark -
Peter Krempa