[PATCH 1/1] qemu: stop silently narrowing the guest CPU during live migration
qemuDomainMakeCPUMigratable() strips features marked added='yes' (in src/cpu_map/x86_*.xml) from the migration cookie when the source CPU was specified as host-model. The intent was libvirt-protocol compat with older destinations; the cost is guest CPU compat, paid silently on every migration. Every Intel x86 CPU model from Westmere through Sapphire Rapids carries 60+ added='yes' features, including vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any of these models drops those features from the destination's qemu argv. Modern qemu gates the nested VMX capability MSRs on the explicit -cpu list, so the guest's MSR view shifts. Linux distributions that load kvm_intel by default snapshot those MSRs at module-load time and validate every newly online CPU against that snapshot. An automated test that boots a guest, live-migrates it, and hot-plugs additional vCPUs reliably trips kvm_intel: Inconsistent VMCS config on CPU N kvm: enabling virtualization on CPUN failed smpboot: CPU N is now offline the guest agent's online attempt returns -EIO and the hot-plug fails. Live migration must preserve the guest's CPU specification bit for bit -- this is a hard contract, not an optimisation target. Drop the strip. If a destination libvirt does not know a feature in the cookie, its parser rejects the migration with a precise unknown-feature error: operators can upgrade or narrow the source CPU definition. Either is visible; the status quo is not. This effectively reverts 14d3517410 ("qemu: domain: Drop added features from migratable CPU") together with its follow-up aae8a5774b ("qemu: Drop vmx-* from migratable CPU model only when origCPU is set"), and removes the now-unused origCPU plumbing in qemuDomainMakeCPUMigratable() and its callers. Signed-off-by: Denis V. Lunev <den@openvz.org> --- src/qemu/qemu_domain.c | 46 ++------------------------------ src/qemu/qemu_domain.h | 3 +-- src/qemu/qemu_migration_cookie.c | 5 +--- 3 files changed, 4 insertions(+), 50 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 84c8645259..eccd279bbe 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -5307,36 +5307,10 @@ qemuDomainDefCopy(virQEMUDriver *driver, } -typedef struct { - const char * const *added; - GStrv keep; -} qemuDomainDropAddedCPUFeaturesData; - - -static bool -qemuDomainDropAddedCPUFeatures(const char *name, - virCPUFeaturePolicy policy G_GNUC_UNUSED, - void *opaque) -{ - qemuDomainDropAddedCPUFeaturesData *data = opaque; - - if (!g_strv_contains(data->added, name)) - return true; - - if (data->keep && g_strv_contains((const char **) data->keep, name)) - return true; - - return false; -} - - int qemuDomainMakeCPUMigratable(virArch arch, - virCPUDef *cpu, - virCPUDef *origCPU) + virCPUDef *cpu) { - qemuDomainDropAddedCPUFeaturesData data = { 0 }; - if (cpu->mode != VIR_CPU_MODE_CUSTOM || !cpu->model || !ARCH_IS_X86(arch)) @@ -5353,22 +5327,6 @@ qemuDomainMakeCPUMigratable(virArch arch, virCPUDefUpdateFeature(cpu, "pconfig", VIR_CPU_FEATURE_DISABLE); } - if (origCPU) { - if (virCPUx86GetAddedFeatures(cpu->model, &data.added) < 0) - return -1; - - /* Drop features marked as added in a cpu model, but only - * when they are not mentioned in origCPU, i.e., when they were not - * explicitly mentioned by the user. - */ - if (data.added) { - g_auto(GStrv) keep = virCPUDefListExplicitFeatures(origCPU); - data.keep = keep; - - virCPUDefFilterFeatures(cpu, qemuDomainDropAddedCPUFeatures, &data); - } - } - return 0; } @@ -5569,7 +5527,7 @@ qemuDomainDefFormatBufInternal(virQEMUDriver *driver, } if (def->cpu && - qemuDomainMakeCPUMigratable(def->os.arch, def->cpu, origCPU) < 0) + qemuDomainMakeCPUMigratable(def->os.arch, def->cpu) < 0) return -1; /* Old libvirt doesn't understand <audio> elements so diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index b321a64e96..0600af8b7c 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -1066,8 +1066,7 @@ qemuDomainValidateActualNetDef(const virDomainNetDef *net, int qemuDomainMakeCPUMigratable(virArch arch, - virCPUDef *cpu, - virCPUDef *origCPU); + virCPUDef *cpu); int qemuDomainInitializePflashStorageSource(virDomainObj *vm, diff --git a/src/qemu/qemu_migration_cookie.c b/src/qemu/qemu_migration_cookie.c index 7311a8294b..ebc2072ab0 100644 --- a/src/qemu/qemu_migration_cookie.c +++ b/src/qemu/qemu_migration_cookie.c @@ -535,15 +535,12 @@ static int qemuMigrationCookieAddCPU(qemuMigrationCookie *mig, virDomainObj *vm) { - qemuDomainObjPrivate *priv = vm->privateData; - if (mig->cpu) return 0; mig->cpu = virCPUDefCopy(vm->def->cpu); - if (qemuDomainMakeCPUMigratable(vm->def->os.arch, mig->cpu, - priv->origCPU) < 0) + if (qemuDomainMakeCPUMigratable(vm->def->os.arch, mig->cpu) < 0) return -1; mig->flags |= QEMU_MIGRATION_COOKIE_CPU; -- 2.51.0
On Tue, May 26, 2026 at 17:17:10 +0200, Denis V. Lunev wrote:
qemuDomainMakeCPUMigratable() strips features marked added='yes' (in src/cpu_map/x86_*.xml) from the migration cookie when the source CPU was specified as host-model. The intent was libvirt-protocol compat with older destinations; the cost is guest CPU compat, paid silently on every migration.
Sigh, yeah there's still (at least) one thing missing around vmx migration...
Every Intel x86 CPU model from Westmere through Sapphire Rapids carries 60+ added='yes' features, including vmx-exit-load-perf-global-ctrl and vmx-entry-load-perf-global-ctrl that control the LOAD_IA32_PERF_GLOBAL_CTRL allowed-1 bits of MSR_IA32_VMX_{EXIT,ENTRY}_CTLS. A host-model live migration on any of these models drops those features from the destination's qemu argv.
Yeah, unless those vmx features are explicitly specified in the domain XML, libvirt adds them according to what QEMU enabled based on the -cpu command line and then removes them during migration. The assumption is that if QEMU automatically added them on the source host, it will also add them on the destination host.
Modern qemu gates the nested VMX capability MSRs on the explicit -cpu list, so the guest's MSR view shifts.
What exactly "modern qemu" means? Do you have an exact version? Anyway, this sounds like a QEMU bug to me. If a combination of -cpu command line and a machine type enabled the features on the source, the same combination should enable them on the source as well. The machine type does not change during migration and treating vmx features shouldn't change either.
Drop the strip. If a destination libvirt does not know a feature in the cookie, its parser rejects the migration with a precise unknown-feature error: operators can upgrade or narrow the source CPU definition. Either is visible; the status quo is not.
The problem is old libvirt was not tracking vmx features at all and thus any domain started on new libvirt would fail to migrate to an older libvirt. If a user explicitly required a vmx feature in domain XML, old libvirt would correctly refuse incoming migration because of an unknown CPU feature. But it's new libvirt suddenly recognizing features QEMU always enabled and adding them to domain XML. We always deal with such situation by dropping the automatically added XML elements for migration compatibility.
This effectively reverts 14d3517410 ("qemu: domain: Drop added features from migratable CPU") together with its follow-up aae8a5774b ("qemu: Drop vmx-* from migratable CPU model only when origCPU is set"), and removes the now-unused origCPU plumbing in qemuDomainMakeCPUMigratable() and its callers.
Unfortunately this is not the correct solution. We don't have a policy for backward migration compatibility with old libvirt so we should keep the compatibility as long as the domain XML does not contain anything unknown to the old libvirt. That said, we're missing a code that would transfer all the removed vmx flags in a migration cookie to make sure new libvirt can see the exact CPU definition and thus can explicitly request all the vmx features the source QEMU added. Jirka
participants (2)
-
Denis V. Lunev -
Jiří Denemark