[PATCH 0/2] qemu: Switch to new -numa memdev=

In a way, libvirt already uses -numa memdev= in a few cases. In fact, in as few cases as possible - only configurations which can not be configured with -numa mem=, because these two ways are incompatible when it comes to migration. My approach to solve this is to have a privateData flag which tells which directs libvirt to generate old or new cmd line. And then this flag is saved into migration cookie so that the destination is directed the same way. Problem with this approach is that, while migration 6.3.0 -> 6.4.0 -> 6.3.0 works, migration where the machine is started on newer libvirt 6.4.0 and then migrated back to 6.3.0 won't work (in fact is explicitly denied by 2/2) even though there is nothing visible that should prevent the migration. I am not sure whether we have a good move here, because even if we waited until QEMU removes the old way, it won't help us really. We would be just leaving the problem for future us. Michal Prívozník (2): qemu: Switch to new -numa memdev= qemu: Disallow migration to older -numa if newer is used src/qemu/qemu_command.c | 20 ++--- src/qemu/qemu_domain.c | 38 +++++++++- src/qemu/qemu_domain.h | 3 + src/qemu/qemu_migration.c | 18 ++++- src/qemu/qemu_migration_cookie.c | 73 +++++++++++++++++++ src/qemu/qemu_migration_cookie.h | 12 +++ src/qemu/qemu_process.c | 25 ++++++- .../qemustatusxml2xmldata/backup-pull-in.xml | 1 + .../blockjob-blockdev-in.xml | 1 + .../blockjob-mirror-in.xml | 1 + .../disk-secinfo-upgrade-in.xml | 1 + .../disk-secinfo-upgrade-out.xml | 1 + .../migration-in-params-in.xml | 1 + .../migration-out-nbd-in.xml | 1 + .../migration-out-nbd-out.xml | 1 + .../migration-out-nbd-tls-in.xml | 1 + .../migration-out-nbd-tls-out.xml | 1 + .../migration-out-params-in.xml | 1 + tests/qemustatusxml2xmldata/modern-in.xml | 1 + .../qemustatusxml2xmldata/vcpus-multi-in.xml | 1 + .../hugepages-numa-default-2M.args | 10 ++- .../hugepages-numa-default-dimm.args | 8 +- .../hugepages-numa-default.args | 6 +- .../memory-hotplug-dimm-addr.args | 3 +- .../qemuxml2argvdata/memory-hotplug-dimm.args | 3 +- ...y-hotplug-nvdimm-access.x86_64-latest.args | 3 +- ...ry-hotplug-nvdimm-align.x86_64-latest.args | 3 +- ...ry-hotplug-nvdimm-label.x86_64-latest.args | 3 +- ...ory-hotplug-nvdimm-pmem.x86_64-latest.args | 3 +- ...ory-hotplug-nvdimm-ppc64.ppc64-latest.args | 3 +- ...hotplug-nvdimm-readonly.x86_64-latest.args | 3 +- .../memory-hotplug-nvdimm.x86_64-latest.args | 3 +- .../numatune-auto-prefer.args | 4 +- .../qemuxml2argvdata/pages-dimm-discard.args | 3 +- .../pages-discard-hugepages.args | 11 ++- tests/qemuxml2argvdata/pages-discard.args | 12 ++- 36 files changed, 236 insertions(+), 47 deletions(-) -- 2.26.2

In v4.1.0-rc0~9^2~25 QEMU deprecated -numa mem= in favor of -numa memdev= + -object memory-backend-*. However, the problem is these two are incompatible. A domain started with older cmd line can't be migrated to the newer one and vice versa. That is why libvirt hasn't switched to the new cmd line, until now. Starting with this commit, the new cmd line is used and this fact is then recorded in domain private data object under priv->forceNewNuma. This means, that the status XML and migration cookie have it too and thus can instruct the other daemon which cmd line to generate. Unfortunately, migration from newer to older style will fail. I'm saving the explicit check for a separate commit. Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1783355 Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_command.c | 20 ++--- src/qemu/qemu_domain.c | 38 +++++++++- src/qemu/qemu_domain.h | 3 + src/qemu/qemu_migration.c | 5 +- src/qemu/qemu_migration_cookie.c | 73 +++++++++++++++++++ src/qemu/qemu_migration_cookie.h | 12 +++ src/qemu/qemu_process.c | 25 ++++++- .../qemustatusxml2xmldata/backup-pull-in.xml | 1 + .../blockjob-blockdev-in.xml | 1 + .../blockjob-mirror-in.xml | 1 + .../disk-secinfo-upgrade-in.xml | 1 + .../disk-secinfo-upgrade-out.xml | 1 + .../migration-in-params-in.xml | 1 + .../migration-out-nbd-in.xml | 1 + .../migration-out-nbd-out.xml | 1 + .../migration-out-nbd-tls-in.xml | 1 + .../migration-out-nbd-tls-out.xml | 1 + .../migration-out-params-in.xml | 1 + tests/qemustatusxml2xmldata/modern-in.xml | 1 + .../qemustatusxml2xmldata/vcpus-multi-in.xml | 1 + .../hugepages-numa-default-2M.args | 10 ++- .../hugepages-numa-default-dimm.args | 8 +- .../hugepages-numa-default.args | 6 +- .../memory-hotplug-dimm-addr.args | 3 +- .../qemuxml2argvdata/memory-hotplug-dimm.args | 3 +- ...y-hotplug-nvdimm-access.x86_64-latest.args | 3 +- ...ry-hotplug-nvdimm-align.x86_64-latest.args | 3 +- ...ry-hotplug-nvdimm-label.x86_64-latest.args | 3 +- ...ory-hotplug-nvdimm-pmem.x86_64-latest.args | 3 +- ...ory-hotplug-nvdimm-ppc64.ppc64-latest.args | 3 +- ...hotplug-nvdimm-readonly.x86_64-latest.args | 3 +- .../memory-hotplug-nvdimm.x86_64-latest.args | 3 +- .../numatune-auto-prefer.args | 4 +- .../qemuxml2argvdata/pages-dimm-discard.args | 3 +- .../pages-discard-hugepages.args | 11 ++- tests/qemuxml2argvdata/pages-discard.args | 12 ++- 36 files changed, 225 insertions(+), 45 deletions(-) diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c index 2d8a6fb0dd..3316aae419 100644 --- a/src/qemu/qemu_command.c +++ b/src/qemu/qemu_command.c @@ -7028,8 +7028,6 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg, char *tmpmask = NULL; char *next = NULL; virBufferPtr nodeBackends = NULL; - bool needBackend = false; - int rc; int ret = -1; size_t ncells = virDomainNumaGetNodeCount(def->numa); @@ -7039,23 +7037,21 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg, if (VIR_ALLOC_N(nodeBackends, ncells) < 0) goto cleanup; - /* using of -numa memdev= cannot be combined with -numa mem=, thus we - * need to check which approach to use */ + /* Using of -numa memdev= cannot be combined with -numa mem=. + * However, as of QEMU 4.1.0 using the latter is deprecated + * and the former is preferred. */ for (i = 0; i < ncells; i++) { if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_RAM) || virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_FILE) || virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_MEMFD)) { - if ((rc = qemuBuildMemoryCellBackendStr(def, cfg, i, priv, - &nodeBackends[i])) < 0) + if (qemuBuildMemoryCellBackendStr(def, cfg, i, priv, + &nodeBackends[i]) < 0) goto cleanup; - - if (rc == 0) - needBackend = true; } } - if (!needBackend && + if (priv->forceNewNuma == VIR_TRISTATE_BOOL_NO && qemuBuildMemPathStr(def, cmd, priv) < 0) goto cleanup; @@ -7064,7 +7060,7 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg, if (!(cpumask = virBitmapFormat(virDomainNumaGetNodeCpumask(def->numa, i)))) goto cleanup; - if (needBackend) { + if (priv->forceNewNuma == VIR_TRISTATE_BOOL_YES) { virCommandAddArg(cmd, "-object"); virCommandAddArgBuffer(cmd, &nodeBackends[i]); } @@ -7079,7 +7075,7 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg, virBufferAdd(&buf, tmpmask, -1); } - if (needBackend) + if (priv->forceNewNuma == VIR_TRISTATE_BOOL_YES) virBufferAsprintf(&buf, ",memdev=ram-node%zu", i); else virBufferAsprintf(&buf, ",mem=%llu", diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index a1b250fd0b..4751685452 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -2283,6 +2283,7 @@ qemuDomainObjPrivateDataClear(qemuDomainObjPrivatePtr priv) VIR_FREE(priv->channelTargetDir); priv->memPrealloc = false; + priv->forceNewNuma = VIR_TRISTATE_BOOL_ABSENT; /* remove automatic pinning data */ virBitmapFree(priv->autoNodeset); @@ -2802,6 +2803,15 @@ qemuDomainObjPrivateXMLFormatAllowReboot(virBufferPtr buf, } +static void +qemuDomainObjPrivateXMLFormatForceNewNuma(virBufferPtr buf, + virTristateBool forceNewNuma) +{ + virBufferAsprintf(buf, "<forceNewNuma value='%s'/>\n", + virTristateBoolTypeToString(forceNewNuma)); +} + + static void qemuDomainObjPrivateXMLFormatPR(virBufferPtr buf, qemuDomainObjPrivatePtr priv) @@ -3086,6 +3096,8 @@ qemuDomainObjPrivateXMLFormat(virBufferPtr buf, if (priv->memPrealloc) virBufferAddLit(buf, "<memPrealloc/>\n"); + qemuDomainObjPrivateXMLFormatForceNewNuma(buf, priv->forceNewNuma); + if (qemuDomainObjPrivateXMLFormatBlockjobs(buf, vm) < 0) return -1; @@ -3552,6 +3564,26 @@ qemuDomainObjPrivateXMLParseAllowReboot(xmlXPathContextPtr ctxt, } +static int +qemuDomainObjPrivateXMLParseForceNewNuma(xmlXPathContextPtr ctxt, + virTristateBool *forceNewNuma) +{ + int val; + g_autofree char *valStr = NULL; + + if ((valStr = virXPathString("string(./forceNewNuma/@value)", ctxt))) { + if ((val = virTristateBoolTypeFromString(valStr)) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("invalid allowReboot value '%s'"), valStr); + return -1; + } + *forceNewNuma = val; + } + + return 0; +} + + static void qemuDomainObjPrivateXMLParsePR(xmlXPathContextPtr ctxt, bool *prDaemonRunning) @@ -3936,7 +3968,8 @@ qemuDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt, priv->chardevStdioLogd = virXPathBoolean("boolean(./chardevStdioLogd)", ctxt) == 1; - qemuDomainObjPrivateXMLParseAllowReboot(ctxt, &priv->allowReboot); + if (qemuDomainObjPrivateXMLParseAllowReboot(ctxt, &priv->allowReboot) < 0) + goto error; qemuDomainObjPrivateXMLParsePR(ctxt, &priv->prDaemonRunning); @@ -3956,6 +3989,9 @@ qemuDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt, priv->memPrealloc = virXPathBoolean("boolean(./memPrealloc)", ctxt) == 1; + if (qemuDomainObjPrivateXMLParseForceNewNuma(ctxt, &priv->forceNewNuma) < 0) + goto error; + return 0; error: diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 41d3f1561d..dafb1da52f 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -418,6 +418,9 @@ struct _qemuDomainObjPrivate { /* true if global -mem-prealloc appears on cmd line */ bool memPrealloc; + /* true if new -numa memdev is forced */ + virTristateBool forceNewNuma; + /* running block jobs */ virHashTablePtr blockjobs; diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 02e8271e42..423713e00b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2140,6 +2140,7 @@ qemuMigrationSrcBeginPhase(virQEMUDriverPtr driver, cookieFlags |= QEMU_MIGRATION_COOKIE_CPU; cookieFlags |= QEMU_MIGRATION_COOKIE_ALLOW_REBOOT; + cookieFlags |= QEMU_MIGRATION_COOKIE_PRIVATE; if (!(flags & VIR_MIGRATE_OFFLINE)) cookieFlags |= QEMU_MIGRATION_COOKIE_CAPS; @@ -2501,7 +2502,8 @@ qemuMigrationDstPrepareAny(virQEMUDriverPtr driver, QEMU_MIGRATION_COOKIE_CPU_HOTPLUG | QEMU_MIGRATION_COOKIE_CPU | QEMU_MIGRATION_COOKIE_ALLOW_REBOOT | - QEMU_MIGRATION_COOKIE_CAPS))) + QEMU_MIGRATION_COOKIE_CAPS | + QEMU_MIGRATION_COOKIE_PRIVATE))) goto cleanup; if (!(vm = virDomainObjListAdd(driver->domains, *def, @@ -2556,6 +2558,7 @@ qemuMigrationDstPrepareAny(virQEMUDriverPtr driver, stopProcess = true; priv->allowReboot = mig->allowReboot; + priv->forceNewNuma = mig->priv && mig->priv->forceNewNuma; if (!(incoming = qemuMigrationDstPrepare(vm, tunnel, protocol, listenAddress, port, diff --git a/src/qemu/qemu_migration_cookie.c b/src/qemu/qemu_migration_cookie.c index fb8b5bcd92..032d954dca 100644 --- a/src/qemu/qemu_migration_cookie.c +++ b/src/qemu/qemu_migration_cookie.c @@ -51,6 +51,7 @@ VIR_ENUM_IMPL(qemuMigrationCookieFlag, "cpu", "allowReboot", "capabilities", + "private", ); @@ -580,6 +581,20 @@ qemuMigrationCookieAddCaps(qemuMigrationCookiePtr mig, } +static void +qemuMigrationCookieAddPrivate(qemuMigrationCookiePtr mig, + qemuDomainObjPrivatePtr priv) +{ + if (!mig->priv) + mig->priv = g_new0(qemuMigrationCookiePrivate, 1); + + mig->priv->forceNewNuma = priv->forceNewNuma; + + mig->flags |= QEMU_MIGRATION_COOKIE_PRIVATE; +} + + + static void qemuMigrationCookieGraphicsXMLFormat(virBufferPtr buf, qemuMigrationCookieGraphicsPtr grap) @@ -770,6 +785,25 @@ qemuMigrationCookieCapsXMLFormat(virBufferPtr buf, } +#define FORMAT_PRIV(member) \ + virBufferAsprintf(buf, "<member name='" #member "' value='%s'/>\n", \ + virTristateBoolTypeToString(priv->member)) + +static void +qemuMigrationCookiePrivateXMLFormat(virBufferPtr buf, + qemuMigrationCookiePrivatePtr priv) +{ + virBufferAddLit(buf, "<privateData>\n"); + virBufferAdjustIndent(buf, 2); + + FORMAT_PRIV(forceNewNuma); + + virBufferAdjustIndent(buf, -2); + virBufferAddLit(buf, "</privateData>\n"); +} + +#undef FORMAT_PRIV + static int qemuMigrationCookieXMLFormat(virQEMUDriverPtr driver, virQEMUCapsPtr qemuCaps, @@ -858,6 +892,9 @@ qemuMigrationCookieXMLFormat(virQEMUDriverPtr driver, if (mig->flags & QEMU_MIGRATION_COOKIE_CAPS) qemuMigrationCookieCapsXMLFormat(buf, mig->caps); + if (mig->flags & QEMU_MIGRATION_COOKIE_PRIVATE) + qemuMigrationCookiePrivateXMLFormat(buf, mig->priv); + virBufferAdjustIndent(buf, -2); virBufferAddLit(buf, "</qemu-migration>\n"); return 0; @@ -1179,6 +1216,35 @@ qemuMigrationCookieCapsXMLParse(xmlXPathContextPtr ctxt) } +#define PARSE_PRIV(member) \ + do { \ + int val; \ + g_autofree char *valStr = virXPathString("string(./privateData/member" \ + "[@name='" #member "']/@value)", ctxt); \ + if (valStr) { \ + if ((val = virTristateBoolTypeFromString(valStr)) < 0) { \ + virReportError(VIR_ERR_INTERNAL_ERROR, \ + _("Malformed member '%s' value %s"), \ + #member, valStr); \ + return NULL; \ + } \ + priv->member = val; \ + } \ + } while (0) + +static qemuMigrationCookiePrivatePtr +qemuMigrationCookiePrivateXMLParse(xmlXPathContextPtr ctxt) +{ + g_autofree qemuMigrationCookiePrivatePtr priv = NULL; + + priv = g_new0(qemuMigrationCookiePrivate, 1); + + PARSE_PRIV(forceNewNuma); + + return g_steal_pointer(&priv); +} + + static int qemuMigrationCookieXMLParse(qemuMigrationCookiePtr mig, virQEMUDriverPtr driver, @@ -1357,6 +1423,10 @@ qemuMigrationCookieXMLParse(qemuMigrationCookiePtr mig, !(mig->caps = qemuMigrationCookieCapsXMLParse(ctxt))) goto error; + if (flags & QEMU_MIGRATION_COOKIE_PRIVATE && + !(mig->priv = qemuMigrationCookiePrivateXMLParse(ctxt))) + goto error; + return 0; error: @@ -1446,6 +1516,9 @@ qemuMigrationBakeCookie(qemuMigrationCookiePtr mig, qemuMigrationCookieAddCaps(mig, dom, party) < 0) return -1; + if (flags & QEMU_MIGRATION_COOKIE_PRIVATE) + qemuMigrationCookieAddPrivate(mig, priv); + if (!(*cookieout = qemuMigrationCookieXMLFormatStr(driver, priv->qemuCaps, mig))) return -1; diff --git a/src/qemu/qemu_migration_cookie.h b/src/qemu/qemu_migration_cookie.h index 1e88684589..e1f44682a6 100644 --- a/src/qemu/qemu_migration_cookie.h +++ b/src/qemu/qemu_migration_cookie.h @@ -33,6 +33,7 @@ typedef enum { QEMU_MIGRATION_COOKIE_FLAG_CPU, QEMU_MIGRATION_COOKIE_FLAG_ALLOW_REBOOT, QEMU_MIGRATION_COOKIE_FLAG_CAPS, + QEMU_MIGRATION_COOKIE_FLAG_PRIVATE, QEMU_MIGRATION_COOKIE_FLAG_LAST } qemuMigrationCookieFlags; @@ -51,6 +52,7 @@ typedef enum { QEMU_MIGRATION_COOKIE_CPU = (1 << QEMU_MIGRATION_COOKIE_FLAG_CPU), QEMU_MIGRATION_COOKIE_ALLOW_REBOOT = (1 << QEMU_MIGRATION_COOKIE_FLAG_ALLOW_REBOOT), QEMU_MIGRATION_COOKIE_CAPS = (1 << QEMU_MIGRATION_COOKIE_FLAG_CAPS), + QEMU_MIGRATION_COOKIE_PRIVATE = (1 << QEMU_MIGRATION_COOKIE_FLAG_PRIVATE), } qemuMigrationCookieFeatures; typedef struct _qemuMigrationCookieGraphics qemuMigrationCookieGraphics; @@ -105,6 +107,13 @@ struct _qemuMigrationCookieCaps { virBitmapPtr automatic; }; + +typedef struct _qemuMigrationCookiePrivate qemuMigrationCookiePrivate; +typedef qemuMigrationCookiePrivate *qemuMigrationCookiePrivatePtr; +struct _qemuMigrationCookiePrivate { + virTristateBool forceNewNuma; +}; + typedef struct _qemuMigrationCookie qemuMigrationCookie; typedef qemuMigrationCookie *qemuMigrationCookiePtr; struct _qemuMigrationCookie { @@ -148,6 +157,9 @@ struct _qemuMigrationCookie { /* If flags & QEMU_MIGRATION_COOKIE_CAPS */ qemuMigrationCookieCapsPtr caps; + + /* If flags & QEMU_MIGRATION_COOKIE_PRIVATE */ + qemuMigrationCookiePrivatePtr priv; }; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index dee3f3fb63..c84f6e2d4d 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -6191,6 +6191,27 @@ qemuProcessPrepareAllowReboot(virDomainObjPtr vm) } +static void +qemuProcessPrepareForceNewNuma(virDomainObjPtr vm, + unsigned int flags) +{ + qemuDomainObjPrivatePtr priv = vm->privateData; + virQEMUCapsPtr qemuCaps = priv->qemuCaps; + + if (priv->forceNewNuma != VIR_TRISTATE_BOOL_ABSENT) + return; + + if (flags & VIR_QEMU_PROCESS_START_NEW && + (virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_RAM) || + virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_FILE) || + virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_MEMFD))) { + priv->forceNewNuma = VIR_TRISTATE_BOOL_YES; + } else { + priv->forceNewNuma = VIR_TRISTATE_BOOL_NO; + } +} + + /** * qemuProcessPrepareDomain: * @driver: qemu driver @@ -6244,6 +6265,7 @@ qemuProcessPrepareDomain(virQEMUDriverPtr driver, priv->rememberOwner = cfg->rememberOwner; qemuProcessPrepareAllowReboot(vm); + qemuProcessPrepareForceNewNuma(vm, flags); /* * Normally PCI addresses are assigned in the virDomainCreate @@ -7972,8 +7994,9 @@ qemuProcessReconnect(void *opaque) goto error; /* If we are connecting to a guest started by old libvirt there is no - * allowReboot in status XML and we need to initialize it. */ + * allowReboot or forceNewNuma in status XML and we need to initialize it. */ qemuProcessPrepareAllowReboot(obj); + qemuProcessPrepareForceNewNuma(obj, 0); if (qemuHostdevUpdateActiveDomainDevices(driver, obj->def) < 0) goto error; diff --git a/tests/qemustatusxml2xmldata/backup-pull-in.xml b/tests/qemustatusxml2xmldata/backup-pull-in.xml index 1db978a3ac..ce66c1e768 100644 --- a/tests/qemustatusxml2xmldata/backup-pull-in.xml +++ b/tests/qemustatusxml2xmldata/backup-pull-in.xml @@ -234,6 +234,7 @@ <chardevStdioLogd/> <allowReboot value='yes'/> <nodename index='0'/> + <forceNewNuma value='yes'/> <blockjobs active='yes'> <blockjob name='backup-vda-libvirt-3-format' type='backup' state='running' jobflags='0x0'> <disk dst='vda'/> diff --git a/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml b/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml index cc17a17ff4..6d9baee6c3 100644 --- a/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml +++ b/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml @@ -233,6 +233,7 @@ <chardevStdioLogd/> <allowReboot value='yes'/> <nodename index='0'/> + <forceNewNuma value='yes'/> <blockjobs active='yes'> <blockjob name='pull-vdb-libvirt-3-format' type='pull' state='running'> <disk dst='vdb'/> diff --git a/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml b/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml index bcef338fda..30f72c0747 100644 --- a/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml +++ b/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml @@ -22,6 +22,7 @@ <libDir path='/tmp'/> <channelTargetDir path='/tmp/channel'/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='yes'/> <agentTimeout>-2</agentTimeout> <domain type='qemu' id='1'> diff --git a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml index ce55a70637..0b7a69fc5e 100644 --- a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml +++ b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml @@ -257,6 +257,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-1-upstream'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <domain type='kvm' id='1'> <name>upstream</name> diff --git a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml index 5d3287606f..e0c8dd11ae 100644 --- a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml +++ b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml @@ -257,6 +257,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-1-upstream'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> <domain type='kvm' id='1'> diff --git a/tests/qemustatusxml2xmldata/migration-in-params-in.xml b/tests/qemustatusxml2xmldata/migration-in-params-in.xml index fdc2d39173..c16bb62543 100644 --- a/tests/qemustatusxml2xmldata/migration-in-params-in.xml +++ b/tests/qemustatusxml2xmldata/migration-in-params-in.xml @@ -256,6 +256,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-1-nest'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> <domain type='kvm' id='1'> diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml b/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml index 636accf054..dde2b0d966 100644 --- a/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml +++ b/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml @@ -259,6 +259,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-4-upstream'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <domain type='kvm' id='4'> <name>upstream</name> diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml b/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml index 304fb1b77f..9de5d71ba0 100644 --- a/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml +++ b/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml @@ -259,6 +259,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-4-upstream'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> <domain type='kvm' id='4'> diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml index 2cd6c9a5e9..0e7fc97aa8 100644 --- a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml +++ b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml @@ -286,6 +286,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-3-upstream'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <domain type='kvm' id='3'> <name>upstream</name> diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml index d69796e029..6baeaa3dec 100644 --- a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml +++ b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml @@ -288,6 +288,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-3-upstream'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> <domain type='kvm' id='3'> diff --git a/tests/qemustatusxml2xmldata/migration-out-params-in.xml b/tests/qemustatusxml2xmldata/migration-out-params-in.xml index 1956eac120..3536fd63d2 100644 --- a/tests/qemustatusxml2xmldata/migration-out-params-in.xml +++ b/tests/qemustatusxml2xmldata/migration-out-params-in.xml @@ -270,6 +270,7 @@ <channelTargetDir path='/var/lib/libvirt/qemu/channel/target/domain-7-nest'/> <chardevStdioLogd/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> <domain type='kvm' id='7'> diff --git a/tests/qemustatusxml2xmldata/modern-in.xml b/tests/qemustatusxml2xmldata/modern-in.xml index 64d42200e4..db8d9539b0 100644 --- a/tests/qemustatusxml2xmldata/modern-in.xml +++ b/tests/qemustatusxml2xmldata/modern-in.xml @@ -260,6 +260,7 @@ <chardevStdioLogd/> <allowReboot value='yes'/> <nodename index='123'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> <domain type='kvm' id='1'> diff --git a/tests/qemustatusxml2xmldata/vcpus-multi-in.xml b/tests/qemustatusxml2xmldata/vcpus-multi-in.xml index be48c55fe0..ca74d03012 100644 --- a/tests/qemustatusxml2xmldata/vcpus-multi-in.xml +++ b/tests/qemustatusxml2xmldata/vcpus-multi-in.xml @@ -308,6 +308,7 @@ <libDir path='/tmp'/> <channelTargetDir path='/tmp/channel'/> <allowReboot value='yes'/> + <forceNewNuma value='yes'/> <blockjobs active='no'/> <agentTimeout>-2</agentTimeout> <domain type='kvm' id='1729'> diff --git a/tests/qemuxml2argvdata/hugepages-numa-default-2M.args b/tests/qemuxml2argvdata/hugepages-numa-default-2M.args index bbf3c9f67a..bb64f3ea2a 100644 --- a/tests/qemuxml2argvdata/hugepages-numa-default-2M.args +++ b/tests/qemuxml2argvdata/hugepages-numa-default-2M.args @@ -14,10 +14,12 @@ QEMU_AUDIO_DRV=none \ -m 1024 \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ --mem-prealloc \ --mem-path /dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu \ --numa node,nodeid=0,cpus=0,mem=256 \ --numa node,nodeid=1,cpus=1,mem=768 \ +-object memory-backend-file,id=ram-node0,prealloc=yes,\ +mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,size=268435456 \ +-numa node,nodeid=0,cpus=0,memdev=ram-node0 \ +-object memory-backend-file,id=ram-node1,prealloc=yes,\ +mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,size=805306368 \ +-numa node,nodeid=1,cpus=1,memdev=ram-node1 \ -uuid ef1bdff4-27f3-4e85-a807-5fb4d58463cc \ -display none \ -no-user-config \ diff --git a/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args b/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args index e219914bc3..b20778418d 100644 --- a/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args +++ b/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args @@ -14,10 +14,10 @@ QEMU_AUDIO_DRV=none \ -m size=1048576k,slots=16,maxmem=1099511627776k \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ --mem-prealloc \ --mem-path /dev/hugepages2M/libvirt/qemu/-1-fedora \ --numa node,nodeid=0,cpus=0-1,mem=1024 \ --object memory-backend-file,id=memdimm0,\ +-object memory-backend-file,id=ram-node0,prealloc=yes,\ +mem-path=/dev/hugepages2M/libvirt/qemu/-1-fedora,size=1073741824 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ +-object memory-backend-file,id=memdimm0,prealloc=yes,\ mem-path=/dev/hugepages1G/libvirt/qemu/-1-fedora,size=1073741824,\ host-nodes=1-3,policy=bind \ -device pc-dimm,node=0,memdev=memdimm0,id=dimm0,slot=0 \ diff --git a/tests/qemuxml2argvdata/hugepages-numa-default.args b/tests/qemuxml2argvdata/hugepages-numa-default.args index 674bb517d1..95dcdebbef 100644 --- a/tests/qemuxml2argvdata/hugepages-numa-default.args +++ b/tests/qemuxml2argvdata/hugepages-numa-default.args @@ -14,9 +14,9 @@ QEMU_AUDIO_DRV=none \ -m 1024 \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ --mem-prealloc \ --mem-path /dev/hugepages2M/libvirt/qemu/-1-fedora \ --numa node,nodeid=0,cpus=0-1,mem=1024 \ +-object memory-backend-file,id=ram-node0,prealloc=yes,\ +mem-path=/dev/hugepages2M/libvirt/qemu/-1-fedora,size=1073741824 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -uuid 63840878-0deb-4095-97e6-fc444d9bc9fa \ -display none \ -no-user-config \ diff --git a/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args b/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args index 9ba3f8b7a0..1a93f0a9fd 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args +++ b/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args @@ -14,7 +14,8 @@ QEMU_AUDIO_DRV=none \ -m size=219136k,slots=16,maxmem=1099511627776k \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=214 \ +-object memory-backend-ram,id=ram-node0,size=224395264 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memdimm0,prealloc=yes,\ mem-path=/dev/hugepages2M/libvirt/qemu/-1-QEMUGuest1,size=536870912,\ host-nodes=1-3,policy=bind \ diff --git a/tests/qemuxml2argvdata/memory-hotplug-dimm.args b/tests/qemuxml2argvdata/memory-hotplug-dimm.args index 0ac48db399..ce39721fde 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-dimm.args +++ b/tests/qemuxml2argvdata/memory-hotplug-dimm.args @@ -14,7 +14,8 @@ QEMU_AUDIO_DRV=none \ -m size=219136k,slots=16,maxmem=1099511627776k \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=214 \ +-object memory-backend-ram,id=ram-node0,size=224395264 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-ram,id=memdimm0,size=536870912 \ -device pc-dimm,node=0,memdev=memdimm0,id=dimm0,slot=0 \ -object memory-backend-file,id=memdimm1,prealloc=yes,\ diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args index 89138f46c4..faa85355d3 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args +++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args @@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ -m size=219136k,slots=16,maxmem=1099511627776k \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=214 \ +-object memory-backend-ram,id=ram-node0,size=224395264 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\ share=no,size=536870912 \ -device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \ diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args index 1a8e7932dc..074d8f590b 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args +++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args @@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ -m size=219136k,slots=16,maxmem=1099511627776k \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=214 \ +-object memory-backend-ram,id=ram-node0,size=224395264 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\ share=no,size=536870912,align=2097152 \ -device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \ diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args index ef32c663de..522a5bd2fd 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args +++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args @@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ -m size=219136k,slots=16,maxmem=1099511627776k \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=214 \ +-object memory-backend-ram,id=ram-node0,size=224395264 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\ share=no,size=536870912 \ -device nvdimm,node=0,label-size=131072,memdev=memnvdimm0,id=nvdimm0,slot=0 \ diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args index 5dfba9b50a..b7dceaa3b5 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args +++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args @@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ -m size=219136k,slots=16,maxmem=1099511627776k \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=214 \ +-object memory-backend-ram,id=ram-node0,size=224395264 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\ share=no,size=536870912,pmem=on \ -device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \ diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args index eff80dcf80..9ba904e667 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args +++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args @@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ -m size=1048576k,slots=16,maxmem=1099511627776k \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=1024 \ +-object memory-backend-ram,id=ram-node0,size=1073741824 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\ size=537001984 \ -device nvdimm,node=0,label-size=131072,\ diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args index 7088a4f054..e070bd74f9 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args +++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args @@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ -m size=219136k,slots=16,maxmem=1099511627776k \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=214 \ +-object memory-backend-ram,id=ram-node0,size=224395264 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\ share=no,size=536870912 \ -device nvdimm,node=0,unarmed=on,memdev=memnvdimm0,id=nvdimm0,slot=0 \ diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args b/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args index 60d6d207c5..893360f9e6 100644 --- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args +++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args @@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \ -m size=1048576k,slots=16,maxmem=1099511627776k \ -overcommit mem-lock=off \ -smp 2,sockets=2,dies=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=1024 \ +-object memory-backend-ram,id=ram-node0,size=1073741824 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\ size=536870912 \ -device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \ diff --git a/tests/qemuxml2argvdata/numatune-auto-prefer.args b/tests/qemuxml2argvdata/numatune-auto-prefer.args index babb23b938..b60d77705d 100644 --- a/tests/qemuxml2argvdata/numatune-auto-prefer.args +++ b/tests/qemuxml2argvdata/numatune-auto-prefer.args @@ -14,7 +14,9 @@ QEMU_AUDIO_DRV=none \ -m 64 \ -realtime mlock=off \ -smp 1,sockets=1,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0,mem=64 \ +-object memory-backend-ram,id=ram-node0,size=67108864,host-nodes=0-3,\ +policy=preferred \ +-numa node,nodeid=0,cpus=0,memdev=ram-node0 \ -uuid 9f4b6512-e73a-4a25-93e8-5307802821ce \ -display none \ -no-user-config \ diff --git a/tests/qemuxml2argvdata/pages-dimm-discard.args b/tests/qemuxml2argvdata/pages-dimm-discard.args index 96e9ffdec3..4746ed6861 100644 --- a/tests/qemuxml2argvdata/pages-dimm-discard.args +++ b/tests/qemuxml2argvdata/pages-dimm-discard.args @@ -14,7 +14,8 @@ QEMU_AUDIO_DRV=none \ -m size=1048576k,slots=16,maxmem=1099511627776k \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0-1,mem=1024 \ +-object memory-backend-ram,id=ram-node0,size=1073741824 \ +-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \ -object memory-backend-file,id=memdimm0,prealloc=yes,\ mem-path=/dev/hugepages1G/libvirt/qemu/-1-fedora,size=1073741824,\ host-nodes=1-3,policy=bind \ diff --git a/tests/qemuxml2argvdata/pages-discard-hugepages.args b/tests/qemuxml2argvdata/pages-discard-hugepages.args index bbf3c9f67a..a549ee4030 100644 --- a/tests/qemuxml2argvdata/pages-discard-hugepages.args +++ b/tests/qemuxml2argvdata/pages-discard-hugepages.args @@ -14,10 +14,13 @@ QEMU_AUDIO_DRV=none \ -m 1024 \ -realtime mlock=off \ -smp 2,sockets=2,cores=1,threads=1 \ --mem-prealloc \ --mem-path /dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu \ --numa node,nodeid=0,cpus=0,mem=256 \ --numa node,nodeid=1,cpus=1,mem=768 \ +-object memory-backend-file,id=ram-node0,prealloc=yes,\ +mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,size=268435456 \ +-numa node,nodeid=0,cpus=0,memdev=ram-node0 \ +-object memory-backend-file,id=ram-node1,prealloc=yes,\ +mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,\ +discard-data=yes,size=805306368 \ +-numa node,nodeid=1,cpus=1,memdev=ram-node1 \ -uuid ef1bdff4-27f3-4e85-a807-5fb4d58463cc \ -display none \ -no-user-config \ diff --git a/tests/qemuxml2argvdata/pages-discard.args b/tests/qemuxml2argvdata/pages-discard.args index b2aed77ff9..2d7fc2a50c 100644 --- a/tests/qemuxml2argvdata/pages-discard.args +++ b/tests/qemuxml2argvdata/pages-discard.args @@ -14,10 +14,14 @@ QEMU_AUDIO_DRV=none \ -m 4096 \ -realtime mlock=off \ -smp 4,sockets=4,cores=1,threads=1 \ --numa node,nodeid=0,cpus=0,mem=1024 \ --numa node,nodeid=1,cpus=1,mem=1024 \ --numa node,nodeid=2,cpus=2,mem=1024 \ --numa node,nodeid=3,cpus=3,mem=1024 \ +-object memory-backend-ram,id=ram-node0,size=1073741824 \ +-numa node,nodeid=0,cpus=0,memdev=ram-node0 \ +-object memory-backend-ram,id=ram-node1,size=1073741824 \ +-numa node,nodeid=1,cpus=1,memdev=ram-node1 \ +-object memory-backend-ram,id=ram-node2,size=1073741824 \ +-numa node,nodeid=2,cpus=2,memdev=ram-node2 \ +-object memory-backend-ram,id=ram-node3,size=1073741824 \ +-numa node,nodeid=3,cpus=3,memdev=ram-node3 \ -uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \ -display none \ -no-user-config \ -- 2.26.2

On Tue, May 12, 2020 at 04:52:46PM +0200, Michal Privoznik wrote:
In v4.1.0-rc0~9^2~25 QEMU deprecated -numa mem= in favor of -numa memdev= + -object memory-backend-*. However, the problem is these two are incompatible. A domain started with older cmd line can't be migrated to the newer one and vice versa. That is why libvirt hasn't switched to the new cmd line, until now.
Starting with this commit, the new cmd line is used and this fact is then recorded in domain private data object under priv->forceNewNuma. This means, that the status XML and migration cookie have it too and thus can instruct the other daemon which cmd line to generate.
Unfortunately, migration from newer to older style will fail. I'm saving the explicit check for a separate commit.
AFAICT, this isn't the approach we had discussed previously with QEMU. QEMU introduced a flag on the machine type "numa-mem-supported". If libvirt sees "numa-mem-supported=false", then we must use the new syntax, otherwise we must keep using the old syntax for migration compatibility. Essentially this lets us implement if machine type version > XXX new syntax else old syntax without having to actually put a version number check into libvirt. QEMU has not actually set numa-mem-supported=false on any machine types upstream yet, becuase they're waiting for libvirt to get this wired up. The intent is that we can get some releases of libvirt into the wild which support numa-mem-supported=false. Then when QEMU flips the switch in new machine types we'll just "do the right thing". It will only break for people who use an old libvirt with a new QEMU which should be relatively uncommon. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|

As advertised in the previous commit, migration from -numa memdev= to -numa mem= is not supported and results in error. Fortunately, we can check whether the destination has used the style we told it to, or if the corresponding flag in the migration cookie is missing then we know we are talking to older daemon which would have used the old approach. Anyway, let's deny migration then. Signed-off-by: Michal Privoznik <mprivozn@redhat.com> --- src/qemu/qemu_migration.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 423713e00b..dde491b720 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2424,7 +2424,8 @@ qemuMigrationDstPrepareAny(virQEMUDriverPtr driver, cookieFlags = 0; } else { cookieFlags = QEMU_MIGRATION_COOKIE_GRAPHICS | - QEMU_MIGRATION_COOKIE_CAPS; + QEMU_MIGRATION_COOKIE_CAPS | + QEMU_MIGRATION_COOKIE_PRIVATE; } if (flags & VIR_MIGRATE_POSTCOPY && @@ -3549,10 +3550,18 @@ qemuMigrationSrcRun(virQEMUDriverPtr driver, cookiein, cookieinlen, cookieFlags | QEMU_MIGRATION_COOKIE_GRAPHICS | - QEMU_MIGRATION_COOKIE_CAPS); + QEMU_MIGRATION_COOKIE_CAPS | + QEMU_MIGRATION_COOKIE_PRIVATE); if (!mig) goto error; + if (priv->forceNewNuma == VIR_TRISTATE_BOOL_YES && + !(mig->priv && mig->priv->forceNewNuma == VIR_TRISTATE_BOOL_YES)) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s", + _("Migration to older numa unsupported")); + goto error; + } + if (qemuMigrationSrcGraphicsRelocate(driver, vm, mig, graphicsuri) < 0) VIR_WARN("unable to provide data for graphics client relocation"); -- 2.26.2

On 5/12/20 11:52 AM, Michal Privoznik wrote:
In a way, libvirt already uses -numa memdev= in a few cases. In fact, in as few cases as possible - only configurations which can not be configured with -numa mem=, because these two ways are incompatible when it comes to migration.
My approach to solve this is to have a privateData flag which tells which directs libvirt to generate old or new cmd line. And then this flag is saved into migration cookie so that the destination is directed the same way.
Problem with this approach is that, while migration 6.3.0 -> 6.4.0 -> 6.3.0 works, migration where the machine is started on newer libvirt 6.4.0 and then migrated back to 6.3.0 won't work (in fact is explicitly denied by 2/2) even though there is nothing visible that should prevent the migration.
Is there any advantage in using the new "memdev" format instead of the old one? An actual benefit inside QEMU? Or is it just an API change? If there is a benefit, I'd say we make the user choose to use the new memdev format, explicitly, in the NUMA XML. Or, if we want to default to use the new format, the user needs to set a "legacy" flag in the XML. This makes the migration compatibility an user problem,with clear benefits and drawbacks, and then the user can decide whether it is worth breaking migration for it. Eventually, when QEMU moves on, the 'legacy' option can be deprecated. If there is no benefit aside from "-numa mem=" stop working one day, I'd say to keep the legacy format as long as possible. When the QEMU binary stops supporting it, then we have no choice other than to stop using legacy format with the newer binaries. But the user will move on to the newer QEMU binaries across the env, so it's ok to not use the legacy format any longer. This approach also avoids dealing with migration issues in Libvirt side.
I am not sure whether we have a good move here, because even if we waited until QEMU removes the old way, it won't help us really. We would be just leaving the problem for future us.
I think the problem is simpler to handle in Libvirt when QEMU deprecates the old format entirely. Otherwise the user will have to also keep track of Libvirt versions to understand why the migration is failing here and there. Thanks, DHB
Michal Prívozník (2): qemu: Switch to new -numa memdev= qemu: Disallow migration to older -numa if newer is used
src/qemu/qemu_command.c | 20 ++--- src/qemu/qemu_domain.c | 38 +++++++++- src/qemu/qemu_domain.h | 3 + src/qemu/qemu_migration.c | 18 ++++- src/qemu/qemu_migration_cookie.c | 73 +++++++++++++++++++ src/qemu/qemu_migration_cookie.h | 12 +++ src/qemu/qemu_process.c | 25 ++++++- .../qemustatusxml2xmldata/backup-pull-in.xml | 1 + .../blockjob-blockdev-in.xml | 1 + .../blockjob-mirror-in.xml | 1 + .../disk-secinfo-upgrade-in.xml | 1 + .../disk-secinfo-upgrade-out.xml | 1 + .../migration-in-params-in.xml | 1 + .../migration-out-nbd-in.xml | 1 + .../migration-out-nbd-out.xml | 1 + .../migration-out-nbd-tls-in.xml | 1 + .../migration-out-nbd-tls-out.xml | 1 + .../migration-out-params-in.xml | 1 + tests/qemustatusxml2xmldata/modern-in.xml | 1 + .../qemustatusxml2xmldata/vcpus-multi-in.xml | 1 + .../hugepages-numa-default-2M.args | 10 ++- .../hugepages-numa-default-dimm.args | 8 +- .../hugepages-numa-default.args | 6 +- .../memory-hotplug-dimm-addr.args | 3 +- .../qemuxml2argvdata/memory-hotplug-dimm.args | 3 +- ...y-hotplug-nvdimm-access.x86_64-latest.args | 3 +- ...ry-hotplug-nvdimm-align.x86_64-latest.args | 3 +- ...ry-hotplug-nvdimm-label.x86_64-latest.args | 3 +- ...ory-hotplug-nvdimm-pmem.x86_64-latest.args | 3 +- ...ory-hotplug-nvdimm-ppc64.ppc64-latest.args | 3 +- ...hotplug-nvdimm-readonly.x86_64-latest.args | 3 +- .../memory-hotplug-nvdimm.x86_64-latest.args | 3 +- .../numatune-auto-prefer.args | 4 +- .../qemuxml2argvdata/pages-dimm-discard.args | 3 +- .../pages-discard-hugepages.args | 11 ++- tests/qemuxml2argvdata/pages-discard.args | 12 ++- 36 files changed, 236 insertions(+), 47 deletions(-)

On 5/13/20 4:48 PM, Daniel Henrique Barboza wrote:
On 5/12/20 11:52 AM, Michal Privoznik wrote:
In a way, libvirt already uses -numa memdev= in a few cases. In fact, in as few cases as possible - only configurations which can not be configured with -numa mem=, because these two ways are incompatible when it comes to migration.
My approach to solve this is to have a privateData flag which tells which directs libvirt to generate old or new cmd line. And then this flag is saved into migration cookie so that the destination is directed the same way.
Problem with this approach is that, while migration 6.3.0 -> 6.4.0 -> 6.3.0 works, migration where the machine is started on newer libvirt 6.4.0 and then migrated back to 6.3.0 won't work (in fact is explicitly denied by 2/2) even though there is nothing visible that should prevent the migration.
Is there any advantage in using the new "memdev" format instead of the old one? An actual benefit inside QEMU? Or is it just an API change?
I'm not aware of any real benefit. IIUC, qemu is trying to clean up its own code so they changed the API. The qemu commit I'm referencing in 1/2 mentions possible performance benefits of using the new API. Also, the commit mentions better control over memory allocation because we won't be using global -mem-prealloc and -mem-path but in libvirt, we would use corresponding attributes of memory-backend-* anyway. Therefore, from my POV it's only API change.
If there is a benefit, I'd say we make the user choose to use the new memdev format, explicitly, in the NUMA XML. Or, if we want to default to use the new format, the user needs to set a "legacy" flag in the XML. This makes the migration compatibility an user problem,with clear benefits and drawbacks, and then the user can decide whether it is worth breaking migration for it. Eventually, when QEMU moves on, the 'legacy' option can be deprecated.
Interesting idea. This is basically the same as making the private variable I'm inventing controllable by user. Let me check if there is a way to actually detect whether qemu supports only the old way, or only the new way or both. And it looks like it sort of can. I mean, there is numa-mem-supported attribute to a machine type in output of 'query-machines'. But there doesn't seem to be anything for -numa memdev=.
If there is no benefit aside from "-numa mem=" stop working one day, I'd say to keep the legacy format as long as possible. When the QEMU binary stops supporting it, then we have no choice other than to stop using legacy format with the newer binaries. But the user will move on to the newer QEMU binaries across the env, so it's ok to not use the legacy format any longer. This approach also avoids dealing with migration issues in Libvirt side.
Is that so? Qemu deprecated the old API and just for the sake of the argument, imagine they already removed it. Now, on libvirt level, how do we handle it? I mean, you are not able to migrate from older qemu to this hypothetical qemu. Sure, we can blame qemu, or we can start using the new API and some machines started with older qemu would be able to migrate to the hypothetical qemu. Yeah, we don't have a good option.
I am not sure whether we have a good move here, because even if we waited until QEMU removes the old way, it won't help us really. We would be just leaving the problem for future us.
I think the problem is simpler to handle in Libvirt when QEMU deprecates the old format entirely. Otherwise the user will have to also keep track of Libvirt versions to understand why the migration is failing here and there.
I think they will have to track both actually. They will have to track QEMU version, no argument there. But also Libvirt version - they minimal required version would have to be the one where Libvirt switches from old to the new API. Michal

On 5/13/20 12:32 PM, Michal Privoznik wrote:
On 5/13/20 4:48 PM, Daniel Henrique Barboza wrote:
On 5/12/20 11:52 AM, Michal Privoznik wrote:
[...]
I think the problem is simpler to handle in Libvirt when QEMU deprecates the old format entirely. Otherwise the user will have to also keep track of Libvirt versions to understand why the migration is failing here and there.
I think they will have to track both actually. They will have to track QEMU version, no argument there. But also Libvirt version - they minimal required version would have to be the one where Libvirt switches from old to the new API.
In that case, I believe that offering a XML knob to choose which NUMA mode the user wants (like I mentioned in the previous message) is a good way to mitigate the overall annoyance of the situation. It's simpler to add "mode='legacy'" in the domain XML than to update Libvirt. We can keep this mechanism option around until we're certain that (most) users moved on to newer QEMU/Libvirt versions. Granted, some guests will end up breaking when the option is deprecated anyway. At least this is a bit more of a "controlled explosion" scenario. Thanks, DHB
Michal

On Wed, May 13, 2020 at 01:36:05PM -0300, Daniel Henrique Barboza wrote:
On 5/13/20 12:32 PM, Michal Privoznik wrote:
On 5/13/20 4:48 PM, Daniel Henrique Barboza wrote:
On 5/12/20 11:52 AM, Michal Privoznik wrote:
[...]
I think the problem is simpler to handle in Libvirt when QEMU deprecates the old format entirely. Otherwise the user will have to also keep track of Libvirt versions to understand why the migration is failing here and there.
I think they will have to track both actually. They will have to track QEMU version, no argument there. But also Libvirt version - they minimal required version would have to be the one where Libvirt switches from old to the new API.
In that case, I believe that offering a XML knob to choose which NUMA mode the user wants (like I mentioned in the previous message) is a good way to mitigate the overall annoyance of the situation. It's simpler to add "mode='legacy'" in the domain XML than to update Libvirt.
This is something we really don't want todo, as this is supposed to be an internal impl detail. The fact that it had migration impact was accident. If we expose it as a knob, then you need to teach mgmt apps to set the knob which has a huge ripple effect through the consumers of libvirt. This is what the QEMU feature advertizement against machine type is supposed to avoid Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
participants (3)
-
Daniel Henrique Barboza
-
Daniel P. Berrangé
-
Michal Privoznik