9:52 a.m.
In a way, libvirt already uses -numa memdev= in a few cases. In fact, in
as few cases as possible - only configurations which can not be
configured with -numa mem=, because these two ways are incompatible when
it comes to migration.
My approach to solve this is to have a privateData flag which tells
which directs libvirt to generate old or new cmd line. And then this
flag is saved into migration cookie so that the destination is directed
the same way.
Problem with this approach is that, while migration 6.3.0 -> 6.4.0 ->
6.3.0 works, migration where the machine is started on newer libvirt
6.4.0 and then migrated back to 6.3.0 won't work (in fact is explicitly
denied by 2/2) even though there is nothing visible that should prevent
the migration.
I am not sure whether we have a good move here, because even if we
waited until QEMU removes the old way, it won't help us really. We
would be just leaving the problem for future us.
Michal Prívozník (2):
qemu: Switch to new -numa memdev=
qemu: Disallow migration to older -numa if newer is used
src/qemu/qemu_command.c | 20 ++---
src/qemu/qemu_domain.c | 38 +++++++++-
src/qemu/qemu_domain.h | 3 +
src/qemu/qemu_migration.c | 18 ++++-
src/qemu/qemu_migration_cookie.c | 73 +++++++++++++++++++
src/qemu/qemu_migration_cookie.h | 12 +++
src/qemu/qemu_process.c | 25 ++++++-
.../qemustatusxml2xmldata/backup-pull-in.xml | 1 +
.../blockjob-blockdev-in.xml | 1 +
.../blockjob-mirror-in.xml | 1 +
.../disk-secinfo-upgrade-in.xml | 1 +
.../disk-secinfo-upgrade-out.xml | 1 +
.../migration-in-params-in.xml | 1 +
.../migration-out-nbd-in.xml | 1 +
.../migration-out-nbd-out.xml | 1 +
.../migration-out-nbd-tls-in.xml | 1 +
.../migration-out-nbd-tls-out.xml | 1 +
.../migration-out-params-in.xml | 1 +
tests/qemustatusxml2xmldata/modern-in.xml | 1 +
.../qemustatusxml2xmldata/vcpus-multi-in.xml | 1 +
.../hugepages-numa-default-2M.args | 10 ++-
.../hugepages-numa-default-dimm.args | 8 +-
.../hugepages-numa-default.args | 6 +-
.../memory-hotplug-dimm-addr.args | 3 +-
.../qemuxml2argvdata/memory-hotplug-dimm.args | 3 +-
...y-hotplug-nvdimm-access.x86_64-latest.args | 3 +-
...ry-hotplug-nvdimm-align.x86_64-latest.args | 3 +-
...ry-hotplug-nvdimm-label.x86_64-latest.args | 3 +-
...ory-hotplug-nvdimm-pmem.x86_64-latest.args | 3 +-
...ory-hotplug-nvdimm-ppc64.ppc64-latest.args | 3 +-
...hotplug-nvdimm-readonly.x86_64-latest.args | 3 +-
.../memory-hotplug-nvdimm.x86_64-latest.args | 3 +-
.../numatune-auto-prefer.args | 4 +-
.../qemuxml2argvdata/pages-dimm-discard.args | 3 +-
.../pages-discard-hugepages.args | 11 ++-
tests/qemuxml2argvdata/pages-discard.args | 12 ++-
36 files changed, 236 insertions(+), 47 deletions(-)
--
2.26.2
9:52 a.m.
New subject: [PATCH 1/2] qemu: Switch to new -numa memdev=
In v4.1.0-rc0~9^2~25 QEMU deprecated -numa mem= in favor of -numa
memdev= + -object memory-backend-*. However, the problem is these
two are incompatible. A domain started with older cmd line can't
be migrated to the newer one and vice versa. That is why libvirt
hasn't switched to the new cmd line, until now.
Starting with this commit, the new cmd line is used and this fact
is then recorded in domain private data object under
priv->forceNewNuma. This means, that the status XML and migration
cookie have it too and thus can instruct the other daemon which
cmd line to generate.
Unfortunately, migration from newer to older style will fail.
I'm saving the explicit check for a separate commit.
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1783355
Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com>
---
src/qemu/qemu_command.c | 20 ++---
src/qemu/qemu_domain.c | 38 +++++++++-
src/qemu/qemu_domain.h | 3 +
src/qemu/qemu_migration.c | 5 +-
src/qemu/qemu_migration_cookie.c | 73 +++++++++++++++++++
src/qemu/qemu_migration_cookie.h | 12 +++
src/qemu/qemu_process.c | 25 ++++++-
.../qemustatusxml2xmldata/backup-pull-in.xml | 1 +
.../blockjob-blockdev-in.xml | 1 +
.../blockjob-mirror-in.xml | 1 +
.../disk-secinfo-upgrade-in.xml | 1 +
.../disk-secinfo-upgrade-out.xml | 1 +
.../migration-in-params-in.xml | 1 +
.../migration-out-nbd-in.xml | 1 +
.../migration-out-nbd-out.xml | 1 +
.../migration-out-nbd-tls-in.xml | 1 +
.../migration-out-nbd-tls-out.xml | 1 +
.../migration-out-params-in.xml | 1 +
tests/qemustatusxml2xmldata/modern-in.xml | 1 +
.../qemustatusxml2xmldata/vcpus-multi-in.xml | 1 +
.../hugepages-numa-default-2M.args | 10 ++-
.../hugepages-numa-default-dimm.args | 8 +-
.../hugepages-numa-default.args | 6 +-
.../memory-hotplug-dimm-addr.args | 3 +-
.../qemuxml2argvdata/memory-hotplug-dimm.args | 3 +-
...y-hotplug-nvdimm-access.x86_64-latest.args | 3 +-
...ry-hotplug-nvdimm-align.x86_64-latest.args | 3 +-
...ry-hotplug-nvdimm-label.x86_64-latest.args | 3 +-
...ory-hotplug-nvdimm-pmem.x86_64-latest.args | 3 +-
...ory-hotplug-nvdimm-ppc64.ppc64-latest.args | 3 +-
...hotplug-nvdimm-readonly.x86_64-latest.args | 3 +-
.../memory-hotplug-nvdimm.x86_64-latest.args | 3 +-
.../numatune-auto-prefer.args | 4 +-
.../qemuxml2argvdata/pages-dimm-discard.args | 3 +-
.../pages-discard-hugepages.args | 11 ++-
tests/qemuxml2argvdata/pages-discard.args | 12 ++-
36 files changed, 225 insertions(+), 45 deletions(-)
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c
index 2d8a6fb0dd..3316aae419 100644
--- a/src/qemu/qemu_command.c
+++ b/src/qemu/qemu_command.c
@@ -7028,8 +7028,6 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg,
char *tmpmask = NULL;
char *next = NULL;
virBufferPtr nodeBackends = NULL;
- bool needBackend = false;
- int rc;
int ret = -1;
size_t ncells = virDomainNumaGetNodeCount(def->numa);
@@ -7039,23 +7037,21 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg,
if (VIR_ALLOC_N(nodeBackends, ncells) < 0)
goto cleanup;
- /* using of -numa memdev= cannot be combined with -numa mem=, thus we
- * need to check which approach to use */
+ /* Using of -numa memdev= cannot be combined with -numa mem=.
+ * However, as of QEMU 4.1.0 using the latter is deprecated
+ * and the former is preferred. */
for (i = 0; i < ncells; i++) {
if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_RAM) ||
virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_FILE) ||
virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_MEMFD)) {
- if ((rc = qemuBuildMemoryCellBackendStr(def, cfg, i, priv,
- &nodeBackends[i])) < 0)
+ if (qemuBuildMemoryCellBackendStr(def, cfg, i, priv,
+ &nodeBackends[i]) < 0)
goto cleanup;
-
- if (rc == 0)
- needBackend = true;
}
}
- if (!needBackend &&
+ if (priv->forceNewNuma == VIR_TRISTATE_BOOL_NO &&
qemuBuildMemPathStr(def, cmd, priv) < 0)
goto cleanup;
@@ -7064,7 +7060,7 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg,
if (!(cpumask = virBitmapFormat(virDomainNumaGetNodeCpumask(def->numa, i))))
goto cleanup;
- if (needBackend) {
+ if (priv->forceNewNuma == VIR_TRISTATE_BOOL_YES) {
virCommandAddArg(cmd, "-object");
virCommandAddArgBuffer(cmd, &nodeBackends[i]);
}
@@ -7079,7 +7075,7 @@ qemuBuildNumaArgStr(virQEMUDriverConfigPtr cfg,
virBufferAdd(&buf, tmpmask, -1);
}
- if (needBackend)
+ if (priv->forceNewNuma == VIR_TRISTATE_BOOL_YES)
virBufferAsprintf(&buf, ",memdev=ram-node%zu", i);
else
virBufferAsprintf(&buf, ",mem=%llu",
diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c
index a1b250fd0b..4751685452 100644
--- a/src/qemu/qemu_domain.c
+++ b/src/qemu/qemu_domain.c
@@ -2283,6 +2283,7 @@ qemuDomainObjPrivateDataClear(qemuDomainObjPrivatePtr priv)
VIR_FREE(priv->channelTargetDir);
priv->memPrealloc = false;
+ priv->forceNewNuma = VIR_TRISTATE_BOOL_ABSENT;
/* remove automatic pinning data */
virBitmapFree(priv->autoNodeset);
@@ -2802,6 +2803,15 @@ qemuDomainObjPrivateXMLFormatAllowReboot(virBufferPtr buf,
}
+static void
+qemuDomainObjPrivateXMLFormatForceNewNuma(virBufferPtr buf,
+ virTristateBool forceNewNuma)
+{
+ virBufferAsprintf(buf, "<forceNewNuma value='%s'/>\n",
+ virTristateBoolTypeToString(forceNewNuma));
+}
+
+
static void
qemuDomainObjPrivateXMLFormatPR(virBufferPtr buf,
qemuDomainObjPrivatePtr priv)
@@ -3086,6 +3096,8 @@ qemuDomainObjPrivateXMLFormat(virBufferPtr buf,
if (priv->memPrealloc)
virBufferAddLit(buf, "<memPrealloc/>\n");
+ qemuDomainObjPrivateXMLFormatForceNewNuma(buf, priv->forceNewNuma);
+
if (qemuDomainObjPrivateXMLFormatBlockjobs(buf, vm) < 0)
return -1;
@@ -3552,6 +3564,26 @@ qemuDomainObjPrivateXMLParseAllowReboot(xmlXPathContextPtr ctxt,
}
+static int
+qemuDomainObjPrivateXMLParseForceNewNuma(xmlXPathContextPtr ctxt,
+ virTristateBool *forceNewNuma)
+{
+ int val;
+ g_autofree char *valStr = NULL;
+
+ if ((valStr = virXPathString("string(./forceNewNuma/@value)", ctxt))) {
+ if ((val = virTristateBoolTypeFromString(valStr)) < 0) {
+ virReportError(VIR_ERR_INTERNAL_ERROR,
+ _("invalid allowReboot value '%s'"),
valStr);
+ return -1;
+ }
+ *forceNewNuma = val;
+ }
+
+ return 0;
+}
+
+
static void
qemuDomainObjPrivateXMLParsePR(xmlXPathContextPtr ctxt,
bool *prDaemonRunning)
@@ -3936,7 +3968,8 @@ qemuDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt,
priv->chardevStdioLogd = virXPathBoolean("boolean(./chardevStdioLogd)",
ctxt) == 1;
- qemuDomainObjPrivateXMLParseAllowReboot(ctxt, &priv->allowReboot);
+ if (qemuDomainObjPrivateXMLParseAllowReboot(ctxt, &priv->allowReboot) < 0)
+ goto error;
qemuDomainObjPrivateXMLParsePR(ctxt, &priv->prDaemonRunning);
@@ -3956,6 +3989,9 @@ qemuDomainObjPrivateXMLParse(xmlXPathContextPtr ctxt,
priv->memPrealloc = virXPathBoolean("boolean(./memPrealloc)", ctxt) ==
1;
+ if (qemuDomainObjPrivateXMLParseForceNewNuma(ctxt, &priv->forceNewNuma) <
0)
+ goto error;
+
return 0;
error:
diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h
index 41d3f1561d..dafb1da52f 100644
--- a/src/qemu/qemu_domain.h
+++ b/src/qemu/qemu_domain.h
@@ -418,6 +418,9 @@ struct _qemuDomainObjPrivate {
/* true if global -mem-prealloc appears on cmd line */
bool memPrealloc;
+ /* true if new -numa memdev is forced */
+ virTristateBool forceNewNuma;
+
/* running block jobs */
virHashTablePtr blockjobs;
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index 02e8271e42..423713e00b 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -2140,6 +2140,7 @@ qemuMigrationSrcBeginPhase(virQEMUDriverPtr driver,
cookieFlags |= QEMU_MIGRATION_COOKIE_CPU;
cookieFlags |= QEMU_MIGRATION_COOKIE_ALLOW_REBOOT;
+ cookieFlags |= QEMU_MIGRATION_COOKIE_PRIVATE;
if (!(flags & VIR_MIGRATE_OFFLINE))
cookieFlags |= QEMU_MIGRATION_COOKIE_CAPS;
@@ -2501,7 +2502,8 @@ qemuMigrationDstPrepareAny(virQEMUDriverPtr driver,
QEMU_MIGRATION_COOKIE_CPU_HOTPLUG |
QEMU_MIGRATION_COOKIE_CPU |
QEMU_MIGRATION_COOKIE_ALLOW_REBOOT |
- QEMU_MIGRATION_COOKIE_CAPS)))
+ QEMU_MIGRATION_COOKIE_CAPS |
+ QEMU_MIGRATION_COOKIE_PRIVATE)))
goto cleanup;
if (!(vm = virDomainObjListAdd(driver->domains, *def,
@@ -2556,6 +2558,7 @@ qemuMigrationDstPrepareAny(virQEMUDriverPtr driver,
stopProcess = true;
priv->allowReboot = mig->allowReboot;
+ priv->forceNewNuma = mig->priv && mig->priv->forceNewNuma;
if (!(incoming = qemuMigrationDstPrepare(vm, tunnel, protocol,
listenAddress, port,
diff --git a/src/qemu/qemu_migration_cookie.c b/src/qemu/qemu_migration_cookie.c
index fb8b5bcd92..032d954dca 100644
--- a/src/qemu/qemu_migration_cookie.c
+++ b/src/qemu/qemu_migration_cookie.c
@@ -51,6 +51,7 @@ VIR_ENUM_IMPL(qemuMigrationCookieFlag,
"cpu",
"allowReboot",
"capabilities",
+ "private",
);
@@ -580,6 +581,20 @@ qemuMigrationCookieAddCaps(qemuMigrationCookiePtr mig,
}
+static void
+qemuMigrationCookieAddPrivate(qemuMigrationCookiePtr mig,
+ qemuDomainObjPrivatePtr priv)
+{
+ if (!mig->priv)
+ mig->priv = g_new0(qemuMigrationCookiePrivate, 1);
+
+ mig->priv->forceNewNuma = priv->forceNewNuma;
+
+ mig->flags |= QEMU_MIGRATION_COOKIE_PRIVATE;
+}
+
+
+
static void
qemuMigrationCookieGraphicsXMLFormat(virBufferPtr buf,
qemuMigrationCookieGraphicsPtr grap)
@@ -770,6 +785,25 @@ qemuMigrationCookieCapsXMLFormat(virBufferPtr buf,
}
+#define FORMAT_PRIV(member) \
+ virBufferAsprintf(buf, "<member name='" #member "'
value='%s'/>\n", \
+ virTristateBoolTypeToString(priv->member))
+
+static void
+qemuMigrationCookiePrivateXMLFormat(virBufferPtr buf,
+ qemuMigrationCookiePrivatePtr priv)
+{
+ virBufferAddLit(buf, "<privateData>\n");
+ virBufferAdjustIndent(buf, 2);
+
+ FORMAT_PRIV(forceNewNuma);
+
+ virBufferAdjustIndent(buf, -2);
+ virBufferAddLit(buf, "</privateData>\n");
+}
+
+#undef FORMAT_PRIV
+
static int
qemuMigrationCookieXMLFormat(virQEMUDriverPtr driver,
virQEMUCapsPtr qemuCaps,
@@ -858,6 +892,9 @@ qemuMigrationCookieXMLFormat(virQEMUDriverPtr driver,
if (mig->flags & QEMU_MIGRATION_COOKIE_CAPS)
qemuMigrationCookieCapsXMLFormat(buf, mig->caps);
+ if (mig->flags & QEMU_MIGRATION_COOKIE_PRIVATE)
+ qemuMigrationCookiePrivateXMLFormat(buf, mig->priv);
+
virBufferAdjustIndent(buf, -2);
virBufferAddLit(buf, "</qemu-migration>\n");
return 0;
@@ -1179,6 +1216,35 @@ qemuMigrationCookieCapsXMLParse(xmlXPathContextPtr ctxt)
}
+#define PARSE_PRIV(member) \
+ do { \
+ int val; \
+ g_autofree char *valStr = virXPathString("string(./privateData/member"
\
+ "[@name='" #member
"']/@value)", ctxt); \
+ if (valStr) { \
+ if ((val = virTristateBoolTypeFromString(valStr)) < 0) { \
+ virReportError(VIR_ERR_INTERNAL_ERROR, \
+ _("Malformed member '%s' value %s"), \
+ #member, valStr); \
+ return NULL; \
+ } \
+ priv->member = val; \
+ } \
+ } while (0)
+
+static qemuMigrationCookiePrivatePtr
+qemuMigrationCookiePrivateXMLParse(xmlXPathContextPtr ctxt)
+{
+ g_autofree qemuMigrationCookiePrivatePtr priv = NULL;
+
+ priv = g_new0(qemuMigrationCookiePrivate, 1);
+
+ PARSE_PRIV(forceNewNuma);
+
+ return g_steal_pointer(&priv);
+}
+
+
static int
qemuMigrationCookieXMLParse(qemuMigrationCookiePtr mig,
virQEMUDriverPtr driver,
@@ -1357,6 +1423,10 @@ qemuMigrationCookieXMLParse(qemuMigrationCookiePtr mig,
!(mig->caps = qemuMigrationCookieCapsXMLParse(ctxt)))
goto error;
+ if (flags & QEMU_MIGRATION_COOKIE_PRIVATE &&
+ !(mig->priv = qemuMigrationCookiePrivateXMLParse(ctxt)))
+ goto error;
+
return 0;
error:
@@ -1446,6 +1516,9 @@ qemuMigrationBakeCookie(qemuMigrationCookiePtr mig,
qemuMigrationCookieAddCaps(mig, dom, party) < 0)
return -1;
+ if (flags & QEMU_MIGRATION_COOKIE_PRIVATE)
+ qemuMigrationCookieAddPrivate(mig, priv);
+
if (!(*cookieout = qemuMigrationCookieXMLFormatStr(driver, priv->qemuCaps, mig)))
return -1;
diff --git a/src/qemu/qemu_migration_cookie.h b/src/qemu/qemu_migration_cookie.h
index 1e88684589..e1f44682a6 100644
--- a/src/qemu/qemu_migration_cookie.h
+++ b/src/qemu/qemu_migration_cookie.h
@@ -33,6 +33,7 @@ typedef enum {
QEMU_MIGRATION_COOKIE_FLAG_CPU,
QEMU_MIGRATION_COOKIE_FLAG_ALLOW_REBOOT,
QEMU_MIGRATION_COOKIE_FLAG_CAPS,
+ QEMU_MIGRATION_COOKIE_FLAG_PRIVATE,
QEMU_MIGRATION_COOKIE_FLAG_LAST
} qemuMigrationCookieFlags;
@@ -51,6 +52,7 @@ typedef enum {
QEMU_MIGRATION_COOKIE_CPU = (1 << QEMU_MIGRATION_COOKIE_FLAG_CPU),
QEMU_MIGRATION_COOKIE_ALLOW_REBOOT = (1 <<
QEMU_MIGRATION_COOKIE_FLAG_ALLOW_REBOOT),
QEMU_MIGRATION_COOKIE_CAPS = (1 << QEMU_MIGRATION_COOKIE_FLAG_CAPS),
+ QEMU_MIGRATION_COOKIE_PRIVATE = (1 << QEMU_MIGRATION_COOKIE_FLAG_PRIVATE),
} qemuMigrationCookieFeatures;
typedef struct _qemuMigrationCookieGraphics qemuMigrationCookieGraphics;
@@ -105,6 +107,13 @@ struct _qemuMigrationCookieCaps {
virBitmapPtr automatic;
};
+
+typedef struct _qemuMigrationCookiePrivate qemuMigrationCookiePrivate;
+typedef qemuMigrationCookiePrivate *qemuMigrationCookiePrivatePtr;
+struct _qemuMigrationCookiePrivate {
+ virTristateBool forceNewNuma;
+};
+
typedef struct _qemuMigrationCookie qemuMigrationCookie;
typedef qemuMigrationCookie *qemuMigrationCookiePtr;
struct _qemuMigrationCookie {
@@ -148,6 +157,9 @@ struct _qemuMigrationCookie {
/* If flags & QEMU_MIGRATION_COOKIE_CAPS */
qemuMigrationCookieCapsPtr caps;
+
+ /* If flags & QEMU_MIGRATION_COOKIE_PRIVATE */
+ qemuMigrationCookiePrivatePtr priv;
};
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index dee3f3fb63..c84f6e2d4d 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -6191,6 +6191,27 @@ qemuProcessPrepareAllowReboot(virDomainObjPtr vm)
}
+static void
+qemuProcessPrepareForceNewNuma(virDomainObjPtr vm,
+ unsigned int flags)
+{
+ qemuDomainObjPrivatePtr priv = vm->privateData;
+ virQEMUCapsPtr qemuCaps = priv->qemuCaps;
+
+ if (priv->forceNewNuma != VIR_TRISTATE_BOOL_ABSENT)
+ return;
+
+ if (flags & VIR_QEMU_PROCESS_START_NEW &&
+ (virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_RAM) ||
+ virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_FILE) ||
+ virQEMUCapsGet(qemuCaps, QEMU_CAPS_OBJECT_MEMORY_MEMFD))) {
+ priv->forceNewNuma = VIR_TRISTATE_BOOL_YES;
+ } else {
+ priv->forceNewNuma = VIR_TRISTATE_BOOL_NO;
+ }
+}
+
+
/**
* qemuProcessPrepareDomain:
* @driver: qemu driver
@@ -6244,6 +6265,7 @@ qemuProcessPrepareDomain(virQEMUDriverPtr driver,
priv->rememberOwner = cfg->rememberOwner;
qemuProcessPrepareAllowReboot(vm);
+ qemuProcessPrepareForceNewNuma(vm, flags);
/*
* Normally PCI addresses are assigned in the virDomainCreate
@@ -7972,8 +7994,9 @@ qemuProcessReconnect(void *opaque)
goto error;
/* If we are connecting to a guest started by old libvirt there is no
- * allowReboot in status XML and we need to initialize it. */
+ * allowReboot or forceNewNuma in status XML and we need to initialize it. */
qemuProcessPrepareAllowReboot(obj);
+ qemuProcessPrepareForceNewNuma(obj, 0);
if (qemuHostdevUpdateActiveDomainDevices(driver, obj->def) < 0)
goto error;
diff --git a/tests/qemustatusxml2xmldata/backup-pull-in.xml
b/tests/qemustatusxml2xmldata/backup-pull-in.xml
index 1db978a3ac..ce66c1e768 100644
--- a/tests/qemustatusxml2xmldata/backup-pull-in.xml
+++ b/tests/qemustatusxml2xmldata/backup-pull-in.xml
@@ -234,6 +234,7 @@
<chardevStdioLogd/>
<allowReboot value='yes'/>
<nodename index='0'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='yes'>
<blockjob name='backup-vda-libvirt-3-format' type='backup'
state='running' jobflags='0x0'>
<disk dst='vda'/>
diff --git a/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml
b/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml
index cc17a17ff4..6d9baee6c3 100644
--- a/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml
+++ b/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml
@@ -233,6 +233,7 @@
<chardevStdioLogd/>
<allowReboot value='yes'/>
<nodename index='0'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='yes'>
<blockjob name='pull-vdb-libvirt-3-format' type='pull'
state='running'>
<disk dst='vdb'/>
diff --git a/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml
b/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml
index bcef338fda..30f72c0747 100644
--- a/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml
+++ b/tests/qemustatusxml2xmldata/blockjob-mirror-in.xml
@@ -22,6 +22,7 @@
<libDir path='/tmp'/>
<channelTargetDir path='/tmp/channel'/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='yes'/>
<agentTimeout>-2</agentTimeout>
<domain type='qemu' id='1'>
diff --git a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml
b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml
index ce55a70637..0b7a69fc5e 100644
--- a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml
+++ b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-in.xml
@@ -257,6 +257,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-1-upstream'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<domain type='kvm' id='1'>
<name>upstream</name>
diff --git a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml
b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml
index 5d3287606f..e0c8dd11ae 100644
--- a/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml
+++ b/tests/qemustatusxml2xmldata/disk-secinfo-upgrade-out.xml
@@ -257,6 +257,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-1-upstream'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<agentTimeout>-2</agentTimeout>
<domain type='kvm' id='1'>
diff --git a/tests/qemustatusxml2xmldata/migration-in-params-in.xml
b/tests/qemustatusxml2xmldata/migration-in-params-in.xml
index fdc2d39173..c16bb62543 100644
--- a/tests/qemustatusxml2xmldata/migration-in-params-in.xml
+++ b/tests/qemustatusxml2xmldata/migration-in-params-in.xml
@@ -256,6 +256,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-1-nest'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<agentTimeout>-2</agentTimeout>
<domain type='kvm' id='1'>
diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml
b/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml
index 636accf054..dde2b0d966 100644
--- a/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml
+++ b/tests/qemustatusxml2xmldata/migration-out-nbd-in.xml
@@ -259,6 +259,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-4-upstream'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<domain type='kvm' id='4'>
<name>upstream</name>
diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml
b/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml
index 304fb1b77f..9de5d71ba0 100644
--- a/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml
+++ b/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml
@@ -259,6 +259,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-4-upstream'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<agentTimeout>-2</agentTimeout>
<domain type='kvm' id='4'>
diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml
b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml
index 2cd6c9a5e9..0e7fc97aa8 100644
--- a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml
+++ b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-in.xml
@@ -286,6 +286,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-3-upstream'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<domain type='kvm' id='3'>
<name>upstream</name>
diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml
b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml
index d69796e029..6baeaa3dec 100644
--- a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml
+++ b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml
@@ -288,6 +288,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-3-upstream'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<agentTimeout>-2</agentTimeout>
<domain type='kvm' id='3'>
diff --git a/tests/qemustatusxml2xmldata/migration-out-params-in.xml
b/tests/qemustatusxml2xmldata/migration-out-params-in.xml
index 1956eac120..3536fd63d2 100644
--- a/tests/qemustatusxml2xmldata/migration-out-params-in.xml
+++ b/tests/qemustatusxml2xmldata/migration-out-params-in.xml
@@ -270,6 +270,7 @@
<channelTargetDir
path='/var/lib/libvirt/qemu/channel/target/domain-7-nest'/>
<chardevStdioLogd/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<agentTimeout>-2</agentTimeout>
<domain type='kvm' id='7'>
diff --git a/tests/qemustatusxml2xmldata/modern-in.xml
b/tests/qemustatusxml2xmldata/modern-in.xml
index 64d42200e4..db8d9539b0 100644
--- a/tests/qemustatusxml2xmldata/modern-in.xml
+++ b/tests/qemustatusxml2xmldata/modern-in.xml
@@ -260,6 +260,7 @@
<chardevStdioLogd/>
<allowReboot value='yes'/>
<nodename index='123'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<agentTimeout>-2</agentTimeout>
<domain type='kvm' id='1'>
diff --git a/tests/qemustatusxml2xmldata/vcpus-multi-in.xml
b/tests/qemustatusxml2xmldata/vcpus-multi-in.xml
index be48c55fe0..ca74d03012 100644
--- a/tests/qemustatusxml2xmldata/vcpus-multi-in.xml
+++ b/tests/qemustatusxml2xmldata/vcpus-multi-in.xml
@@ -308,6 +308,7 @@
<libDir path='/tmp'/>
<channelTargetDir path='/tmp/channel'/>
<allowReboot value='yes'/>
+ <forceNewNuma value='yes'/>
<blockjobs active='no'/>
<agentTimeout>-2</agentTimeout>
<domain type='kvm' id='1729'>
diff --git a/tests/qemuxml2argvdata/hugepages-numa-default-2M.args
b/tests/qemuxml2argvdata/hugepages-numa-default-2M.args
index bbf3c9f67a..bb64f3ea2a 100644
--- a/tests/qemuxml2argvdata/hugepages-numa-default-2M.args
+++ b/tests/qemuxml2argvdata/hugepages-numa-default-2M.args
@@ -14,10 +14,12 @@ QEMU_AUDIO_DRV=none \
-m 1024 \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
--mem-prealloc \
--mem-path /dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu \
--numa node,nodeid=0,cpus=0,mem=256 \
--numa node,nodeid=1,cpus=1,mem=768 \
+-object memory-backend-file,id=ram-node0,prealloc=yes,\
+mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,size=268435456 \
+-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
+-object memory-backend-file,id=ram-node1,prealloc=yes,\
+mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,size=805306368 \
+-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-uuid ef1bdff4-27f3-4e85-a807-5fb4d58463cc \
-display none \
-no-user-config \
diff --git a/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args
b/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args
index e219914bc3..b20778418d 100644
--- a/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args
+++ b/tests/qemuxml2argvdata/hugepages-numa-default-dimm.args
@@ -14,10 +14,10 @@ QEMU_AUDIO_DRV=none \
-m size=1048576k,slots=16,maxmem=1099511627776k \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
--mem-prealloc \
--mem-path /dev/hugepages2M/libvirt/qemu/-1-fedora \
--numa node,nodeid=0,cpus=0-1,mem=1024 \
--object memory-backend-file,id=memdimm0,\
+-object memory-backend-file,id=ram-node0,prealloc=yes,\
+mem-path=/dev/hugepages2M/libvirt/qemu/-1-fedora,size=1073741824 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
+-object memory-backend-file,id=memdimm0,prealloc=yes,\
mem-path=/dev/hugepages1G/libvirt/qemu/-1-fedora,size=1073741824,\
host-nodes=1-3,policy=bind \
-device pc-dimm,node=0,memdev=memdimm0,id=dimm0,slot=0 \
diff --git a/tests/qemuxml2argvdata/hugepages-numa-default.args
b/tests/qemuxml2argvdata/hugepages-numa-default.args
index 674bb517d1..95dcdebbef 100644
--- a/tests/qemuxml2argvdata/hugepages-numa-default.args
+++ b/tests/qemuxml2argvdata/hugepages-numa-default.args
@@ -14,9 +14,9 @@ QEMU_AUDIO_DRV=none \
-m 1024 \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
--mem-prealloc \
--mem-path /dev/hugepages2M/libvirt/qemu/-1-fedora \
--numa node,nodeid=0,cpus=0-1,mem=1024 \
+-object memory-backend-file,id=ram-node0,prealloc=yes,\
+mem-path=/dev/hugepages2M/libvirt/qemu/-1-fedora,size=1073741824 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-uuid 63840878-0deb-4095-97e6-fc444d9bc9fa \
-display none \
-no-user-config \
diff --git a/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args
b/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args
index 9ba3f8b7a0..1a93f0a9fd 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-dimm-addr.args
@@ -14,7 +14,8 @@ QEMU_AUDIO_DRV=none \
-m size=219136k,slots=16,maxmem=1099511627776k \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=214 \
+-object memory-backend-ram,id=ram-node0,size=224395264 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memdimm0,prealloc=yes,\
mem-path=/dev/hugepages2M/libvirt/qemu/-1-QEMUGuest1,size=536870912,\
host-nodes=1-3,policy=bind \
diff --git a/tests/qemuxml2argvdata/memory-hotplug-dimm.args
b/tests/qemuxml2argvdata/memory-hotplug-dimm.args
index 0ac48db399..ce39721fde 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-dimm.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-dimm.args
@@ -14,7 +14,8 @@ QEMU_AUDIO_DRV=none \
-m size=219136k,slots=16,maxmem=1099511627776k \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=214 \
+-object memory-backend-ram,id=ram-node0,size=224395264 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-ram,id=memdimm0,size=536870912 \
-device pc-dimm,node=0,memdev=memdimm0,id=dimm0,slot=0 \
-object memory-backend-file,id=memdimm1,prealloc=yes,\
diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args
b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args
index 89138f46c4..faa85355d3 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-access.x86_64-latest.args
@@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \
-m size=219136k,slots=16,maxmem=1099511627776k \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=214 \
+-object memory-backend-ram,id=ram-node0,size=224395264 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\
share=no,size=536870912 \
-device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \
diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args
b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args
index 1a8e7932dc..074d8f590b 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-align.x86_64-latest.args
@@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \
-m size=219136k,slots=16,maxmem=1099511627776k \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=214 \
+-object memory-backend-ram,id=ram-node0,size=224395264 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\
share=no,size=536870912,align=2097152 \
-device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \
diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args
b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args
index ef32c663de..522a5bd2fd 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-label.x86_64-latest.args
@@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \
-m size=219136k,slots=16,maxmem=1099511627776k \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=214 \
+-object memory-backend-ram,id=ram-node0,size=224395264 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\
share=no,size=536870912 \
-device nvdimm,node=0,label-size=131072,memdev=memnvdimm0,id=nvdimm0,slot=0 \
diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args
b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args
index 5dfba9b50a..b7dceaa3b5 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-pmem.x86_64-latest.args
@@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \
-m size=219136k,slots=16,maxmem=1099511627776k \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=214 \
+-object memory-backend-ram,id=ram-node0,size=224395264 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\
share=no,size=536870912,pmem=on \
-device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \
diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args
b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args
index eff80dcf80..9ba904e667 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-ppc64.ppc64-latest.args
@@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \
-m size=1048576k,slots=16,maxmem=1099511627776k \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=1024 \
+-object memory-backend-ram,id=ram-node0,size=1073741824 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\
size=537001984 \
-device nvdimm,node=0,label-size=131072,\
diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args
b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args
index 7088a4f054..e070bd74f9 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm-readonly.x86_64-latest.args
@@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \
-m size=219136k,slots=16,maxmem=1099511627776k \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=214 \
+-object memory-backend-ram,id=ram-node0,size=224395264 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\
share=no,size=536870912 \
-device nvdimm,node=0,unarmed=on,memdev=memnvdimm0,id=nvdimm0,slot=0 \
diff --git a/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args
b/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args
index 60d6d207c5..893360f9e6 100644
--- a/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args
+++ b/tests/qemuxml2argvdata/memory-hotplug-nvdimm.x86_64-latest.args
@@ -17,7 +17,8 @@ file=/tmp/lib/domain--1-QEMUGuest1/master-key.aes \
-m size=1048576k,slots=16,maxmem=1099511627776k \
-overcommit mem-lock=off \
-smp 2,sockets=2,dies=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=1024 \
+-object memory-backend-ram,id=ram-node0,size=1073741824 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memnvdimm0,prealloc=yes,mem-path=/tmp/nvdimm,\
size=536870912 \
-device nvdimm,node=0,memdev=memnvdimm0,id=nvdimm0,slot=0 \
diff --git a/tests/qemuxml2argvdata/numatune-auto-prefer.args
b/tests/qemuxml2argvdata/numatune-auto-prefer.args
index babb23b938..b60d77705d 100644
--- a/tests/qemuxml2argvdata/numatune-auto-prefer.args
+++ b/tests/qemuxml2argvdata/numatune-auto-prefer.args
@@ -14,7 +14,9 @@ QEMU_AUDIO_DRV=none \
-m 64 \
-realtime mlock=off \
-smp 1,sockets=1,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0,mem=64 \
+-object memory-backend-ram,id=ram-node0,size=67108864,host-nodes=0-3,\
+policy=preferred \
+-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
-uuid 9f4b6512-e73a-4a25-93e8-5307802821ce \
-display none \
-no-user-config \
diff --git a/tests/qemuxml2argvdata/pages-dimm-discard.args
b/tests/qemuxml2argvdata/pages-dimm-discard.args
index 96e9ffdec3..4746ed6861 100644
--- a/tests/qemuxml2argvdata/pages-dimm-discard.args
+++ b/tests/qemuxml2argvdata/pages-dimm-discard.args
@@ -14,7 +14,8 @@ QEMU_AUDIO_DRV=none \
-m size=1048576k,slots=16,maxmem=1099511627776k \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0-1,mem=1024 \
+-object memory-backend-ram,id=ram-node0,size=1073741824 \
+-numa node,nodeid=0,cpus=0-1,memdev=ram-node0 \
-object memory-backend-file,id=memdimm0,prealloc=yes,\
mem-path=/dev/hugepages1G/libvirt/qemu/-1-fedora,size=1073741824,\
host-nodes=1-3,policy=bind \
diff --git a/tests/qemuxml2argvdata/pages-discard-hugepages.args
b/tests/qemuxml2argvdata/pages-discard-hugepages.args
index bbf3c9f67a..a549ee4030 100644
--- a/tests/qemuxml2argvdata/pages-discard-hugepages.args
+++ b/tests/qemuxml2argvdata/pages-discard-hugepages.args
@@ -14,10 +14,13 @@ QEMU_AUDIO_DRV=none \
-m 1024 \
-realtime mlock=off \
-smp 2,sockets=2,cores=1,threads=1 \
--mem-prealloc \
--mem-path /dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu \
--numa node,nodeid=0,cpus=0,mem=256 \
--numa node,nodeid=1,cpus=1,mem=768 \
+-object memory-backend-file,id=ram-node0,prealloc=yes,\
+mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,size=268435456 \
+-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
+-object memory-backend-file,id=ram-node1,prealloc=yes,\
+mem-path=/dev/hugepages2M/libvirt/qemu/-1-SomeDummyHugepagesGu,\
+discard-data=yes,size=805306368 \
+-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
-uuid ef1bdff4-27f3-4e85-a807-5fb4d58463cc \
-display none \
-no-user-config \
diff --git a/tests/qemuxml2argvdata/pages-discard.args
b/tests/qemuxml2argvdata/pages-discard.args
index b2aed77ff9..2d7fc2a50c 100644
--- a/tests/qemuxml2argvdata/pages-discard.args
+++ b/tests/qemuxml2argvdata/pages-discard.args
@@ -14,10 +14,14 @@ QEMU_AUDIO_DRV=none \
-m 4096 \
-realtime mlock=off \
-smp 4,sockets=4,cores=1,threads=1 \
--numa node,nodeid=0,cpus=0,mem=1024 \
--numa node,nodeid=1,cpus=1,mem=1024 \
--numa node,nodeid=2,cpus=2,mem=1024 \
--numa node,nodeid=3,cpus=3,mem=1024 \
+-object memory-backend-ram,id=ram-node0,size=1073741824 \
+-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
+-object memory-backend-ram,id=ram-node1,size=1073741824 \
+-numa node,nodeid=1,cpus=1,memdev=ram-node1 \
+-object memory-backend-ram,id=ram-node2,size=1073741824 \
+-numa node,nodeid=2,cpus=2,memdev=ram-node2 \
+-object memory-backend-ram,id=ram-node3,size=1073741824 \
+-numa node,nodeid=3,cpus=3,memdev=ram-node3 \
-uuid c7a5fdbd-edaf-9455-926a-d65c16db1809 \
-display none \
-no-user-config \
--
2.26.2
10:44 a.m.
New subject: [PATCH 1/2] qemu: Switch to new -numa memdev=
On Tue, May 12, 2020 at 04:52:46PM +0200, Michal Privoznik wrote:
In v4.1.0-rc0~9^2~25 QEMU deprecated -numa mem= in favor of -numa
memdev= + -object memory-backend-*. However, the problem is these
two are incompatible. A domain started with older cmd line can't
be migrated to the newer one and vice versa. That is why libvirt
hasn't switched to the new cmd line, until now.
Starting with this commit, the new cmd line is used and this fact
is then recorded in domain private data object under
priv->forceNewNuma. This means, that the status XML and migration
cookie have it too and thus can instruct the other daemon which
cmd line to generate.
Unfortunately, migration from newer to older style will fail.
I'm saving the explicit check for a separate commit.
AFAICT, this isn't the approach we had discussed previously with
QEMU.
QEMU introduced a flag on the machine type "numa-mem-supported".
If libvirt sees "numa-mem-supported=false", then we must use
the new syntax, otherwise we must keep using the old syntax
for migration compatibility.
Essentially this lets us implement
if machine type version > XXX
new syntax
else
old syntax
without having to actually put a version number check into
libvirt.
QEMU has not actually set numa-mem-supported=false on any
machine types upstream yet, becuase they're waiting for libvirt
to get this wired up.
The intent is that we can get some releases of libvirt into the
wild which support numa-mem-supported=false. Then when QEMU
flips the switch in new machine types we'll just "do the right
thing".
It will only break for people who use an old libvirt with a new
QEMU which should be relatively uncommon.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
9:52 a.m.
New subject: [PATCH 2/2] qemu: Disallow migration to older -numa if newer is used
As advertised in the previous commit, migration from -numa
memdev= to -numa mem= is not supported and results in error.
Fortunately, we can check whether the destination has used the
style we told it to, or if the corresponding flag in the
migration cookie is missing then we know we are talking to older
daemon which would have used the old approach. Anyway, let's deny
migration then.
Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com>
---
src/qemu/qemu_migration.c | 13 +++++++++++--
1 file changed, 11 insertions(+), 2 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index 423713e00b..dde491b720 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -2424,7 +2424,8 @@ qemuMigrationDstPrepareAny(virQEMUDriverPtr driver,
cookieFlags = 0;
} else {
cookieFlags = QEMU_MIGRATION_COOKIE_GRAPHICS |
- QEMU_MIGRATION_COOKIE_CAPS;
+ QEMU_MIGRATION_COOKIE_CAPS |
+ QEMU_MIGRATION_COOKIE_PRIVATE;
}
if (flags & VIR_MIGRATE_POSTCOPY &&
@@ -3549,10 +3550,18 @@ qemuMigrationSrcRun(virQEMUDriverPtr driver,
cookiein, cookieinlen,
cookieFlags |
QEMU_MIGRATION_COOKIE_GRAPHICS |
- QEMU_MIGRATION_COOKIE_CAPS);
+ QEMU_MIGRATION_COOKIE_CAPS |
+ QEMU_MIGRATION_COOKIE_PRIVATE);
if (!mig)
goto error;
+ if (priv->forceNewNuma == VIR_TRISTATE_BOOL_YES &&
+ !(mig->priv && mig->priv->forceNewNuma ==
VIR_TRISTATE_BOOL_YES)) {
+ virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s",
+ _("Migration to older numa unsupported"));
+ goto error;
+ }
+
if (qemuMigrationSrcGraphicsRelocate(driver, vm, mig, graphicsuri) < 0)
VIR_WARN("unable to provide data for graphics client relocation");
--
2.26.2
9:48 a.m.
On 5/12/20 11:52 AM, Michal Privoznik wrote:
In a way, libvirt already uses -numa memdev= in a few cases. In fact,
in
as few cases as possible - only configurations which can not be
configured with -numa mem=, because these two ways are incompatible when
it comes to migration.
My approach to solve this is to have a privateData flag which tells
which directs libvirt to generate old or new cmd line. And then this
flag is saved into migration cookie so that the destination is directed
the same way.
Problem with this approach is that, while migration 6.3.0 -> 6.4.0 ->
6.3.0 works, migration where the machine is started on newer libvirt
6.4.0 and then migrated back to 6.3.0 won't work (in fact is explicitly
denied by 2/2) even though there is nothing visible that should prevent
the migration.
Is there any advantage in using the new "memdev" format instead of the old
one? An actual benefit inside QEMU? Or is it just an API change?
If there is a benefit, I'd say we make the user choose to use the new
memdev format, explicitly, in the NUMA XML. Or, if we want to default to
use the new format, the user needs to set a "legacy" flag in the XML.
This makes the migration compatibility an user problem,with clear benefits and
drawbacks, and then the user can decide whether it is worth breaking migration
for it. Eventually, when QEMU moves on, the 'legacy' option can be deprecated.
If there is no benefit aside from "-numa mem=" stop working one day, I'd
say
to keep the legacy format as long as possible. When the QEMU binary stops
supporting it, then we have no choice other than to stop using legacy format
with the newer binaries. But the user will move on to the newer QEMU binaries
across the env, so it's ok to not use the legacy format any longer. This approach
also avoids dealing with migration issues in Libvirt side.
I am not sure whether we have a good move here, because even if we
waited until QEMU removes the old way, it won't help us really. We
would be just leaving the problem for future us.
I think the problem is simpler to handle in Libvirt when QEMU deprecates the old
format entirely. Otherwise the user will have to also keep track of Libvirt versions
to understand why the migration is failing here and there.
Thanks,
DHB
Michal Prívozník (2):
qemu: Switch to new -numa memdev=
qemu: Disallow migration to older -numa if newer is used
src/qemu/qemu_command.c | 20 ++---
src/qemu/qemu_domain.c | 38 +++++++++-
src/qemu/qemu_domain.h | 3 +
src/qemu/qemu_migration.c | 18 ++++-
src/qemu/qemu_migration_cookie.c | 73 +++++++++++++++++++
src/qemu/qemu_migration_cookie.h | 12 +++
src/qemu/qemu_process.c | 25 ++++++-
.../qemustatusxml2xmldata/backup-pull-in.xml | 1 +
.../blockjob-blockdev-in.xml | 1 +
.../blockjob-mirror-in.xml | 1 +
.../disk-secinfo-upgrade-in.xml | 1 +
.../disk-secinfo-upgrade-out.xml | 1 +
.../migration-in-params-in.xml | 1 +
.../migration-out-nbd-in.xml | 1 +
.../migration-out-nbd-out.xml | 1 +
.../migration-out-nbd-tls-in.xml | 1 +
.../migration-out-nbd-tls-out.xml | 1 +
.../migration-out-params-in.xml | 1 +
tests/qemustatusxml2xmldata/modern-in.xml | 1 +
.../qemustatusxml2xmldata/vcpus-multi-in.xml | 1 +
.../hugepages-numa-default-2M.args | 10 ++-
.../hugepages-numa-default-dimm.args | 8 +-
.../hugepages-numa-default.args | 6 +-
.../memory-hotplug-dimm-addr.args | 3 +-
.../qemuxml2argvdata/memory-hotplug-dimm.args | 3 +-
...y-hotplug-nvdimm-access.x86_64-latest.args | 3 +-
...ry-hotplug-nvdimm-align.x86_64-latest.args | 3 +-
...ry-hotplug-nvdimm-label.x86_64-latest.args | 3 +-
...ory-hotplug-nvdimm-pmem.x86_64-latest.args | 3 +-
...ory-hotplug-nvdimm-ppc64.ppc64-latest.args | 3 +-
...hotplug-nvdimm-readonly.x86_64-latest.args | 3 +-
.../memory-hotplug-nvdimm.x86_64-latest.args | 3 +-
.../numatune-auto-prefer.args | 4 +-
.../qemuxml2argvdata/pages-dimm-discard.args | 3 +-
.../pages-discard-hugepages.args | 11 ++-
tests/qemuxml2argvdata/pages-discard.args | 12 ++-
36 files changed, 236 insertions(+), 47 deletions(-)
10:32 a.m.
On 5/13/20 4:48 PM, Daniel Henrique Barboza wrote:
On 5/12/20 11:52 AM, Michal Privoznik wrote:
> In a way, libvirt already uses -numa memdev= in a few cases. In fact, in
> as few cases as possible - only configurations which can not be
> configured with -numa mem=, because these two ways are incompatible when
> it comes to migration.
>
> My approach to solve this is to have a privateData flag which tells
> which directs libvirt to generate old or new cmd line. And then this
> flag is saved into migration cookie so that the destination is directed
> the same way.
>
> Problem with this approach is that, while migration 6.3.0 -> 6.4.0 ->
> 6.3.0 works, migration where the machine is started on newer libvirt
> 6.4.0 and then migrated back to 6.3.0 won't work (in fact is explicitly
> denied by 2/2) even though there is nothing visible that should prevent
> the migration.
Is there any advantage in using the new "memdev" format instead of the old
one? An actual benefit inside QEMU? Or is it just an API change?
I'm not aware of any real benefit. IIUC, qemu is trying to clean up its
own code so they changed the API. The qemu commit I'm referencing in 1/2
mentions possible performance benefits of using the new API. Also, the
commit mentions better control over memory allocation because we won't
be using global -mem-prealloc and -mem-path but in libvirt, we would use
corresponding attributes of memory-backend-* anyway. Therefore, from my
POV it's only API change.
If there is a benefit, I'd say we make the user choose to use the new
memdev format, explicitly, in the NUMA XML. Or, if we want to default to
use the new format, the user needs to set a "legacy" flag in the XML.
This makes the migration compatibility an user problem,with clear
benefits and
drawbacks, and then the user can decide whether it is worth breaking
migration
for it. Eventually, when QEMU moves on, the 'legacy' option can be
deprecated.
Interesting idea. This is basically the same as making the private
variable I'm inventing controllable by user. Let me check if there is a
way to actually detect whether qemu supports only the old way, or only
the new way or both.
And it looks like it sort of can. I mean, there is numa-mem-supported
attribute to a machine type in output of 'query-machines'. But there
doesn't seem to be anything for -numa memdev=.
If there is no benefit aside from "-numa mem=" stop working
one day, I'd
say
to keep the legacy format as long as possible. When the QEMU binary stops
supporting it, then we have no choice other than to stop using legacy
format
with the newer binaries. But the user will move on to the newer QEMU
binaries
across the env, so it's ok to not use the legacy format any longer. This
approach
also avoids dealing with migration issues in Libvirt side.
Is that so? Qemu deprecated the old API and just for the sake of the
argument, imagine they already removed it. Now, on libvirt level, how do
we handle it? I mean, you are not able to migrate from older qemu to
this hypothetical qemu. Sure, we can blame qemu, or we can start using
the new API and some machines started with older qemu would be able to
migrate to the hypothetical qemu.
Yeah, we don't have a good option.
>
> I am not sure whether we have a good move here, because even if we
> waited until QEMU removes the old way, it won't help us really. We
> would be just leaving the problem for future us.
I think the problem is simpler to handle in Libvirt when QEMU deprecates
the old
format entirely. Otherwise the user will have to also keep track of
Libvirt versions
to understand why the migration is failing here and there.
I think they will have to track both actually. They will have to track
QEMU version, no argument there. But also Libvirt version - they minimal
required version would have to be the one where Libvirt switches from
old to the new API.
Michal
11:36 a.m.
On 5/13/20 12:32 PM, Michal Privoznik wrote:
On 5/13/20 4:48 PM, Daniel Henrique Barboza wrote:
>
>
> On 5/12/20 11:52 AM, Michal Privoznik wrote:
[...]
> I think the problem is simpler to handle in Libvirt when QEMU
deprecates the old
> format entirely. Otherwise the user will have to also keep track of Libvirt versions
> to understand why the migration is failing here and there.
I think they will have to track both actually. They will have to track QEMU version, no
argument there. But also Libvirt version - they minimal required version would have to be
the one where Libvirt switches from old to the new API.
In that case, I believe that offering a XML knob to choose which NUMA mode
the user wants (like I mentioned in the previous message) is a good way to
mitigate the overall annoyance of the situation. It's simpler to add
"mode='legacy'"
in the domain XML than to update Libvirt.
We can keep this mechanism option around until we're certain that (most) users
moved on to newer QEMU/Libvirt versions. Granted, some guests will end up
breaking when the option is deprecated anyway. At least this is a bit more of a
"controlled explosion" scenario.
Thanks,
DHB
Michal
11:53 a.m.
On Wed, May 13, 2020 at 01:36:05PM -0300, Daniel Henrique Barboza wrote:
On 5/13/20 12:32 PM, Michal Privoznik wrote:
> On 5/13/20 4:48 PM, Daniel Henrique Barboza wrote:
> >
> >
> > On 5/12/20 11:52 AM, Michal Privoznik wrote:
[...]
> > I think the problem is simpler to handle in Libvirt when QEMU deprecates the
old
> > format entirely. Otherwise the user will have to also keep track of Libvirt
versions
> > to understand why the migration is failing here and there.
>
> I think they will have to track both actually. They will have to track QEMU version,
no argument there. But also Libvirt version - they minimal required version would have to
be the one where Libvirt switches from old to the new API.
In that case, I believe that offering a XML knob to choose which NUMA mode
the user wants (like I mentioned in the previous message) is a good way to
mitigate the overall annoyance of the situation. It's simpler to add
"mode='legacy'"
in the domain XML than to update Libvirt.
This is something we really don't want todo, as this is supposed to be an
internal impl detail. The fact that it had migration impact was accident.
If we expose it as a knob, then you need to teach mgmt apps to set the
knob which has a huge ripple effect through the consumers of libvirt.
This is what the QEMU feature advertizement against machine type is
supposed to avoid
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1685
days inactive
1686
days old
7 comments
3 participants
participants (3)
-
Daniel Henrique Barboza -
Daniel P. Berrangé -
Michal Privoznik