1:38 a.m.
Before this patch set, numatune only has three memory modes:
static, interleave and prefered. These memory policies are
ultimately set by mbind() system call.
Memory policy could be 'hard coded' into the kernel, but none of
above policies fit our requirment under this case. mbind() support
default memory policy, but it requires a NULL nodemask. So obviously
setting allowed memory nodes is cgroups' mission under this case.
So we introduce a new option for mode in numatune named 'restrictive'.
<numatune>
<memory mode="restrictive" nodeset="1-4,^3"/>
<memnode cellid="0" mode="restrictive"
nodeset="1"/>
<memnode cellid="2" mode="restrictive"
nodeset="2"/>
</numatune>
The config above means we only use cgroups to restrict the allowed
memory nodes and not setting any specific memory policies explicitly.
For this new "restrictive" mode, there is a concrete use case about a
new feature in kernel but not merged yet, we call it memory tiering.
(https://lwn.net/Articles/802544/).
If memory tiering is enabled on host, DRAM is top tier memory, and
PMEM(persistent memory) is second tier memory, PMEM is shown as numa node
without cpu. Pages can be migrated between DRAM node and PMEM node based on
DRAM pressure and how cold/hot they are. *this memory policy* is implemented
in kernel. So we need a default mode here, but from libvirt's perspective,
the "defaut" mode is "strict", it's not MPOL_DEFAULT
(https://man7.org/linux/man-pages/man2/mbind.2.html) defined in kernel.
And to make memory tiering works well, cgroups setting is necessary, since
it restricts that the pages can only be migrated between the DRAM and PMEM
nodes that we specified (NUMA affinity support).
Just using cgroups with multiple nodes in the nodeset makes kernel decide
on which node (out of those in the restricted set) to allocate on, but specifying
"strict" basically allocates it sequentially (on the first one until it is
full,
then on the next one and so on).
In a word, if a user requires default mode(MPOL_DEFAULT), that means they want
kernel decide the memory allocation and also want the cgroups to restrict memory
nodes, "restrictive" mode will be useful.
BR,
Luyao
Luyao Zhong (3):
docs: add docs for 'restrictive' option for mode in numatune
schema: add 'restrictive' config option for mode in numatune
qemu: add parser and formatter for 'restrictive' mode in numatune
docs/formatdomain.rst | 7 +++-
docs/schemas/domaincommon.rng | 2 +
include/libvirt/libvirt-domain.h | 1 +
src/conf/numa_conf.c | 9 ++++
src/qemu/qemu_command.c | 6 ++-
src/qemu/qemu_process.c | 27 ++++++++++++
src/util/virnuma.c | 3 ++
.../numatune-memnode-invalid-mode.err | 1 +
.../numatune-memnode-invalid-mode.xml | 33 +++++++++++++++
...emnode-restrictive-mode.x86_64-latest.args | 38 +++++++++++++++++
.../numatune-memnode-restrictive-mode.xml | 33 +++++++++++++++
tests/qemuxml2argvtest.c | 2 +
...memnode-restrictive-mode.x86_64-latest.xml | 41 +++++++++++++++++++
tests/qemuxml2xmltest.c | 1 +
14 files changed, 201 insertions(+), 3 deletions(-)
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
create mode 100644
tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
create mode 100644
tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
--
2.25.4
1:38 a.m.
New subject: [libvirt][PATCH v5 1/3] docs: add docs for 'restrictive' option for mode in numatune
When user would like use cgroups to restrict the allowed memory
nodes, and require not setting any specific memory policy, then
'restrictive' mode is useful.
Reviewed-by: Daniel Henrique Barboza <danielhb413(a)gmail.com>
Signed-off-by: Luyao Zhong <luyao.zhong(a)intel.com>
---
docs/formatdomain.rst | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst
index 29d2e02da1..f32fcf03d5 100644
--- a/docs/formatdomain.rst
+++ b/docs/formatdomain.rst
@@ -1112,8 +1112,11 @@ NUMA Node Tuning
``memory``
The optional ``memory`` element specifies how to allocate memory for the
domain process on a NUMA host. It contains several optional attributes.
- Attribute ``mode`` is either 'interleave', 'strict', or
'preferred', defaults
- to 'strict'. Attribute ``nodeset`` specifies the NUMA nodes, using the same
+ Attribute ``mode`` is either 'interleave', 'strict',
'preferred' or
+ 'restrictive', defaults to 'strict'. The value 'restrictive'
specifies
+ using system default policy and only cgroups is used to restrict the
+ memory nodes, and it requires setting mode to 'restrictive' in ``memnode``
+ elements. Attribute ``nodeset`` specifies the NUMA nodes, using the same
syntax as attribute ``cpuset`` of element ``vcpu``. Attribute ``placement`` (
:since:`since 0.9.12` ) can be used to indicate the memory placement mode for
domain process, its value can be either "static" or "auto",
defaults to
--
2.25.4
1:38 a.m.
New subject: [libvirt][PATCH v5 2/3] schema: add 'restrictive' config option for mode in numatune
support 'restrictive' mode in memory element and memnode
element in numatune:
<domain>
...
<numatune>
<memory mode="restrictive" nodeset="1-4,^3"/>
<memnode cellid="0" mode="restrictive"
nodeset="1"/>
<memnode cellid="2" mode="restrictive"
nodeset="2"/>
</numatune>
...
</domain>
Reviewed-by: Daniel Henrique Barboza <danielhb413(a)gmail.com>
Signed-off-by: Luyao Zhong <luyao.zhong(a)intel.com>
---
docs/schemas/domaincommon.rng | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng
index 99cd873832..20a43148b1 100644
--- a/docs/schemas/domaincommon.rng
+++ b/docs/schemas/domaincommon.rng
@@ -1107,6 +1107,7 @@
<value>strict</value>
<value>preferred</value>
<value>interleave</value>
+ <value>restrictive</value>
</choice>
</attribute>
</optional>
@@ -1139,6 +1140,7 @@
<value>strict</value>
<value>preferred</value>
<value>interleave</value>
+ <value>restrictive</value>
</choice>
</attribute>
<attribute name="nodeset">
--
2.25.4
1:38 a.m.
New subject: [libvirt][PATCH v5 3/3] qemu: add parser and formatter for 'restrictive' mode in numatune
Reviewed-by: Daniel Henrique Barboza <danielhb413(a)gmail.com>
Signed-off-by: Luyao Zhong <luyao.zhong(a)intel.com>
---
include/libvirt/libvirt-domain.h | 1 +
src/conf/numa_conf.c | 9 ++++
src/qemu/qemu_command.c | 6 ++-
src/qemu/qemu_process.c | 27 ++++++++++++
src/util/virnuma.c | 3 ++
.../numatune-memnode-invalid-mode.err | 1 +
.../numatune-memnode-invalid-mode.xml | 33 +++++++++++++++
...emnode-restrictive-mode.x86_64-latest.args | 38 +++++++++++++++++
.../numatune-memnode-restrictive-mode.xml | 33 +++++++++++++++
tests/qemuxml2argvtest.c | 2 +
...memnode-restrictive-mode.x86_64-latest.xml | 41 +++++++++++++++++++
tests/qemuxml2xmltest.c | 1 +
12 files changed, 194 insertions(+), 1 deletion(-)
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
create mode 100644
tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
create mode 100644
tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index 03c119fe26..e99bfb7654 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -1527,6 +1527,7 @@ typedef enum {
VIR_DOMAIN_NUMATUNE_MEM_STRICT = 0,
VIR_DOMAIN_NUMATUNE_MEM_PREFERRED = 1,
VIR_DOMAIN_NUMATUNE_MEM_INTERLEAVE = 2,
+ VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE = 3,
# ifdef VIR_ENUM_SENTINELS
VIR_DOMAIN_NUMATUNE_MEM_LAST /* This constant is subject to change */
diff --git a/src/conf/numa_conf.c b/src/conf/numa_conf.c
index 64b93fd7d1..11093531b5 100644
--- a/src/conf/numa_conf.c
+++ b/src/conf/numa_conf.c
@@ -43,6 +43,7 @@ VIR_ENUM_IMPL(virDomainNumatuneMemMode,
"strict",
"preferred",
"interleave",
+ "restrictive",
);
VIR_ENUM_IMPL(virDomainNumatunePlacement,
@@ -234,6 +235,14 @@ virDomainNumatuneNodeParseXML(virDomainNumaPtr numa,
_("Invalid mode attribute in memnode
element"));
goto cleanup;
}
+
+ if (numa->memory.mode == VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE &&
+ mode != VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) {
+ virReportError(VIR_ERR_XML_ERROR, "%s",
+ _("'restrictive' mode is required in memnode
element "
+ "when mode is 'restrictive' in memory
element"));
+ goto cleanup;
+ }
VIR_FREE(tmp);
mem_node->mode = mode;
}
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c
index 45eb0dc976..a6a1717115 100644
--- a/src/qemu/qemu_command.c
+++ b/src/qemu/qemu_command.c
@@ -175,6 +175,7 @@ VIR_ENUM_IMPL(qemuNumaPolicy,
"bind",
"preferred",
"interleave",
+ "restricted",
);
VIR_ENUM_DECL(qemuAudioDriver);
@@ -3300,7 +3301,10 @@ qemuBuildMemoryBackendProps(virJSONValuePtr *backendProps,
return -1;
}
- if (nodemask) {
+ /* If mode is "restrictive", we should only use cgroups setting allowed
memory
+ * nodes, and skip passing the host-nodes and policy parameters to QEMU command
+ * line which means we will use system default memory policy. */
+ if (nodemask && mode != VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) {
if (!virNumaNodesetIsAvailable(nodemask))
return -1;
if (virJSONValueObjectAdd(props,
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index b74afedd07..c887cb35f0 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -2692,6 +2692,7 @@ qemuProcessSetupPid(virDomainObjPtr vm,
g_autoptr(virBitmap) hostcpumap = NULL;
g_autofree char *mem_mask = NULL;
int ret = -1;
+ size_t i;
if ((period || quota) &&
!virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPU)) {
@@ -2732,6 +2733,32 @@ qemuProcessSetupPid(virDomainObjPtr vm,
&mem_mask, -1) < 0)
goto cleanup;
+ /* For vCPU threads, mem_mask is different among cells and mem_mask
+ * is used to set cgroups cpuset.mems for vcpu threads. If we specify
+ * 'restrictive' mode, that means we will set system default memory
+ * policy and only use cgroups to restrict allowed memory nodes. */
+ if (nameval == VIR_CGROUP_THREAD_VCPU) {
+ virDomainNumaPtr numatune = vm->def->numa;
+ virBitmapPtr numanode_cpumask = NULL;
+ for (i = 0; i < virDomainNumaGetNodeCount(numatune); i++) {
+ numanode_cpumask = virDomainNumaGetNodeCpumask(numatune, i);
+ /* 'i' indicates the cell id, if the vCPU id is in this cell
+ * and mode is 'restrictive', we need get the corresponding
+ * nodeset. */
+ if (virBitmapIsBitSet(numanode_cpumask, id) &&
+ virDomainNumatuneGetMode(numatune, i, &mem_mode) == 0 &&
+ mem_mode == VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) {
+ if (virDomainNumatuneMaybeFormatNodeset(numatune,
+ priv->autoNodeset,
+ &mem_mask, i) < 0) {
+ goto cleanup;
+ } else {
+ break;
+ }
+ }
+ }
+ }
+
if (virCgroupNewThread(priv->cgroup, nameval, id, true, &cgroup) < 0)
goto cleanup;
diff --git a/src/util/virnuma.c b/src/util/virnuma.c
index 6c194b54d1..34db746d28 100644
--- a/src/util/virnuma.c
+++ b/src/util/virnuma.c
@@ -152,6 +152,9 @@ virNumaSetupMemoryPolicy(virDomainNumatuneMemMode mode,
numa_set_interleave_mask(&mask);
break;
+ case VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE:
+ break;
+
case VIR_DOMAIN_NUMATUNE_MEM_LAST:
break;
}
diff --git a/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
new file mode 100644
index 0000000000..180e64d1d8
--- /dev/null
+++ b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
@@ -0,0 +1 @@
+XML error: 'restrictive' mode is required in memnode element when mode is
'restrictive' in memory element
diff --git a/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
new file mode 100644
index 0000000000..a7c18d4d50
--- /dev/null
+++ b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
@@ -0,0 +1,33 @@
+<domain type='qemu'>
+ <name>QEMUGuest</name>
+ <uuid>9f4b6512-e73a-4a25-93e8-5307802821ce</uuid>
+ <memory unit='KiB'>24682468</memory>
+ <currentMemory unit='KiB'>24682468</currentMemory>
+ <vcpu placement='static'>32</vcpu>
+ <numatune>
+ <memory mode='restrictive' nodeset='0-7'/>
+ <memnode cellid='0' mode='restrictive' nodeset='3'/>
+ <memnode cellid='2' mode='strict'
nodeset='1-2,5-7,^6'/>
+ </numatune>
+ <os>
+ <type arch='x86_64' machine='pc'>hvm</type>
+ <boot dev='hd'/>
+ </os>
+ <cpu>
+ <numa>
+ <cell id='0' cpus='0' memory='20002'
unit='KiB'/>
+ <cell id='1' cpus='1-27,29' memory='660066'
unit='KiB'/>
+ <cell id='2' cpus='28,30-31' memory='24002400'
unit='KiB'/>
+ </numa>
+ </cpu>
+ <clock offset='utc'/>
+ <on_poweroff>destroy</on_poweroff>
+ <on_reboot>restart</on_reboot>
+ <on_crash>destroy</on_crash>
+ <devices>
+ <emulator>/usr/bin/qemu-system-x86_64</emulator>
+ <controller type='usb' index='0'/>
+ <controller type='pci' index='0' model='pci-root'/>
+ <memballoon model='virtio'/>
+ </devices>
+</domain>
diff --git a/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
new file mode 100644
index 0000000000..6d5e2eb76e
--- /dev/null
+++ b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
@@ -0,0 +1,38 @@
+LC_ALL=C \
+PATH=/bin \
+HOME=/tmp/lib/domain--1-QEMUGuest \
+USER=test \
+LOGNAME=test \
+XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest/.local/share \
+XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest/.cache \
+XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest/.config \
+/usr/bin/qemu-system-x86_64 \
+-name guest=QEMUGuest,debug-threads=on \
+-S \
+-object
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/tmp/lib/domain--1-QEMUGuest/master-key.aes"}'
\
+-machine pc,accel=tcg,usb=off,dump-guest-core=off \
+-cpu qemu64 \
+-m 24105 \
+-overcommit mem-lock=off \
+-smp 32,sockets=32,cores=1,threads=1 \
+-object
'{"qom-type":"memory-backend-ram","id":"ram-node0","size":20971520}'
\
+-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
+-object
'{"qom-type":"memory-backend-ram","id":"ram-node1","size":676331520}'
\
+-numa node,nodeid=1,cpus=1-27,cpus=29,memdev=ram-node1 \
+-object
'{"qom-type":"memory-backend-ram","id":"ram-node2","size":24578621440}'
\
+-numa node,nodeid=2,cpus=28,cpus=30-31,memdev=ram-node2 \
+-uuid 9f4b6512-e73a-4a25-93e8-5307802821ce \
+-display none \
+-no-user-config \
+-nodefaults \
+-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \
+-mon chardev=charmonitor,id=monitor,mode=control \
+-rtc base=utc \
+-no-shutdown \
+-no-acpi \
+-boot strict=on \
+-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
+-audiodev id=audio1,driver=none \
+-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 \
+-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
+-msg timestamp=on
diff --git a/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
new file mode 100644
index 0000000000..72949b0657
--- /dev/null
+++ b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
@@ -0,0 +1,33 @@
+<domain type='qemu'>
+ <name>QEMUGuest</name>
+ <uuid>9f4b6512-e73a-4a25-93e8-5307802821ce</uuid>
+ <memory unit='KiB'>24682468</memory>
+ <currentMemory unit='KiB'>24682468</currentMemory>
+ <vcpu placement='static'>32</vcpu>
+ <numatune>
+ <memnode cellid='0' mode='restrictive' nodeset='3'/>
+ <memory mode='restrictive' nodeset='0-7'/>
+ <memnode cellid='2' mode='restrictive'
nodeset='1-2,5-7,^6'/>
+ </numatune>
+ <os>
+ <type arch='x86_64' machine='pc'>hvm</type>
+ <boot dev='hd'/>
+ </os>
+ <cpu>
+ <numa>
+ <cell id='0' cpus='0' memory='20002'
unit='KiB'/>
+ <cell id='1' cpus='1-27,29' memory='660066'
unit='KiB'/>
+ <cell id='2' cpus='28,30-31' memory='24002400'
unit='KiB'/>
+ </numa>
+ </cpu>
+ <clock offset='utc'/>
+ <on_poweroff>destroy</on_poweroff>
+ <on_reboot>restart</on_reboot>
+ <on_crash>destroy</on_crash>
+ <devices>
+ <emulator>/usr/bin/qemu-system-x86_64</emulator>
+ <controller type='usb' index='0'/>
+ <controller type='pci' index='0' model='pci-root'/>
+ <memballoon model='virtio'/>
+ </devices>
+</domain>
diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c
index 317ee3267d..b6d2e8323d 100644
--- a/tests/qemuxml2argvtest.c
+++ b/tests/qemuxml2argvtest.c
@@ -2171,6 +2171,8 @@ mymain(void)
DO_TEST_PARSE_ERROR("numatune-memnode", NONE);
DO_TEST_CAPS_VER("numatune-memnode", "5.2.0");
DO_TEST_CAPS_LATEST("numatune-memnode");
+ DO_TEST_CAPS_LATEST("numatune-memnode-restrictive-mode");
+ DO_TEST_PARSE_ERROR("numatune-memnode-invalid-mode", NONE);
DO_TEST("numatune-memnode-no-memory",
QEMU_CAPS_NUMA,
diff --git a/tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
b/tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
new file mode 100644
index 0000000000..012c526460
--- /dev/null
+++ b/tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
@@ -0,0 +1,41 @@
+<domain type='qemu'>
+ <name>QEMUGuest</name>
+ <uuid>9f4b6512-e73a-4a25-93e8-5307802821ce</uuid>
+ <memory unit='KiB'>24682468</memory>
+ <currentMemory unit='KiB'>24682468</currentMemory>
+ <vcpu placement='static'>32</vcpu>
+ <numatune>
+ <memory mode='restrictive' nodeset='0-7'/>
+ <memnode cellid='0' mode='restrictive' nodeset='3'/>
+ <memnode cellid='2' mode='restrictive'
nodeset='1-2,5,7'/>
+ </numatune>
+ <os>
+ <type arch='x86_64' machine='pc'>hvm</type>
+ <boot dev='hd'/>
+ </os>
+ <cpu mode='custom' match='exact' check='none'>
+ <model fallback='forbid'>qemu64</model>
+ <numa>
+ <cell id='0' cpus='0' memory='20002'
unit='KiB'/>
+ <cell id='1' cpus='1-27,29' memory='660066'
unit='KiB'/>
+ <cell id='2' cpus='28,30-31' memory='24002400'
unit='KiB'/>
+ </numa>
+ </cpu>
+ <clock offset='utc'/>
+ <on_poweroff>destroy</on_poweroff>
+ <on_reboot>restart</on_reboot>
+ <on_crash>destroy</on_crash>
+ <devices>
+ <emulator>/usr/bin/qemu-system-x86_64</emulator>
+ <controller type='usb' index='0' model='piix3-uhci'>
+ <address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x2'/>
+ </controller>
+ <controller type='pci' index='0' model='pci-root'/>
+ <input type='mouse' bus='ps2'/>
+ <input type='keyboard' bus='ps2'/>
+ <audio id='1' type='none'/>
+ <memballoon model='virtio'>
+ <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0'/>
+ </memballoon>
+ </devices>
+</domain>
diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c
index a5343cd6f8..32b68fcfb0 100644
--- a/tests/qemuxml2xmltest.c
+++ b/tests/qemuxml2xmltest.c
@@ -1125,6 +1125,7 @@ mymain(void)
DO_TEST("numatune-distances", QEMU_CAPS_NUMA, QEMU_CAPS_NUMA_DIST);
DO_TEST("numatune-no-vcpu", QEMU_CAPS_NUMA);
DO_TEST("numatune-hmat", QEMU_CAPS_NUMA_HMAT,
QEMU_CAPS_OBJECT_MEMORY_RAM);
+ DO_TEST_CAPS_LATEST("numatune-memnode-restrictive-mode");
DO_TEST("bios-nvram", NONE);
DO_TEST("bios-nvram-os-interleave", NONE);
--
2.25.4
1:41 a.m.
New subject: [libvirt][PATCH v5 3/3] qemu: add parser and formatter for 'restrictive' mode in numatune
@luoyao.zhong
I tried to test this patch, but met some compile errors. Could you take a
look at them?
#gcc --version
gcc (GCC) 10.2.1 20201125 (Red Hat 10.2.1-9)
#git describe
v7.2.0-194-ga0491637e1
../src/qemu/qemu_process.c: In function ‘qemuProcessSetupPid’:
../src/qemu/qemu_process.c:2741:13: error: unknown type name
‘virDomainNumaPtr’; did you mean ‘virDomainNuma’?
2741 | virDomainNumaPtr numatune = vm->def->numa;
| ^~~~~~~~~~~~~~~~
| virDomainNuma
../src/qemu/qemu_process.c:2741:41: error: initialization of ‘int’ from
‘virDomainNuma *’ {aka ‘struct _virDomainNuma *’} makes integer from
pointer without a cast [-Werror=int-conversion]
2741 | virDomainNumaPtr numatune = vm->def->numa;
| ^~
../src/qemu/qemu_process.c:2742:13: error: unknown type name
‘virBitmapPtr’; did you mean ‘virBitmap’?
2742 | virBitmapPtr numanode_cpumask = NULL;
| ^~~~~~~~~~~~
| virBitmap
../src/qemu/qemu_process.c:2742:45: error: initialization of ‘int’ from
‘void *’ makes integer from pointer without a cast [-Werror=int-conversion]
2742 | virBitmapPtr numanode_cpumask = NULL;
| ^~~~
../src/qemu/qemu_process.c:2743:55: error: passing argument 1 of
‘virDomainNumaGetNodeCount’ makes pointer from integer without a cast
[-Werror=int-conversion]
2743 | for (i = 0; i < virDomainNumaGetNodeCount(numatune);
i++) {
| ^~~~~~~~
| |
| int
In file included from ../src/conf/cpu_conf.h:27,
from ../src/conf/capabilities.h:27,
from ../src/qemu/qemu_conf.h:28,
from ../src/qemu/qemu_process.h:24,
from ../src/qemu/qemu_process.c:41:
../src/conf/numa_conf.h:117:49: note: expected ‘virDomainNuma *’ {aka
‘struct _virDomainNuma *’} but argument is of type ‘int’
117 | size_t virDomainNumaGetNodeCount(virDomainNuma *numa);
| ~~~~~~~~~~~~~~~^~~~
../src/qemu/qemu_process.c:2744:64: error: passing argument 1 of
‘virDomainNumaGetNodeCpumask’ makes pointer from integer without a cast
[-Werror=int-conversion]
2744 | numanode_cpumask =
virDomainNumaGetNodeCpumask(numatune, i);
|
^~~~~~~~
| |
| int
In file included from ../src/conf/cpu_conf.h:27,
from ../src/conf/capabilities.h:27,
from ../src/qemu/qemu_conf.h:28,
from ../src/qemu/qemu_process.h:24,
from ../src/qemu/qemu_process.c:41:
../src/conf/numa_conf.h:130:55: note: expected ‘virDomainNuma *’ {aka
‘struct _virDomainNuma *’} but argument is of type ‘int’
130 | virBitmap *virDomainNumaGetNodeCpumask(virDomainNuma *numa,
| ~~~~~~~~~~~~~~~^~~~
../src/qemu/qemu_process.c:2744:34: error: assignment to ‘int’ from
‘virBitmap *’ {aka ‘struct _virBitmap *’} makes integer from pointer
without a cast [-Werror=int-conversion]
2744 | numanode_cpumask =
virDomainNumaGetNodeCpumask(numatune, i);
| ^
../src/qemu/qemu_process.c:2748:39: error: passing argument 1 of
‘virBitmapIsBitSet’ makes pointer from integer without a cast
[-Werror=int-conversion]
2748 | if (virBitmapIsBitSet(numanode_cpumask, id) &&
| ^~~~~~~~~~~~~~~~
| |
| int
In file included from ../src/conf/cpu_conf.h:25,
from ../src/conf/capabilities.h:27,
from ../src/qemu/qemu_conf.h:28,
from ../src/qemu/qemu_process.h:24,
from ../src/qemu/qemu_process.c:41:
../src/util/virbitmap.h:63:35: note: expected ‘virBitmap *’ {aka ‘struct
_virBitmap *’} but argument is of type ‘int’
63 | bool virBitmapIsBitSet(virBitmap *bitmap, size_t b)
| ~~~~~~~~~~~^~~~~~
../src/qemu/qemu_process.c:2749:46: error: passing argument 1 of
‘virDomainNumatuneGetMode’ makes pointer from integer without a cast
[-Werror=int-conversion]
2749 | virDomainNumatuneGetMode(numatune, i,
&mem_mode) == 0 &&
| ^~~~~~~~
| |
| int
In file included from ../src/conf/cpu_conf.h:27,
from ../src/conf/capabilities.h:27,
from ../src/qemu/qemu_conf.h:28,
from ../src/qemu/qemu_process.h:24,
from ../src/qemu/qemu_process.c:41:
../src/conf/numa_conf.h:104:45: note: expected ‘virDomainNuma *’ {aka
‘struct _virDomainNuma *’} but argument is of type ‘int’
104 | int virDomainNumatuneGetMode(virDomainNuma *numatune,
| ~~~~~~~~~~~~~~~^~~~~~~~
../src/qemu/qemu_process.c:2751:61: error: passing argument 1 of
‘virDomainNumatuneMaybeFormatNodeset’ makes pointer from integer without a
cast [-Werror=int-conversion]
2751 | if
(virDomainNumatuneMaybeFormatNodeset(numatune,
| ^~~~~~~~
| |
| int
In file included from ../src/conf/cpu_conf.h:27,
from ../src/conf/capabilities.h:27,
from ../src/qemu/qemu_conf.h:28,
from ../src/qemu/qemu_process.h:24,
from ../src/qemu/qemu_process.c:41:
../src/conf/numa_conf.h:155:56: note: expected ‘virDomainNuma *’ {aka
‘struct _virDomainNuma *’} but argument is of type ‘int’
155 | int virDomainNumatuneMaybeFormatNodeset(virDomainNuma *numatune,
| ~~~~~~~~~~~~~~~^~~~~~~~
cc1: all warnings being treated as errors
Thanks,
Jing Qi
On Tue, Apr 13, 2021 at 2:37 PM Luyao Zhong <luyao.zhong(a)intel.com> wrote:
Reviewed-by: Daniel Henrique Barboza <danielhb413(a)gmail.com>
Signed-off-by: Luyao Zhong <luyao.zhong(a)intel.com>
---
include/libvirt/libvirt-domain.h | 1 +
src/conf/numa_conf.c | 9 ++++
src/qemu/qemu_command.c | 6 ++-
src/qemu/qemu_process.c | 27 ++++++++++++
src/util/virnuma.c | 3 ++
.../numatune-memnode-invalid-mode.err | 1 +
.../numatune-memnode-invalid-mode.xml | 33 +++++++++++++++
...emnode-restrictive-mode.x86_64-latest.args | 38 +++++++++++++++++
.../numatune-memnode-restrictive-mode.xml | 33 +++++++++++++++
tests/qemuxml2argvtest.c | 2 +
...memnode-restrictive-mode.x86_64-latest.xml | 41 +++++++++++++++++++
tests/qemuxml2xmltest.c | 1 +
12 files changed, 194 insertions(+), 1 deletion(-)
create mode 100644
tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
create mode 100644
tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
create mode 100644
tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
create mode 100644
tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
create mode 100644
tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
diff --git a/include/libvirt/libvirt-domain.h
b/include/libvirt/libvirt-domain.h
index 03c119fe26..e99bfb7654 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -1527,6 +1527,7 @@ typedef enum {
VIR_DOMAIN_NUMATUNE_MEM_STRICT = 0,
VIR_DOMAIN_NUMATUNE_MEM_PREFERRED = 1,
VIR_DOMAIN_NUMATUNE_MEM_INTERLEAVE = 2,
+ VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE = 3,
# ifdef VIR_ENUM_SENTINELS
VIR_DOMAIN_NUMATUNE_MEM_LAST /* This constant is subject to change */
diff --git a/src/conf/numa_conf.c b/src/conf/numa_conf.c
index 64b93fd7d1..11093531b5 100644
--- a/src/conf/numa_conf.c
+++ b/src/conf/numa_conf.c
@@ -43,6 +43,7 @@ VIR_ENUM_IMPL(virDomainNumatuneMemMode,
"strict",
"preferred",
"interleave",
+ "restrictive",
);
VIR_ENUM_IMPL(virDomainNumatunePlacement,
@@ -234,6 +235,14 @@ virDomainNumatuneNodeParseXML(virDomainNumaPtr numa,
_("Invalid mode attribute in memnode
element"));
goto cleanup;
}
+
+ if (numa->memory.mode == VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE
&&
+ mode != VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) {
+ virReportError(VIR_ERR_XML_ERROR, "%s",
+ _("'restrictive' mode is required in
memnode element "
+ "when mode is 'restrictive' in memory
element"));
+ goto cleanup;
+ }
VIR_FREE(tmp);
mem_node->mode = mode;
}
diff --git a/src/qemu/qemu_command.c b/src/qemu/qemu_command.c
index 45eb0dc976..a6a1717115 100644
--- a/src/qemu/qemu_command.c
+++ b/src/qemu/qemu_command.c
@@ -175,6 +175,7 @@ VIR_ENUM_IMPL(qemuNumaPolicy,
"bind",
"preferred",
"interleave",
+ "restricted",
);
VIR_ENUM_DECL(qemuAudioDriver);
@@ -3300,7 +3301,10 @@ qemuBuildMemoryBackendProps(virJSONValuePtr
*backendProps,
return -1;
}
- if (nodemask) {
+ /* If mode is "restrictive", we should only use cgroups setting
allowed memory
+ * nodes, and skip passing the host-nodes and policy parameters to
QEMU command
+ * line which means we will use system default memory policy. */
+ if (nodemask && mode != VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) {
if (!virNumaNodesetIsAvailable(nodemask))
return -1;
if (virJSONValueObjectAdd(props,
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c
index b74afedd07..c887cb35f0 100644
--- a/src/qemu/qemu_process.c
+++ b/src/qemu/qemu_process.c
@@ -2692,6 +2692,7 @@ qemuProcessSetupPid(virDomainObjPtr vm,
g_autoptr(virBitmap) hostcpumap = NULL;
g_autofree char *mem_mask = NULL;
int ret = -1;
+ size_t i;
if ((period || quota) &&
!virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPU))
{
@@ -2732,6 +2733,32 @@ qemuProcessSetupPid(virDomainObjPtr vm,
&mem_mask, -1) < 0)
goto cleanup;
+ /* For vCPU threads, mem_mask is different among cells and
mem_mask
+ * is used to set cgroups cpuset.mems for vcpu threads. If we
specify
+ * 'restrictive' mode, that means we will set system default
memory
+ * policy and only use cgroups to restrict allowed memory nodes.
*/
+ if (nameval == VIR_CGROUP_THREAD_VCPU) {
+ virDomainNumaPtr numatune = vm->def->numa;
+ virBitmapPtr numanode_cpumask = NULL;
+ for (i = 0; i < virDomainNumaGetNodeCount(numatune); i++) {
+ numanode_cpumask = virDomainNumaGetNodeCpumask(numatune,
i);
+ /* 'i' indicates the cell id, if the vCPU id is in this
cell
+ * and mode is 'restrictive', we need get the
corresponding
+ * nodeset. */
+ if (virBitmapIsBitSet(numanode_cpumask, id) &&
+ virDomainNumatuneGetMode(numatune, i, &mem_mode) == 0
&&
+ mem_mode == VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE) {
+ if (virDomainNumatuneMaybeFormatNodeset(numatune,
+
priv->autoNodeset,
+ &mem_mask, i)
< 0) {
+ goto cleanup;
+ } else {
+ break;
+ }
+ }
+ }
+ }
+
if (virCgroupNewThread(priv->cgroup, nameval, id, true, &cgroup)
< 0)
goto cleanup;
diff --git a/src/util/virnuma.c b/src/util/virnuma.c
index 6c194b54d1..34db746d28 100644
--- a/src/util/virnuma.c
+++ b/src/util/virnuma.c
@@ -152,6 +152,9 @@ virNumaSetupMemoryPolicy(virDomainNumatuneMemMode mode,
numa_set_interleave_mask(&mask);
break;
+ case VIR_DOMAIN_NUMATUNE_MEM_RESTRICTIVE:
+ break;
+
case VIR_DOMAIN_NUMATUNE_MEM_LAST:
break;
}
diff --git a/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
new file mode 100644
index 0000000000..180e64d1d8
--- /dev/null
+++ b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
@@ -0,0 +1 @@
+XML error: 'restrictive' mode is required in memnode element when mode is
'restrictive' in memory element
diff --git a/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
new file mode 100644
index 0000000000..a7c18d4d50
--- /dev/null
+++ b/tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
@@ -0,0 +1,33 @@
+<domain type='qemu'>
+ <name>QEMUGuest</name>
+ <uuid>9f4b6512-e73a-4a25-93e8-5307802821ce</uuid>
+ <memory unit='KiB'>24682468</memory>
+ <currentMemory unit='KiB'>24682468</currentMemory>
+ <vcpu placement='static'>32</vcpu>
+ <numatune>
+ <memory mode='restrictive' nodeset='0-7'/>
+ <memnode cellid='0' mode='restrictive' nodeset='3'/>
+ <memnode cellid='2' mode='strict'
nodeset='1-2,5-7,^6'/>
+ </numatune>
+ <os>
+ <type arch='x86_64' machine='pc'>hvm</type>
+ <boot dev='hd'/>
+ </os>
+ <cpu>
+ <numa>
+ <cell id='0' cpus='0' memory='20002'
unit='KiB'/>
+ <cell id='1' cpus='1-27,29' memory='660066'
unit='KiB'/>
+ <cell id='2' cpus='28,30-31' memory='24002400'
unit='KiB'/>
+ </numa>
+ </cpu>
+ <clock offset='utc'/>
+ <on_poweroff>destroy</on_poweroff>
+ <on_reboot>restart</on_reboot>
+ <on_crash>destroy</on_crash>
+ <devices>
+ <emulator>/usr/bin/qemu-system-x86_64</emulator>
+ <controller type='usb' index='0'/>
+ <controller type='pci' index='0' model='pci-root'/>
+ <memballoon model='virtio'/>
+ </devices>
+</domain>
diff --git
a/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
new file mode 100644
index 0000000000..6d5e2eb76e
--- /dev/null
+++
b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
@@ -0,0 +1,38 @@
+LC_ALL=C \
+PATH=/bin \
+HOME=/tmp/lib/domain--1-QEMUGuest \
+USER=test \
+LOGNAME=test \
+XDG_DATA_HOME=/tmp/lib/domain--1-QEMUGuest/.local/share \
+XDG_CACHE_HOME=/tmp/lib/domain--1-QEMUGuest/.cache \
+XDG_CONFIG_HOME=/tmp/lib/domain--1-QEMUGuest/.config \
+/usr/bin/qemu-system-x86_64 \
+-name guest=QEMUGuest,debug-threads=on \
+-S \
+-object
'{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/tmp/lib/domain--1-QEMUGuest/master-key.aes"}'
\
+-machine pc,accel=tcg,usb=off,dump-guest-core=off \
+-cpu qemu64 \
+-m 24105 \
+-overcommit mem-lock=off \
+-smp 32,sockets=32,cores=1,threads=1 \
+-object
'{"qom-type":"memory-backend-ram","id":"ram-node0","size":20971520}'
\
+-numa node,nodeid=0,cpus=0,memdev=ram-node0 \
+-object
'{"qom-type":"memory-backend-ram","id":"ram-node1","size":676331520}'
\
+-numa node,nodeid=1,cpus=1-27,cpus=29,memdev=ram-node1 \
+-object
'{"qom-type":"memory-backend-ram","id":"ram-node2","size":24578621440}'
\
+-numa node,nodeid=2,cpus=28,cpus=30-31,memdev=ram-node2 \
+-uuid 9f4b6512-e73a-4a25-93e8-5307802821ce \
+-display none \
+-no-user-config \
+-nodefaults \
+-chardev socket,id=charmonitor,fd=1729,server=on,wait=off \
+-mon chardev=charmonitor,id=monitor,mode=control \
+-rtc base=utc \
+-no-shutdown \
+-no-acpi \
+-boot strict=on \
+-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 \
+-audiodev id=audio1,driver=none \
+-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x2 \
+-sandbox
on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
+-msg timestamp=on
diff --git a/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
new file mode 100644
index 0000000000..72949b0657
--- /dev/null
+++ b/tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
@@ -0,0 +1,33 @@
+<domain type='qemu'>
+ <name>QEMUGuest</name>
+ <uuid>9f4b6512-e73a-4a25-93e8-5307802821ce</uuid>
+ <memory unit='KiB'>24682468</memory>
+ <currentMemory unit='KiB'>24682468</currentMemory>
+ <vcpu placement='static'>32</vcpu>
+ <numatune>
+ <memnode cellid='0' mode='restrictive' nodeset='3'/>
+ <memory mode='restrictive' nodeset='0-7'/>
+ <memnode cellid='2' mode='restrictive'
nodeset='1-2,5-7,^6'/>
+ </numatune>
+ <os>
+ <type arch='x86_64' machine='pc'>hvm</type>
+ <boot dev='hd'/>
+ </os>
+ <cpu>
+ <numa>
+ <cell id='0' cpus='0' memory='20002'
unit='KiB'/>
+ <cell id='1' cpus='1-27,29' memory='660066'
unit='KiB'/>
+ <cell id='2' cpus='28,30-31' memory='24002400'
unit='KiB'/>
+ </numa>
+ </cpu>
+ <clock offset='utc'/>
+ <on_poweroff>destroy</on_poweroff>
+ <on_reboot>restart</on_reboot>
+ <on_crash>destroy</on_crash>
+ <devices>
+ <emulator>/usr/bin/qemu-system-x86_64</emulator>
+ <controller type='usb' index='0'/>
+ <controller type='pci' index='0' model='pci-root'/>
+ <memballoon model='virtio'/>
+ </devices>
+</domain>
diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c
index 317ee3267d..b6d2e8323d 100644
--- a/tests/qemuxml2argvtest.c
+++ b/tests/qemuxml2argvtest.c
@@ -2171,6 +2171,8 @@ mymain(void)
DO_TEST_PARSE_ERROR("numatune-memnode", NONE);
DO_TEST_CAPS_VER("numatune-memnode", "5.2.0");
DO_TEST_CAPS_LATEST("numatune-memnode");
+ DO_TEST_CAPS_LATEST("numatune-memnode-restrictive-mode");
+ DO_TEST_PARSE_ERROR("numatune-memnode-invalid-mode", NONE);
DO_TEST("numatune-memnode-no-memory",
QEMU_CAPS_NUMA,
diff --git
a/tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
b/tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
new file mode 100644
index 0000000000..012c526460
--- /dev/null
+++
b/tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
@@ -0,0 +1,41 @@
+<domain type='qemu'>
+ <name>QEMUGuest</name>
+ <uuid>9f4b6512-e73a-4a25-93e8-5307802821ce</uuid>
+ <memory unit='KiB'>24682468</memory>
+ <currentMemory unit='KiB'>24682468</currentMemory>
+ <vcpu placement='static'>32</vcpu>
+ <numatune>
+ <memory mode='restrictive' nodeset='0-7'/>
+ <memnode cellid='0' mode='restrictive' nodeset='3'/>
+ <memnode cellid='2' mode='restrictive'
nodeset='1-2,5,7'/>
+ </numatune>
+ <os>
+ <type arch='x86_64' machine='pc'>hvm</type>
+ <boot dev='hd'/>
+ </os>
+ <cpu mode='custom' match='exact' check='none'>
+ <model fallback='forbid'>qemu64</model>
+ <numa>
+ <cell id='0' cpus='0' memory='20002'
unit='KiB'/>
+ <cell id='1' cpus='1-27,29' memory='660066'
unit='KiB'/>
+ <cell id='2' cpus='28,30-31' memory='24002400'
unit='KiB'/>
+ </numa>
+ </cpu>
+ <clock offset='utc'/>
+ <on_poweroff>destroy</on_poweroff>
+ <on_reboot>restart</on_reboot>
+ <on_crash>destroy</on_crash>
+ <devices>
+ <emulator>/usr/bin/qemu-system-x86_64</emulator>
+ <controller type='usb' index='0' model='piix3-uhci'>
+ <address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x2'/>
+ </controller>
+ <controller type='pci' index='0' model='pci-root'/>
+ <input type='mouse' bus='ps2'/>
+ <input type='keyboard' bus='ps2'/>
+ <audio id='1' type='none'/>
+ <memballoon model='virtio'>
+ <address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
+ </memballoon>
+ </devices>
+</domain>
diff --git a/tests/qemuxml2xmltest.c b/tests/qemuxml2xmltest.c
index a5343cd6f8..32b68fcfb0 100644
--- a/tests/qemuxml2xmltest.c
+++ b/tests/qemuxml2xmltest.c
@@ -1125,6 +1125,7 @@ mymain(void)
DO_TEST("numatune-distances", QEMU_CAPS_NUMA, QEMU_CAPS_NUMA_DIST);
DO_TEST("numatune-no-vcpu", QEMU_CAPS_NUMA);
DO_TEST("numatune-hmat", QEMU_CAPS_NUMA_HMAT,
QEMU_CAPS_OBJECT_MEMORY_RAM);
+ DO_TEST_CAPS_LATEST("numatune-memnode-restrictive-mode");
DO_TEST("bios-nvram", NONE);
DO_TEST("bios-nvram-os-interleave", NONE);
--
2.25.4
--
Thanks & Regards,
Jing,Qi
5:05 a.m.
On Tue, Apr 13, 2021 at 02:38:05PM +0800, Luyao Zhong wrote:
Before this patch set, numatune only has three memory modes:
static, interleave and prefered. These memory policies are
ultimately set by mbind() system call.
Memory policy could be 'hard coded' into the kernel, but none of
above policies fit our requirment under this case. mbind() support
default memory policy, but it requires a NULL nodemask. So obviously
setting allowed memory nodes is cgroups' mission under this case.
So we introduce a new option for mode in numatune named 'restrictive'.
<numatune>
<memory mode="restrictive" nodeset="1-4,^3"/>
<memnode cellid="0" mode="restrictive"
nodeset="1"/>
<memnode cellid="2" mode="restrictive"
nodeset="2"/>
</numatune>
The config above means we only use cgroups to restrict the allowed
memory nodes and not setting any specific memory policies explicitly.
For this new "restrictive" mode, there is a concrete use case about a
new feature in kernel but not merged yet, we call it memory tiering.
(https://lwn.net/Articles/802544/).
If memory tiering is enabled on host, DRAM is top tier memory, and
PMEM(persistent memory) is second tier memory, PMEM is shown as numa node
without cpu. Pages can be migrated between DRAM node and PMEM node based on
DRAM pressure and how cold/hot they are. *this memory policy* is implemented
in kernel. So we need a default mode here, but from libvirt's perspective,
the "defaut" mode is "strict", it's not MPOL_DEFAULT
(https://man7.org/linux/man-pages/man2/mbind.2.html) defined in kernel.
And to make memory tiering works well, cgroups setting is necessary, since
it restricts that the pages can only be migrated between the DRAM and PMEM
nodes that we specified (NUMA affinity support).
Just using cgroups with multiple nodes in the nodeset makes kernel decide
on which node (out of those in the restricted set) to allocate on, but specifying
"strict" basically allocates it sequentially (on the first one until it is
full,
then on the next one and so on).
In a word, if a user requires default mode(MPOL_DEFAULT), that means they want
kernel decide the memory allocation and also want the cgroups to restrict memory
nodes, "restrictive" mode will be useful.
I applied the changes locally and fixed some changes that happened in
the meantime. I also split the patches differently as we usually add
conf, docs and schemas (driver-unrelated code) and some possible tests
in one patch and then add support for each applicable driver in separate
patches. I reworded some comments there were also two memory leaks that
I fixed and I will resend the series later to see if we have everything
in order.
If we disagree on the naming, then we can change it until the release,
but I do not think that is something that should stall the patches.
Thanks.
BR,
Luyao
Luyao Zhong (3):
docs: add docs for 'restrictive' option for mode in numatune
schema: add 'restrictive' config option for mode in numatune
qemu: add parser and formatter for 'restrictive' mode in numatune
docs/formatdomain.rst | 7 +++-
docs/schemas/domaincommon.rng | 2 +
include/libvirt/libvirt-domain.h | 1 +
src/conf/numa_conf.c | 9 ++++
src/qemu/qemu_command.c | 6 ++-
src/qemu/qemu_process.c | 27 ++++++++++++
src/util/virnuma.c | 3 ++
.../numatune-memnode-invalid-mode.err | 1 +
.../numatune-memnode-invalid-mode.xml | 33 +++++++++++++++
...emnode-restrictive-mode.x86_64-latest.args | 38 +++++++++++++++++
.../numatune-memnode-restrictive-mode.xml | 33 +++++++++++++++
tests/qemuxml2argvtest.c | 2 +
...memnode-restrictive-mode.x86_64-latest.xml | 41 +++++++++++++++++++
tests/qemuxml2xmltest.c | 1 +
14 files changed, 201 insertions(+), 3 deletions(-)
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.err
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-invalid-mode.xml
create mode 100644
tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.x86_64-latest.args
create mode 100644 tests/qemuxml2argvdata/numatune-memnode-restrictive-mode.xml
create mode 100644
tests/qemuxml2xmloutdata/numatune-memnode-restrictive-mode.x86_64-latest.xml
--
2.25.4
1316
days inactive
1319
days old
5 comments
3 participants
participants (3)
-
Jing Qi -
Luyao Zhong -
Martin Kletzander