8:56 p.m.
Hi Daniel et.al,
Running into the following errors when attempting to start up a multi-vNUMA
node VM backed by 1G huge page per physical NUMA node. (details below)
error: unsupported configuration: Per-node memory binding is not supported
with this QEMU
(or in another case)
error: unsupported configuration: huge pages per NUMA node are not
supported with this QEMU
What am I missing ?
Thanks!
Vinod
System : (CentOS 7.3)
---------
# numactl -H
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23
node 0 size: 65320 MB
node 0 free: 62672 MB
node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31
node 1 size: 65536 MB
node 1 free: 63323 MB
node distances:
node 0 1
0: 10 20
1: 20 10
# uname -a
Linux <hostname> 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux
# virsh version
Compiled against library: libvirt 2.0.0
Using library: libvirt 2.0.0
Using API: QEMU 2.0.0
Running hypervisor: QEMU 1.5.3
# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-514.el7.x86_64 root=/dev/mapper/cl-root ro
crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet
LANG=en_US.UTF-8 intel_iommu=on iommu=pt default_hugepagesz=1G
hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=4096
# mount
...
...
none on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G)
none on /dev/hugepages2M type hugetlbfs (rw,relatime,seclabel,pagesize=2M)
-----------
Test 1: 1G Huge page memoryBacking & memory binding per numa node, and
virtual numa nodes.
--------
<domain type='kvm'>
<name>vm1</name>
<uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memtune>
<hard_limit unit='KiB'>9437184</hard_limit>
</memtune>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
<page size='1048576' unit='KiB' nodeset='1'/>
</hugepages>
</memoryBacking>
<vcpu placement='static'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='8'/>
<vcpupin vcpu='3' cpuset='9'/>
</cputune>
<numatune>
<memory mode='strict' nodeset='0-1'/>
<memnode cellid='0' mode='strict' nodeset='0'/>
<memnode cellid='1' mode='strict' nodeset='1'/>
</numatune>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>IvyBridge</model>
<numa>
<cell id='0' cpus='0-1' memory='4194304'
unit='KiB'/>
<cell id='1' cpus='2-3' memory='4194304'
unit='KiB'/>
</numa>
</cpu>
...
# virsh start vm1
error: Failed to start domain vm1
error: unsupported configuration: Per-node memory binding is not supported
with this QEMU
Test 2: 1G Huge page memoryBacking per NUMA node and virtual numa nodes.
------
<domain type='kvm'>
<name>vm1</name>
<uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memtune>
<hard_limit unit='KiB'>9437184</hard_limit>
</memtune>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
<page size='1048576' unit='KiB' nodeset='1'/>
</hugepages>
</memoryBacking>
<vcpu placement='static'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='8'/>
<vcpupin vcpu='3' cpuset='9'/>
</cputune>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>IvyBridge</model>
<numa>
<cell id='0' cpus='0-1' memory='4194304'
unit='KiB'/>
<cell id='1' cpus='2-3' memory='4194304'
unit='KiB'/>
</numa>
</cpu>
...
# virsh start vm1
error: Failed to start domain vm1
error: unsupported configuration: huge pages per NUMA node are not
supported with this QEMU
Test 3: Only virtual numa nodes.
------
<main type='kvm'>
<name>vm1</name>
<uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memtune>
<hard_limit unit='KiB'>9437184</hard_limit>
</memtune>
<vcpu placement='static'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='8'/>
<vcpupin vcpu='3' cpuset='9'/>
</cputune>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>IvyBridge</model>
<numa>
<cell id='0' cpus='0-1' memory='4194304'
unit='KiB'/>
<cell id='1' cpus='2-3' memory='4194304'
unit='KiB'/>
</numa>
</cpu>
...
# virsh start vm1
<Success and the guest OS shows the virtual NUMA nodes etc.>
Test 4: Only 1G huge page backing per NUMA node
------
<domain type='kvm'>
<name>vm1</name>
<uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<memtune>
<hard_limit unit='KiB'>9437184</hard_limit>
</memtune>
<memoryBacking>
<hugepages>
<page size='1048576' unit='KiB' nodeset='0'/>
<page size='1048576' unit='KiB' nodeset='1'/>
</hugepages>
</memoryBacking>
<vcpu placement='static'>4</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<vcpupin vcpu='2' cpuset='8'/>
<vcpupin vcpu='3' cpuset='9'/>
</cputune>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>IvyBridge</model>
</cpu>
...
# virsh start vm1
<Success and the 1G pages on the host are consumed for backing VM's memory>