Hi Daniel et.al, Running into the following errors when attempting to start up a multi-vNUMA node VM backed by 1G huge page per physical NUMA node. (details below) error: unsupported configuration: Per-node memory binding is not supported with this QEMU (or in another case) error: unsupported configuration: huge pages per NUMA node are not supported with this QEMU What am I missing ? Thanks! Vinod System : (CentOS 7.3) --------- # numactl -H available: 2 nodes (0-1) node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 node 0 size: 65320 MB node 0 free: 62672 MB node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 node 1 size: 65536 MB node 1 free: 63323 MB node distances: node 0 1 0: 10 20 1: 20 10 # uname -a Linux <hostname> 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux # virsh version Compiled against library: libvirt 2.0.0 Using library: libvirt 2.0.0 Using API: QEMU 2.0.0 Running hypervisor: QEMU 1.5.3 # cat /proc/cmdline BOOT_IMAGE=/vmlinuz-3.10.0-514.el7.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet LANG=en_US.UTF-8 intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=4096 # mount ... ... none on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G) none on /dev/hugepages2M type hugetlbfs (rw,relatime,seclabel,pagesize=2M) ----------- Test 1: 1G Huge page memoryBacking & memory binding per numa node, and virtual numa nodes. -------- <domain type='kvm'> <name>vm1</name> <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memtune> <hard_limit unit='KiB'>9437184</hard_limit> </memtune> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> <page size='1048576' unit='KiB' nodeset='1'/> </hugepages> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='9'/> </cputune> <numatune> <memory mode='strict' nodeset='0-1'/> <memnode cellid='0' mode='strict' nodeset='0'/> <memnode cellid='1' mode='strict' nodeset='1'/> </numatune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>IvyBridge</model> <numa> <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/> <cell id='1' cpus='2-3' memory='4194304' unit='KiB'/> </numa> </cpu> ... # virsh start vm1 error: Failed to start domain vm1 error: unsupported configuration: Per-node memory binding is not supported with this QEMU Test 2: 1G Huge page memoryBacking per NUMA node and virtual numa nodes. ------ <domain type='kvm'> <name>vm1</name> <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memtune> <hard_limit unit='KiB'>9437184</hard_limit> </memtune> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> <page size='1048576' unit='KiB' nodeset='1'/> </hugepages> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='9'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>IvyBridge</model> <numa> <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/> <cell id='1' cpus='2-3' memory='4194304' unit='KiB'/> </numa> </cpu> ... # virsh start vm1 error: Failed to start domain vm1 error: unsupported configuration: huge pages per NUMA node are not supported with this QEMU Test 3: Only virtual numa nodes. ------ <main type='kvm'> <name>vm1</name> <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memtune> <hard_limit unit='KiB'>9437184</hard_limit> </memtune> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='9'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>IvyBridge</model> <numa> <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/> <cell id='1' cpus='2-3' memory='4194304' unit='KiB'/> </numa> </cpu> ... # virsh start vm1 <Success and the guest OS shows the virtual NUMA nodes etc.> Test 4: Only 1G huge page backing per NUMA node ------ <domain type='kvm'> <name>vm1</name> <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid> <memory unit='KiB'>8388608</memory> <currentMemory unit='KiB'>8388608</currentMemory> <memtune> <hard_limit unit='KiB'>9437184</hard_limit> </memtune> <memoryBacking> <hugepages> <page size='1048576' unit='KiB' nodeset='0'/> <page size='1048576' unit='KiB' nodeset='1'/> </hugepages> </memoryBacking> <vcpu placement='static'>4</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='2'/> <vcpupin vcpu='2' cpuset='8'/> <vcpupin vcpu='3' cpuset='9'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='custom' match='exact'> <model fallback='allow'>IvyBridge</model> </cpu> ... # virsh start vm1 <Success and the 1G pages on the host are consumed for backing VM's memory>