Hi Daniel et.al,

Running into the following errors when attempting to start up a multi-vNUMA node VM backed by 1G huge page per physical NUMA node. (details below)

error: unsupported configuration: Per-node memory binding is not supported with this QEMU

(or in another case)

error: unsupported configuration: huge pages per NUMA node are not supported with this QEMU

What am I missing ? 

Thanks!

Vinod

 

 

System : (CentOS 7.3)

---------

 

# numactl -H

available: 2 nodes (0-1)

node 0 cpus: 0 1 2 3 4 5 6 7 16 17 18 19 20 21 22 23 

node 0 size: 65320 MB 

node 0 free: 62672 MB 

node 1 cpus: 8 9 10 11 12 13 14 15 24 25 26 27 28 29 30 31 

node 1 size: 65536 MB 

node 1 free: 63323 MB 

node distances:

node   0   1

  0:  10  20

  1:  20  10

 

# uname -a

Linux <hostname> 3.10.0-514.el7.x86_64 #1 SMP Tue Nov 22 16:42:41 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

 

# virsh version

Compiled against library: libvirt 2.0.0

Using library: libvirt 2.0.0

Using API: QEMU 2.0.0

Running hypervisor: QEMU 1.5.3

 

# cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.10.0-514.el7.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet LANG=en_US.UTF-8 intel_iommu=on iommu=pt default_hugepagesz=1G hugepagesz=1G hugepages=16 hugepagesz=2M hugepages=4096

 

# mount

...

...

none on /dev/hugepages1G type hugetlbfs (rw,relatime,seclabel,pagesize=1G)

none on /dev/hugepages2M type hugetlbfs (rw,relatime,seclabel,pagesize=2M)

-----------

Test 1:  1G Huge page memoryBacking & memory binding per numa node, and virtual numa nodes.

--------

<domain type='kvm'>

  <name>vm1</name>

  <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>

  <memory unit='KiB'>8388608</memory>

  <currentMemory unit='KiB'>8388608</currentMemory>

  <memtune>

    <hard_limit unit='KiB'>9437184</hard_limit>

  </memtune>

  <memoryBacking>

    <hugepages>

      <page size='1048576' unit='KiB' nodeset='0'/>

      <page size='1048576' unit='KiB' nodeset='1'/>

    </hugepages>

  </memoryBacking>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='1'/>

    <vcpupin vcpu='1' cpuset='2'/>

    <vcpupin vcpu='2' cpuset='8'/>

    <vcpupin vcpu='3' cpuset='9'/>

  </cputune>

  <numatune>

    <memory mode='strict' nodeset='0-1'/>

    <memnode cellid='0' mode='strict' nodeset='0'/>

    <memnode cellid='1' mode='strict' nodeset='1'/>

  </numatune>

  <os>

    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='custom' match='exact'>

    <model fallback='allow'>IvyBridge</model>

    <numa>

      <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/>

      <cell id='1' cpus='2-3' memory='4194304' unit='KiB'/>

    </numa>

  </cpu>

...

 

# virsh start vm1

error: Failed to start domain vm1

error: unsupported configuration: Per-node memory binding is not supported with this QEMU

Test 2: 1G Huge page memoryBacking per NUMA node and virtual numa nodes.

------

 

<domain type='kvm'>

  <name>vm1</name>

  <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>

  <memory unit='KiB'>8388608</memory>

  <currentMemory unit='KiB'>8388608</currentMemory>

  <memtune>

    <hard_limit unit='KiB'>9437184</hard_limit>

  </memtune>

  <memoryBacking>

    <hugepages>

      <page size='1048576' unit='KiB' nodeset='0'/>

      <page size='1048576' unit='KiB' nodeset='1'/>

    </hugepages>

  </memoryBacking>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='1'/>

    <vcpupin vcpu='1' cpuset='2'/>

    <vcpupin vcpu='2' cpuset='8'/>

    <vcpupin vcpu='3' cpuset='9'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='custom' match='exact'>

    <model fallback='allow'>IvyBridge</model>

    <numa>

      <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/>

      <cell id='1' cpus='2-3' memory='4194304' unit='KiB'/>

    </numa>

  </cpu>

...

# virsh start vm1

error: Failed to start domain vm1

error: unsupported configuration: huge pages per NUMA node are not supported with this QEMU

 

Test 3: Only virtual numa nodes.

------

 

<main type='kvm'>

  <name>vm1</name>

  <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>

  <memory unit='KiB'>8388608</memory>

  <currentMemory unit='KiB'>8388608</currentMemory>

  <memtune>

    <hard_limit unit='KiB'>9437184</hard_limit>

  </memtune>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='1'/>

    <vcpupin vcpu='1' cpuset='2'/>

    <vcpupin vcpu='2' cpuset='8'/>

    <vcpupin vcpu='3' cpuset='9'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='custom' match='exact'>

    <model fallback='allow'>IvyBridge</model>

    <numa>

      <cell id='0' cpus='0-1' memory='4194304' unit='KiB'/>

      <cell id='1' cpus='2-3' memory='4194304' unit='KiB'/>

    </numa>

  </cpu>

...

 

# virsh start vm1

 

<Success and the guest OS shows the virtual NUMA nodes etc.>

Test 4: Only 1G huge page backing per NUMA node

------

 

<domain type='kvm'>

  <name>vm1</name>

  <uuid>62077ba4-9d7a-4b81-8dfb-b0cd55693cc2</uuid>

  <memory unit='KiB'>8388608</memory>

  <currentMemory unit='KiB'>8388608</currentMemory>

  <memtune>

    <hard_limit unit='KiB'>9437184</hard_limit>

  </memtune>

  <memoryBacking>

    <hugepages>

      <page size='1048576' unit='KiB' nodeset='0'/>

      <page size='1048576' unit='KiB' nodeset='1'/>

    </hugepages>

  </memoryBacking>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='1'/>

    <vcpupin vcpu='1' cpuset='2'/>

    <vcpupin vcpu='2' cpuset='8'/>

    <vcpupin vcpu='3' cpuset='9'/>

  </cputune>

  <os>

    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>

    <boot dev='hd'/>

  </os>

  <features>

    <acpi/>

    <apic/>

  </features>

  <cpu mode='custom' match='exact'>

    <model fallback='allow'>IvyBridge</model>

  </cpu>

 

...

 

# virsh start vm1

 

<Success and the 1G pages on the host are consumed for backing VM's memory>