[please keep the list CC'ed] On 10.06.2015 20:09, Clarylin L wrote: > Hi Michal, > > Thanks a lot. > > If 100 hugepages are pre-allocated, the guest can start without decreasing > number of hugepages. Since the guest requires 128 hugepages, it's kind of > expected that the guest would not take memory from hugepages. > > Before guest start, > > [root@local ~]# cat /proc/meminfo | grep uge > > AnonH*uge*Pages: 0 kB > > H*uge*Pages_Total: 100 > > H*uge*Pages_Free: 100 > > H*uge*Pages_Rsvd: 0 > > H*uge*Pages_Surp: 0 > H*uge*pagesize: 1048576 kB > > After: > > [root@local ~]# cat /proc/meminfo | grep uge > > AnonH*uge*Pages: 134254592 kB > > H*uge*Pages_Total: 100 > > H*uge*Pages_Free: 100 > > H*uge*Pages_Rsvd: 0 > > H*uge*Pages_Surp: 0 > H*uge*pagesize: 1048576 kB > > There is no -mem-prealloc and -mem-path options in qemu command And there can't be. From the command line below, you are defining 2 NUMA nodes for your guest. In order to instruct qemu to back their memory by huge pages you need it to support memory-backend-file object which was introduced in qemu-2.1.0. The other option you have is to not use guest NUMA nodes, in which case global -mem-path can be used.

> > [root@local ~]# ps -ef | grep qemu > > *qemu* 3403 1 99 17:42 ? 00:36:42 /usr/libexec/*qemu*-kvm > -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off -cpu > host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1 > -numa node,nodeid=0,cpus=0-15,mem=65536 -numa > node,nodeid=1,cpus=16-31,mem=65536 -uuid > e1b72349-4a0b-4b91-aedc-fd34e92251e4 -smbios type=1,serial=SCALE-SLOT-03 > -no-user-config -nodefaults -chardev > socket,id=charmonitor,path=/var/lib/libvirt/*qemu*/qvpc-di-03-sf.monitor,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive > file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf-hda.img,if=none,id=drive-ide0-0-0,format=qcow2 > -device > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 > -chardev pty,id=charserial0 -device > isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 > -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga > cirrus -device i6300esb,id=watchdog0,bus=pci.0,addr=0x3 -watchdog-action > reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5 -device > vfio-pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device > vfio-pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on > > > If 140 hugepages are preallocated, the guest cannot start and it complained > not enough memory. > > > The libvirt version is shown as follows: > > virsh # version > > Compiled against library: libvirt 1.2.8 > > Using library: libvirt 1.2.8 > > Using API: QEMU 1.2.8 > > Running hypervisor: QEMU 1.5.3 > > > Also in guest configuration contains numa session. The hugepages are > uniformly distributed to two nodes. In this case, do I need to make > additional configurations to enable usage of hugepages? > > <numatune> > > <memory mode='strict' nodeset='0-1'/> > > </numatune> > This says that all the memory for you guests should be pinned onto host nodes 0-1. If you want to be more specific, you can explicitly wire guest NUMA nodes onto host NUMA nodes in 1:N relationship (where N can be even 1, in which case you will get 1:1), e.g.: <memnode cellid='0' mode='preferred' nodeset='0'/> Michal

Hi Michal, I also tried the other option you mentioned "The other option you have is to not use guest NUMA nodes, in which case global -mem-path can be used." by removing from xml <numatune> <memory mode='strict' nodeset='0-1'/> </numatune> while keeping older qemu-1.5.3 which does not support <page> element, it still complained no enough memory when starting guest. Did I miss something? On Thu, Jun 11, 2015 at 12:00 AM, Michal Privoznik <mprivozn(a)redhat.com&gt; wrote: > [please keep the list CC'ed] > > On 10.06.2015 20:09, Clarylin L wrote: > > Hi Michal, > > > > Thanks a lot. > > > > If 100 hugepages are pre-allocated, the guest can start without > decreasing > > number of hugepages. Since the guest requires 128 hugepages, it's kind > of > > expected that the guest would not take memory from hugepages. > > > > Before guest start, > > > > [root@local ~]# cat /proc/meminfo | grep uge > > > > AnonH*uge*Pages: 0 kB > > > > H*uge*Pages_Total: 100 > > > > H*uge*Pages_Free: 100 > > > > H*uge*Pages_Rsvd: 0 > > > > H*uge*Pages_Surp: 0 > > H*uge*pagesize: 1048576 kB > > > > After: > > > > [root@local ~]# cat /proc/meminfo | grep uge > > > > AnonH*uge*Pages: 134254592 kB > > > > H*uge*Pages_Total: 100 > > > > H*uge*Pages_Free: 100 > > > > H*uge*Pages_Rsvd: 0 > > > > H*uge*Pages_Surp: 0 > > H*uge*pagesize: 1048576 kB > > > > There is no -mem-prealloc and -mem-path options in qemu command > > And there can't be. From the command line below, you are defining 2 NUMA > nodes for your guest. In order to instruct qemu to back their memory by > huge pages you need it to support memory-backend-file object which was > introduced in qemu-2.1.0. > The other option you have is to not use guest NUMA nodes, in which case > global -mem-path can be used. > > > > > [root@local ~]# ps -ef | grep qemu > > > > *qemu* 3403 1 99 17:42 ? 00:36:42 > /usr/libexec/*qemu*-kvm > > -name qvpc-di-03-sf -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off > -cpu > > host -m 131072 -realtime mlock=off -smp 32,sockets=2,cores=16,threads=1 > > -numa node,nodeid=0,cpus=0-15,mem=65536 -numa > > node,nodeid=1,cpus=16-31,mem=65536 -uuid > > e1b72349-4a0b-4b91-aedc-fd34e92251e4 -smbios type=1,serial=SCALE-SLOT-03 > > -no-user-config -nodefaults -chardev > > > socket,id=charmonitor,path=/var/lib/libvirt/*qemu*/qvpc-di-03-sf.monitor,server,nowait > > -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc > -no-shutdown > > -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 > -drive > > > file=/var/lib/libvirt/images/asr5700/qvpc-di-03-sf-hda.img,if=none,id=drive-ide0-0-0,format=qcow2 > > -device > > ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 > -drive > > if=none,id=drive-ide0-1-0,readonly=on,format=raw -device > > ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=1 > > -chardev pty,id=charserial0 -device > > isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 > > -device isa-serial,chardev=charserial1,id=serial1 -vnc 127.0.0.1:0 -vga > > cirrus -device i6300esb,id=watchdog0,bus=pci.0,addr=0x3 -watchdog-action > > reset -device vfio-pci,host=08:00.0,id=hostdev0,bus=pci.0,addr=0x5 > -device > > vfio-pci,host=09:00.0,id=hostdev1,bus=pci.0,addr=0x6 -device > > vfio-pci,host=0a:00.0,id=hostdev2,bus=pci.0,addr=0x7 -device > > virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x4 -msg timestamp=on > > > > > > If 140 hugepages are preallocated, the guest cannot start and it > complained > > not enough memory. > > > > > > The libvirt version is shown as follows: > > > > virsh # version > > > > Compiled against library: libvirt 1.2.8 > > > > Using library: libvirt 1.2.8 > > > > Using API: QEMU 1.2.8 > > > > Running hypervisor: QEMU 1.5.3 > > > > > > > > > Also in guest configuration contains numa session. The hugepages are > > uniformly distributed to two nodes. In this case, do I need to make > > additional configurations to enable usage of hugepages? > > > > <numatune> > > > > <memory mode='strict' nodeset='0-1'/> > > > > </numatune> > > > > This says that all the memory for you guests should be pinned onto host > nodes 0-1. If you want to be more specific, you can explicitly wire > guest NUMA nodes onto host NUMA nodes in 1:N relationship (where N can > be even 1, in which case you will get 1:1), e.g.: > > <memnode cellid='0' mode='preferred' nodeset='0'/> > > Michal >