[libvirt] with vhostuser I cannot use hugepages
In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my xml file I added following lines for the dpdkvhostuser <cpu mode='host-passthrough'> <numa> <cell id="0" cpus="0-6" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> I have enabled hugepages during boot time ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i hugeAnonHugePages: 126976 kBHugePages_Total: 100HugePages_Free: 80HugePages_Rsvd: 0HugePages_Surp: 0Hugepagesize: 1048576 kB But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory from hugepages, rather it is taking memory from total memory. When I checked the log file I see that it is using -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes It should use backend memory as following -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes Any idea how can I use memory from hugepages
From: libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com] On Behalf Of Adnan Mundres Sent: Wednesday, January 24, 2018 6:54 PM To: libvir-list@redhat.com Subject: [libvirt] with vhostuser I cannot use hugepages In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my xml file I added following lines for the dpdkvhostuser <cpu mode='host-passthrough'> <numa> <cell id="0" cpus="0-6" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> [Mooney, Sean K] this xml snippit is not requsting the use of huge page memory. It is requesting memory to be allocated form a single numa node and for that to be mmaped shared. Dpdk does not actually need the guest to use hugepages it just improves the performance. What it requires Is that the gues uses file descriptor backed, pre allocated, locked, shared memory. e.g. <memoryBacking> <locked/> <source type='file'/> <access mode='shared'/> <allocation mode='immediate'/> </memoryBacking> Hugepages will enable all of the above features implicitly. to enable hugepages add <memoryBacking> <hugepages> <page size="16" unit="G" nodeset="0"/> </hugepages> </memoryBacking> I have enabled hugepages during boot time ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i huge AnonHugePages: 126976 kB HugePages_Total: 100 HugePages_Free: 80 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory from hugepages, rather it is taking memory from total memory. When I checked the log file I see that it is using -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes It should use backend memory as following -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes Any idea how can I use memory from hugepages
From: Mooney, Sean K Sent: Wednesday, January 24, 2018 8:52 PM To: Adnan Mundres <amundres@yahoo.com>; libvir-list@redhat.com Cc: Mooney, Sean K <sean.k.mooney@intel.com> Subject: RE: [libvirt] with vhostuser I cannot use hugepages From: libvir-list-bounces@redhat.com<mailto:libvir-list-bounces@redhat.com> [mailto:libvir-list-bounces@redhat.com] On Behalf Of Adnan Mundres Sent: Wednesday, January 24, 2018 6:54 PM To: libvir-list@redhat.com<mailto:libvir-list@redhat.com> Subject: [libvirt] with vhostuser I cannot use hugepages In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my xml file I added following lines for the dpdkvhostuser <cpu mode='host-passthrough'> <numa> <cell id="0" cpus="0-6" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> [Mooney, Sean K] this xml snippit is not requsting the use of huge page memory. It is requesting memory to be allocated form a single numa node and for that to be mmaped shared. Dpdk does not actually need the guest to use hugepages it just improves the performance. What it requires Is that the gues uses file descriptor backed, pre allocated, locked, shared memory. e.g. <memoryBacking> <locked/> <source type='file'/> <access mode='shared'/> <allocation mode='immediate'/> </memoryBacking> Hugepages will enable all of the above features implicitly. to enable hugepages add <memoryBacking> <hugepages> <page size="16" unit="G" nodeset="0"/> </hugepages> </memoryBacking> [Mooney, Sean K] oh I forgot to say you can find the relevant documentation here https://libvirt.org/formatdomain.html#elementsMemoryBacking I have enabled hugepages during boot time ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i huge AnonHugePages: 126976 kB HugePages_Total: 100 HugePages_Free: 80 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory from hugepages, rather it is taking memory from total memory. When I checked the log file I see that it is using -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes It should use backend memory as following -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes Any idea how can I use memory from hugepages
Hi Sean , Thanks for you reply. I do have hugepages configuration in my xml and still it is not getting the memory from Hugepage. Following is my complete xml file <domain type='kvm'> <uuid>c1f7a3d0-ccb7-4a85-b0f0-dd4bab73ba54</uuid> <description>Timos VM</description> <memory>16777216</memory> <currentMemory>16777216</currentMemory> <cpu mode='host-model'> <numa> <cell id="0" cpus="0" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> <memoryBacking> <hugepages/> <page size='1' unit='G' nodeset='0'/> </memoryBacking> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <vcpu current='7'>7</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='7'/> <vcpupin vcpu='6' cpuset='8'/> <emulatorpin cpuset="0,1"/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <smbios mode='sysinfo'/> </os> <sysinfo type ='smbios' <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='delay'/> </clock> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/mvmgptb11/images/flexibed/vm_5/sros-vm.qcow2'/> <target dev='hda' bus='virtio'/> </disk> <interface type='bridge'> <mac address='FA:AC:A5:00:7E:00'/> <source bridge='breth0'/> <model type='virtio'/> </interface> <interface type='bridge'> <mac address='FA:AC:A5:00:7E:01'/> <source bridge='breth1'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:02'/> <source type='unix' path='/var/run/openvswitch/vhost-user-1' mode='client'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:04'/> <source type='unix' path='/var/run/openvswitch/vhost-user-3' mode='client'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:05'/> <source type='unix' path='/var/run/openvswitch/vhost-user-2' mode='client'/> <model type='virtio'/> </interface> <console type='tcp'> <source mode='bind' host='0.0.0.0' service='2501'/> <protocol type='telnet'/> <target type='virtio' port='0'/> </console> <watchdog model='i6300esb' action='reset'/> </devices> <seclabel type='none'/></domain> On Wednesday, January 24, 2018 12:54 PM, "Mooney, Sean K" <sean.k.mooney@intel.com> wrote: #yiv3224014168 #yiv3224014168 -- _filtered #yiv3224014168 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv3224014168 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv3224014168 {}#yiv3224014168 #yiv3224014168 p.yiv3224014168MsoNormal, #yiv3224014168 li.yiv3224014168MsoNormal, #yiv3224014168 div.yiv3224014168MsoNormal {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;}#yiv3224014168 a:link, #yiv3224014168 span.yiv3224014168MsoHyperlink {color:#0563C1;text-decoration:underline;}#yiv3224014168 a:visited, #yiv3224014168 span.yiv3224014168MsoHyperlinkFollowed {color:#954F72;text-decoration:underline;}#yiv3224014168 span.yiv3224014168EmailStyle17 {color:#1F497D;}#yiv3224014168 span.yiv3224014168EmailStyle18 {color:#1F497D;}#yiv3224014168 .yiv3224014168MsoChpDefault {font-size:10.0pt;} _filtered #yiv3224014168 {margin:72.0pt 72.0pt 72.0pt 72.0pt;}#yiv3224014168 div.yiv3224014168WordSection1 {}#yiv3224014168 From: Mooney, Sean K Sent: Wednesday, January 24, 2018 8:52 PM To: Adnan Mundres <amundres@yahoo.com>; libvir-list@redhat.com Cc: Mooney, Sean K <sean.k.mooney@intel.com> Subject: RE: [libvirt] with vhostuser I cannot use hugepages From:libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com]On Behalf Of Adnan Mundres Sent: Wednesday, January 24, 2018 6:54 PM To: libvir-list@redhat.com Subject: [libvirt] with vhostuser I cannot use hugepages In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my xml file I added following lines for the dpdkvhostuser <cpu mode='host-passthrough'> <numa> <cell id="0" cpus="0-6" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> [Mooney, Sean K] this xml snippit is not requsting the use of huge page memory. It is requesting memory to be allocated form a single numa node and for that to be mmaped shared. Dpdk does not actually need the guest to use hugepages it just improves the performance. What it requires Is that the gues uses file descriptor backed, pre allocated, locked, shared memory. e.g. <memoryBacking> <locked/> <source type='file'/> <access mode='shared'/> <allocation mode='immediate'/> </memoryBacking> Hugepages will enable all of the above features implicitly. to enable hugepages add <memoryBacking> <hugepages> <page size="16" unit="G" nodeset="0"/> </hugepages> </memoryBacking> [Mooney, Sean K] oh I forgot to say you can find the relevant documentation herehttps://libvirt.org/formatdomain.html#elementsMemoryBacking I have enabled hugepages during boot time ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i huge AnonHugePages: 126976 kB HugePages_Total: 100 HugePages_Free: 80 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory from hugepages, rather it is taking memory from total memory. When I checked the log file I see that it is using -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes It should use backend memory as following -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes Any idea how can I use memory from hugepages
Ah I see the error You have <memoryBacking> <hugepages/> <page size='1' unit='G' nodeset='0'/> </memoryBacking> Not <memoryBacking> <hugepages> <page size='1' unit='G' nodeset='0'/> </hugepages> </memoryBacking> From: Adnan Mundres [mailto:amundres@yahoo.com] Sent: Wednesday, January 24, 2018 9:05 PM To: Mooney, Sean K <sean.k.mooney@intel.com>; libvir-list@redhat.com Subject: Re: [libvirt] with vhostuser I cannot use hugepages Hi Sean , Thanks for you reply. I do have hugepages configuration in my xml and still it is not getting the memory from Hugepage. Following is my complete xml file <domain type='kvm'> <uuid>c1f7a3d0-ccb7-4a85-b0f0-dd4bab73ba54</uuid> <description>Timos VM</description> <memory>16777216</memory> <currentMemory>16777216</currentMemory> <cpu mode='host-model'> <numa> <cell id="0" cpus="0" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> <memoryBacking> <hugepages/> <page size='1' unit='G' nodeset='0'/> </memoryBacking> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <vcpu current='7'>7</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='7'/> <vcpupin vcpu='6' cpuset='8'/> <emulatorpin cpuset="0,1"/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <smbios mode='sysinfo'/> </os> <sysinfo type ='smbios' <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='delay'/> </clock> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/mvmgptb11/images/flexibed/vm_5/sros-vm.qcow2'/> <target dev='hda' bus='virtio'/> </disk> <interface type='bridge'> <mac address='FA:AC:A5:00:7E:00'/> <source bridge='breth0'/> <model type='virtio'/> </interface> <interface type='bridge'> <mac address='FA:AC:A5:00:7E:01'/> <source bridge='breth1'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:02'/> <source type='unix' path='/var/run/openvswitch/vhost-user-1' mode='client'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:04'/> <source type='unix' path='/var/run/openvswitch/vhost-user-3' mode='client'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:05'/> <source type='unix' path='/var/run/openvswitch/vhost-user-2' mode='client'/> <model type='virtio'/> </interface> <console type='tcp'> <source mode='bind' host='0.0.0.0' service='2501'/> <protocol type='telnet'/> <target type='virtio' port='0'/> </console> <watchdog model='i6300esb' action='reset'/> </devices> <seclabel type='none'/> </domain> On Wednesday, January 24, 2018 12:54 PM, "Mooney, Sean K" <sean.k.mooney@intel.com<mailto:sean.k.mooney@intel.com>> wrote: From: Mooney, Sean K Sent: Wednesday, January 24, 2018 8:52 PM To: Adnan Mundres <amundres@yahoo.com<mailto:amundres@yahoo.com>>; libvir-list@redhat.com<mailto:libvir-list@redhat.com> Cc: Mooney, Sean K <sean.k.mooney@intel.com<mailto:sean.k.mooney@intel.com>> Subject: RE: [libvirt] with vhostuser I cannot use hugepages From: libvir-list-bounces@redhat.com<mailto:libvir-list-bounces@redhat.com> [mailto:libvir-list-bounces@redhat.com] On Behalf Of Adnan Mundres Sent: Wednesday, January 24, 2018 6:54 PM To: libvir-list@redhat.com<mailto:libvir-list@redhat.com> Subject: [libvirt] with vhostuser I cannot use hugepages In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my xml file I added following lines for the dpdkvhostuser <cpu mode='host-passthrough'> <numa> <cell id="0" cpus="0-6" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> [Mooney, Sean K] this xml snippit is not requsting the use of huge page memory. It is requesting memory to be allocated form a single numa node and for that to be mmaped shared. Dpdk does not actually need the guest to use hugepages it just improves the performance. What it requires Is that the gues uses file descriptor backed, pre allocated, locked, shared memory. e.g. <memoryBacking> <locked/> <source type='file'/> <access mode='shared'/> <allocation mode='immediate'/> </memoryBacking> Hugepages will enable all of the above features implicitly. to enable hugepages add <memoryBacking> <hugepages> <page size="16" unit="G" nodeset="0"/> </hugepages> </memoryBacking> [Mooney, Sean K] oh I forgot to say you can find the relevant documentation here https://libvirt.org/formatdomain.html#elementsMemoryBacking I have enabled hugepages during boot time ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i huge AnonHugePages: 126976 kB HugePages_Total: 100 HugePages_Free: 80 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory from hugepages, rather it is taking memory from total memory. When I checked the log file I see that it is using -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes It should use backend memory as following -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes Any idea how can I use memory from hugepages
That was it. Really appreciate. I changed it correctly and it is working. Thanks again for your help On Wednesday, January 24, 2018 1:20 PM, "Mooney, Sean K" <sean.k.mooney@intel.com> wrote: #yiv5366243845 #yiv5366243845 -- _filtered #yiv5366243845 {panose-1:2 4 5 3 5 4 6 3 2 4;} _filtered #yiv5366243845 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;} _filtered #yiv5366243845 {}#yiv5366243845 #yiv5366243845 p.yiv5366243845MsoNormal, #yiv5366243845 li.yiv5366243845MsoNormal, #yiv5366243845 div.yiv5366243845MsoNormal {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;}#yiv5366243845 a:link, #yiv5366243845 span.yiv5366243845MsoHyperlink {color:blue;text-decoration:underline;}#yiv5366243845 a:visited, #yiv5366243845 span.yiv5366243845MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv5366243845 p.yiv5366243845msonormal, #yiv5366243845 li.yiv5366243845msonormal, #yiv5366243845 div.yiv5366243845msonormal {margin-right:0cm;margin-left:0cm;font-size:12.0pt;}#yiv5366243845 p.yiv5366243845msochpdefault, #yiv5366243845 li.yiv5366243845msochpdefault, #yiv5366243845 div.yiv5366243845msochpdefault {margin-right:0cm;margin-left:0cm;font-size:12.0pt;}#yiv5366243845 span.yiv5366243845msohyperlink {}#yiv5366243845 span.yiv5366243845msohyperlinkfollowed {}#yiv5366243845 span.yiv5366243845emailstyle17 {}#yiv5366243845 span.yiv5366243845emailstyle18 {}#yiv5366243845 p.yiv5366243845msonormal1, #yiv5366243845 li.yiv5366243845msonormal1, #yiv5366243845 div.yiv5366243845msonormal1 {margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;}#yiv5366243845 span.yiv5366243845msohyperlink1 {color:#0563C1;text-decoration:underline;}#yiv5366243845 span.yiv5366243845msohyperlinkfollowed1 {color:#954F72;text-decoration:underline;}#yiv5366243845 span.yiv5366243845emailstyle171 {color:#1F497D;}#yiv5366243845 span.yiv5366243845emailstyle181 {color:#1F497D;}#yiv5366243845 p.yiv5366243845msochpdefault1, #yiv5366243845 li.yiv5366243845msochpdefault1, #yiv5366243845 div.yiv5366243845msochpdefault1 {margin-right:0cm;margin-left:0cm;font-size:10.0pt;}#yiv5366243845 span.yiv5366243845EmailStyle29 {color:#1F497D;}#yiv5366243845 .yiv5366243845MsoChpDefault {font-size:10.0pt;} _filtered #yiv5366243845 {margin:72.0pt 72.0pt 72.0pt 72.0pt;}#yiv5366243845 div.yiv5366243845WordSection1 {}#yiv5366243845 Ah I see the error You have <memoryBacking> <hugepages/> <page size='1' unit='G' nodeset='0'/> </memoryBacking> Not <memoryBacking> <hugepages> <page size='1' unit='G' nodeset='0'/> </hugepages> </memoryBacking> From: Adnan Mundres [mailto:amundres@yahoo.com] Sent: Wednesday, January 24, 2018 9:05 PM To: Mooney, Sean K <sean.k.mooney@intel.com>; libvir-list@redhat.com Subject: Re: [libvirt] with vhostuser I cannot use hugepages Hi Sean , Thanks for you reply. I do have hugepages configuration in my xml and still it is not getting the memory from Hugepage. Following is my complete xml file <domain type='kvm'> <uuid>c1f7a3d0-ccb7-4a85-b0f0-dd4bab73ba54</uuid> <description>Timos VM</description> <memory>16777216</memory> <currentMemory>16777216</currentMemory> <cpu mode='host-model'> <numa> <cell id="0" cpus="0" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> <memoryBacking> <hugepages/> <page size='1' unit='G' nodeset='0'/> </memoryBacking> <numatune> <memory mode='strict' nodeset='1'/> </numatune> <vcpu current='7'>7</vcpu> <cputune> <vcpupin vcpu='0' cpuset='2'/> <vcpupin vcpu='1' cpuset='3'/> <vcpupin vcpu='2' cpuset='4'/> <vcpupin vcpu='3' cpuset='5'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='7'/> <vcpupin vcpu='6' cpuset='8'/> <emulatorpin cpuset="0,1"/> </cputune> <os> <type arch='x86_64' machine='pc'>hvm</type> <smbios mode='sysinfo'/> </os> <sysinfo type ='smbios' <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='delay'/> </clock> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/mvmgptb11/images/flexibed/vm_5/sros-vm.qcow2'/> <target dev='hda' bus='virtio'/> </disk> <interface type='bridge'> <mac address='FA:AC:A5:00:7E:00'/> <source bridge='breth0'/> <model type='virtio'/> </interface> <interface type='bridge'> <mac address='FA:AC:A5:00:7E:01'/> <source bridge='breth1'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:02'/> <source type='unix' path='/var/run/openvswitch/vhost-user-1' mode='client'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:04'/> <source type='unix' path='/var/run/openvswitch/vhost-user-3' mode='client'/> <model type='virtio'/> </interface> <interface type='vhostuser'> <mac address='FA:AC:A5:00:7E:05'/> <source type='unix' path='/var/run/openvswitch/vhost-user-2' mode='client'/> <model type='virtio'/> </interface> <console type='tcp'> <source mode='bind' host='0.0.0.0' service='2501'/> <protocol type='telnet'/> <target type='virtio' port='0'/> </console> <watchdog model='i6300esb' action='reset'/> </devices> <seclabel type='none'/> </domain> On Wednesday, January 24, 2018 12:54 PM, "Mooney, Sean K" <sean.k.mooney@intel.com> wrote: From: Mooney, Sean K Sent: Wednesday, January 24, 2018 8:52 PM To: Adnan Mundres <amundres@yahoo.com>;libvir-list@redhat.com Cc: Mooney, Sean K <sean.k.mooney@intel.com> Subject: RE: [libvirt] with vhostuser I cannot use hugepages From:libvir-list-bounces@redhat.com [mailto:libvir-list-bounces@redhat.com]On Behalf Of Adnan Mundres Sent: Wednesday, January 24, 2018 6:54 PM To: libvir-list@redhat.com Subject: [libvirt] with vhostuser I cannot use hugepages In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my xml file I added following lines for the dpdkvhostuser <cpu mode='host-passthrough'> <numa> <cell id="0" cpus="0-6" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> [Mooney, Sean K] this xml snippit is not requsting the use of huge page memory. It is requesting memory to be allocated form a single numa node and for that to be mmaped shared. Dpdk does not actually need the guest to use hugepages it just improves the performance. What it requires Is that the gues uses file descriptor backed, pre allocated, locked, shared memory. e.g. <memoryBacking> <locked/> <source type='file'/> <access mode='shared'/> <allocation mode='immediate'/> </memoryBacking> Hugepages will enable all of the above features implicitly. to enable hugepages add <memoryBacking> <hugepages> <page size="16" unit="G" nodeset="0"/> </hugepages> </memoryBacking> [Mooney, Sean K] oh I forgot to say you can find the relevant documentation herehttps://libvirt.org/formatdomain.html#elementsMemoryBacking I have enabled hugepages during boot time ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i huge AnonHugePages: 126976 kB HugePages_Total: 100 HugePages_Free: 80 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory from hugepages, rather it is taking memory from total memory. When I checked the log file I see that it is using -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes It should use backend memory as following -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes Any idea how can I use memory from hugepages -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 01/24/2018 10:19 PM, Mooney, Sean K wrote:
Ah I see the error
You have <memoryBacking> <hugepages/> <page size='1' unit='G' nodeset='0'/> </memoryBacking>
Not <memoryBacking> <hugepages> <page size='1' unit='G' nodeset='0'/> </hugepages> </memoryBacking>
Well, even in the former case libvirt should pick up some hugepages (with current implementation it usually picks 2M) Michal
On 01/24/2018 07:53 PM, Adnan Mundres wrote:
In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my xml file I added following lines for the dpdkvhostuser <cpu mode='host-passthrough'> <numa> <cell id="0" cpus="0-6" memory="16777216" unit='KiB' memAccess="shared"/> </numa> </cpu> I have enabled hugepages during boot time> ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i hugeAnonHugePages: 126976 kBHugePages_Total: 100HugePages_Free: 80HugePages_Rsvd: 0HugePages_Surp: 0Hugepagesize: 1048576 kB But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory from hugepages, rather it is taking memory from total memory. When I checked the log file I see that it is using -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes It should use backend memory as following -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes Any idea how can I use memory from hugepages
What version of libvirt are you using? There were some fixes made recently. Try latest git HEAD and if the issue still persists please file a bug report attaching domain XML and debug logs. I'm unable to reproduce with the current git. Michal
participants (3)
-
Adnan Mundres -
Michal Privoznik -
Mooney, Sean K