Re: [libvirt] with vhostuser I cannot use hugepages

Wednesday, 24 January 2018

Ah I see the error

You have
<memoryBacking>
    <hugepages/>
      <page size='1' unit='G' nodeset='0'/>
  </memoryBacking>

Not
<memoryBacking>
    <hugepages>
      <page size='1' unit='G' nodeset='0'/>
    </hugepages>
  </memoryBacking>

From: Adnan Mundres [mailto:amundres@yahoo.com]
Sent: Wednesday, January 24, 2018 9:05 PM
To: Mooney, Sean K <sean.k.mooney(a)intel.com>; libvir-list(a)redhat.com
Subject: Re: [libvirt] with vhostuser I cannot use hugepages

Hi Sean
,

Thanks for you reply. I do have hugepages configuration in my xml and still it is not
getting the memory from Hugepage. Following is my complete xml file

<domain type='kvm'>
  <uuid>c1f7a3d0-ccb7-4a85-b0f0-dd4bab73ba54</uuid>
  <description>Timos VM</description>
  <memory>16777216</memory>
  <currentMemory>16777216</currentMemory>
  <cpu mode='host-model'>
    <numa>
      <cell id="0" cpus="0" memory="16777216"
unit='KiB' memAccess="shared"/>
    </numa>
   </cpu>
  <memoryBacking>
    <hugepages/>
      <page size='1' unit='G' nodeset='0'/>
  </memoryBacking>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <vcpu current='7'>7</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <vcpupin vcpu='6' cpuset='8'/>
    <emulatorpin cpuset="0,1"/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc'>hvm</type>
    <smbios mode='sysinfo'/>
  </os>
  <sysinfo type ='smbios'
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='delay'/>
  </clock>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/mvmgptb11/images/flexibed/vm_5/sros-vm.qcow2'/>
      <target dev='hda' bus='virtio'/>
    </disk>
    <interface type='bridge'>
      <mac address='FA:AC:A5:00:7E:00'/>
      <source bridge='breth0'/>
      <model type='virtio'/>
    </interface>
    <interface type='bridge'>
      <mac address='FA:AC:A5:00:7E:01'/>
      <source bridge='breth1'/>
      <model type='virtio'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='FA:AC:A5:00:7E:02'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user-1'
mode='client'/>
      <model type='virtio'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='FA:AC:A5:00:7E:04'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user-3'
mode='client'/>
      <model type='virtio'/>
    </interface>
    <interface type='vhostuser'>
      <mac address='FA:AC:A5:00:7E:05'/>
      <source type='unix' path='/var/run/openvswitch/vhost-user-2'
mode='client'/>
      <model type='virtio'/>
    </interface>
    <console type='tcp'>
      <source mode='bind' host='0.0.0.0' service='2501'/>
      <protocol type='telnet'/>
      <target type='virtio' port='0'/>
    </console>
    <watchdog model='i6300esb' action='reset'/>
  </devices>
  <seclabel type='none'/>
</domain>

On Wednesday, January 24, 2018 12:54 PM, "Mooney, Sean K"
<sean.k.mooney@intel.com<mailto:sean.k.mooney@intel.com>> wrote:

From: Mooney, Sean K
Sent: Wednesday, January 24, 2018 8:52 PM
To: Adnan Mundres <amundres@yahoo.com<mailto:amundres@yahoo.com>>;
libvir-list@redhat.com<mailto:libvir-list@redhat.com>
Cc: Mooney, Sean K <sean.k.mooney@intel.com<mailto:sean.k.mooney@intel.com>>
Subject: RE: [libvirt] with vhostuser I cannot use hugepages

From: libvir-list-bounces@redhat.com<mailto:libvir-list-bounces@redhat.com>
[mailto:libvir-list-bounces@redhat.com] On Behalf Of Adnan Mundres
Sent: Wednesday, January 24, 2018 6:54 PM
To: libvir-list@redhat.com<mailto:libvir-list@redhat.com>
Subject: [libvirt] with vhostuser I cannot use hugepages

In my setup I am using qemu-kvm without openstack. I am trying to use ovs with dpdk. In my
xml file I added following lines for the dpdkvhostuser

<cpu mode='host-passthrough'>
    <numa>
      <cell id="0" cpus="0-6" memory="16777216"
unit='KiB' memAccess="shared"/>
    </numa>
 </cpu>
[Mooney, Sean K] this xml snippit is not requsting the use of huge page memory.
It is requesting memory to be allocated form a single numa node  and for that to be mmaped
shared.
Dpdk does not actually need the guest to use hugepages it just improves the performance.
What it requires
Is that the gues uses file descriptor backed, pre allocated, locked, shared memory.
e.g.
  <memoryBacking>
    <locked/>
    <source type='file'/>
    <access mode='shared'/>
    <allocation mode='immediate'/>
  </memoryBacking>

Hugepages will enable all of the above features implicitly.

to enable hugepages  add

<memoryBacking>
<hugepages>
      <page size="16" unit="G" nodeset="0"/>
    </hugepages>
</memoryBacking>

[Mooney, Sean K]  oh I forgot to say you can find the relevant documentation here
https://libvirt.org/formatdomain.html#elementsMemoryBacking

I have enabled hugepages during boot time

ot@mvmgptb11hyp01 hyp-1]# cat /proc/meminfo | grep -i huge
AnonHugePages: 126976 kB
HugePages_Total: 100
HugePages_Free: 80
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 1048576 kB

But When I start my vm (virsh start vm1.xml) I am seeing that this vm is not using memory
from hugepages, rather it is taking memory from total memory. When I checked the log file
I see that it is using

-object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/var/lib/libvirt/qemu/ram,share=yes

It should use backend memory as following

-object
memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/vm1,share=yes

Any idea how can I use memory from hugepages

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Re: [libvirt] with vhostuser I cannot use hugepages