11:34 a.m.
Hello,
Hope all is well.
I've been looking at how I can create a virtual machine which is NUMA
capable. I was able to do that by:
140 <qemu:commandline>
143 <qemu:arg value='-numa'/>
144 <qemu:arg value='node'/>
145 <qemu:arg value='-numa'/>
146 <qemu:arg value='node'/>
147 </qemu:commandline>
This creates two NUMA nodes.
Now I would like to assign my virtual network interfaces, of which I have
three, to the different NUMA nodes on the VM.
Anyone know how I can do that? Any help/pointers would be much appreciated.
I'm trying to create a VM that will enable me to test a NUMA based feature.
thanks
amir
1:40 p.m.
On 11/21/2016 12:34 PM, Amir Shehata wrote:
Hello,
Hope all is well.
I've been looking at how I can create a virtual machine which is NUMA
capable. I was able to do that by:
140 <qemu:commandline>
143 <qemu:arg value='-numa'/>
144 <qemu:arg value='node'/>
145 <qemu:arg value='-numa'/>
146 <qemu:arg value='node'/>
147 </qemu:commandline>
qemu commandline passthrough should only be used for qemu features that
libvirt doesn't support directly (and even then it's only intended as a
temporary stopgap until the feature gains official libvirt support).
NUMA has been supported by libvirt for quite a long time.
If you open http://www.libvirt.org/formatdomain.html and search for
"numa" you'll find descriptions of the configuration for defining NUMA
memory regions, and assigning particular CPUs to particular nodes. The
is also a small bit about defining a pci-expander-bus (a type of pci
controller) and tying it to a particular NUMA node - this is how you can
place a guest device on a particular NUMA node; just assign the device
a PCI address that has the given pci-expander-bus' "index" as its
"bus"
attribute.
This creates two NUMA nodes.
Now I would like to assign my virtual network interfaces, of which I
have three, to the different NUMA nodes on the VM.
Anyone know how I can do that? Any help/pointers would be much
appreciated.
I'm trying to create a VM that will enable me to test a NUMA based
feature.
The one thing I'm unclear about is whether the guest can be setup with
NUMA nodes when the host itself doesn't have multiple NUMA nodes...
3:48 p.m.
Laine,
Thank you for your response.
I did use the numa element instead of using qemu commandline and that
worked:
16 <cpu mode='custom' match='exact'>
...
34 <numa>
35 <cell id='0' cpus='0-1' memory='1048576'
unit='KiB'/>
36 <cell id='1' cpus='2-3' memory='1048576'
unit='KiB'/>
37 </numa>
38 </cpu>
However, whenever I try to use the pci-expander-bus I get:
error: XML document failed to validate against schema: Unable to validate
doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content
I tried the following
93 <controller type='pci' index='1'
model='pci-expander-bus'>
94 <address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
function='0x0'/>
95 </controller>
and
93 <controller type='pci' index='1'
model='pci-expander-bus'>
94 <target busNr='254'>
95 <node>1</node>
96 </target>
97 <address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
function='0x0'/>
98 </controller>
I got the syntax from:
https://www.redhat.com/archives/libvir-list/2016-March/msg01224.html
Just to give you a bit more info. This is running on:
Linux ashehata-DellPE 3.19.0-51-generic #58-Ubuntu SMP Fri Feb 26 21:22:26
UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
below is a dump of the working XML file I'm using
thanks
amir
<domain type='kvm'
xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>MR_test01</name>
<uuid>59eafc3c-9d9e-4dab-bac9-48003048c6dc</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-utopic'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>core2duo</model>
<vendor>Intel</vendor>
<feature policy='require' name='pbe'/>
<feature policy='require' name='tm2'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='dca'/>
<feature policy='require' name='lahf_lm'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='cx16'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='acpi'/>
<numa>
<cell id='0' cpus='0-1' memory='1048576'
unit='KiB'/>
<cell id='1' cpus='2-3' memory='1048576'
unit='KiB'/>
</numa>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/media/ashehata/Lustre03/kvm-linux2/disk1.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x09'
function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/media/ashehata/Lustre03/kvm-linux2/disk4.qcow2'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hda' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x08'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01'
function='0x1'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07'
function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:da:1b:af'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03'
function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:84:0a:66'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04'
function='0x0'/>
</interface>
<interface type='network'>
<mac address='52:54:00:17:9d:cd'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05'
function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0'
port='1'/>
</channel>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes'/>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x06'
function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536'
vgamem='16384' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02'
function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x0a'
function='0x0'/>
</memballoon>
</devices>
<qemu:commandline>
<qemu:arg value='-gdb'/>
<qemu:arg value='tcp::1201'/>
</qemu:commandline>
</domain>
On 21 November 2016 at 11:40, Laine Stump <laine(a)laine.org> wrote:
On 11/21/2016 12:34 PM, Amir Shehata wrote:
> Hello,
>
> Hope all is well.
>
> I've been looking at how I can create a virtual machine which is NUMA
> capable. I was able to do that by:
>
> 140 <qemu:commandline>
> 143 <qemu:arg value='-numa'/>
> 144 <qemu:arg value='node'/>
> 145 <qemu:arg value='-numa'/>
> 146 <qemu:arg value='node'/>
> 147 </qemu:commandline>
>
qemu commandline passthrough should only be used for qemu features that
libvirt doesn't support directly (and even then it's only intended as a
temporary stopgap until the feature gains official libvirt support). NUMA
has been supported by libvirt for quite a long time.
If you open http://www.libvirt.org/formatdomain.html and search for
"numa" you'll find descriptions of the configuration for defining NUMA
memory regions, and assigning particular CPUs to particular nodes. The is
also a small bit about defining a pci-expander-bus (a type of pci
controller) and tying it to a particular NUMA node - this is how you can
place a guest device on a particular NUMA node; just assign the device a
PCI address that has the given pci-expander-bus' "index" as its
"bus"
attribute.
> This creates two NUMA nodes.
>
> Now I would like to assign my virtual network interfaces, of which I have
> three, to the different NUMA nodes on the VM.
>
> Anyone know how I can do that? Any help/pointers would be much
> appreciated.
>
> I'm trying to create a VM that will enable me to test a NUMA based
> feature.
>
The one thing I'm unclear about is whether the guest can be setup with
NUMA nodes when the host itself doesn't have multiple NUMA nodes...
4:06 p.m.
On 11/21/2016 04:48 PM, Amir Shehata wrote:
Laine,
Thank you for your response.
I did use the numa element instead of using qemu commandline and that
worked:
16 <cpu mode='custom' match='exact'>
...
34 <numa>
35 <cell id='0' cpus='0-1' memory='1048576'
unit='KiB'/>
36 <cell id='1' cpus='2-3' memory='1048576'
unit='KiB'/>
37 </numa>
38 </cpu>
However, whenever I try to use the pci-expander-bus I get:
error: XML document failed to validate against schema: Unable to
validate doc against /usr/share/libvirt/schemas/domain.rng
Extra element devices in interleave
Element domain failed to validate content
Sorry, I forgot to mention that pci-expander-bus is a fairly recent
addition to libvirt and qemu. For libvirt support of that controller,
you'll need at least libvirt 1.3.4, and qemu 2.4.0. Anything earlier
than that and the guest OS won't have any visibility to the NUMA node of
specific devices.
I tried the following
93 <controller type='pci' index='1'
model='pci-expander-bus'>
94 <address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
function='0x0'/>
95 </controller>
and
93 <controller type='pci' index='1'
model='pci-expander-bus'>
94 <target busNr='254'>
95 <node>1</node>
96 </target>
97 <address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
function='0x0'/>
98 </controller>
I got the syntax from:
https://www.redhat.com/archives/libvir-list/2016-March/msg01224.html
Just to give you a bit more info. This is running on:
Linux ashehata-DellPE 3.19.0-51-generic #58-Ubuntu SMP Fri Feb 26
21:22:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
below is a dump of the working XML file I'm using
I notice you don't have any device with pci bus='1'. I'm assuming you
just hadn't gotten to that part of the experiment yet, correct?
10:04 a.m.
Laine,
Thanks. You're correct that I haven't gotten to use the new pci bus yet.
I'm currently trying to upgrade to the versions you specified. Just having
a bunch of problems... I'll let you know the results of the experiment once
I sort them out.
thanks
amir
On 21 November 2016 at 14:06, Laine Stump <laine(a)laine.org> wrote:
On 11/21/2016 04:48 PM, Amir Shehata wrote:
> Laine,
>
> Thank you for your response.
>
> I did use the numa element instead of using qemu commandline and that
> worked:
>
> 16 <cpu mode='custom' match='exact'>
> ...
> 34 <numa>
> 35 <cell id='0' cpus='0-1' memory='1048576'
unit='KiB'/>
> 36 <cell id='1' cpus='2-3' memory='1048576'
unit='KiB'/>
> 37 </numa>
> 38 </cpu>
>
> However, whenever I try to use the pci-expander-bus I get:
> error: XML document failed to validate against schema: Unable to validate
> doc against /usr/share/libvirt/schemas/domain.rng
> Extra element devices in interleave
> Element domain failed to validate content
>
Sorry, I forgot to mention that pci-expander-bus is a fairly recent
addition to libvirt and qemu. For libvirt support of that controller,
you'll need at least libvirt 1.3.4, and qemu 2.4.0. Anything earlier than
that and the guest OS won't have any visibility to the NUMA node of
specific devices.
> I tried the following
>
> 93 <controller type='pci' index='1'
model='pci-expander-bus'>
> 94 <address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
> function='0x0'/>
> 95 </controller>
>
> and
>
> 93 <controller type='pci' index='1'
model='pci-expander-bus'>
> 94 <target busNr='254'>
> 95 <node>1</node>
> 96 </target>
> 97 <address type='pci' domain='0x0000' bus='0x00'
slot='0x0b'
> function='0x0'/>
> 98 </controller>
>
>
> I got the syntax from: https://www.redhat.com/archive
> s/libvir-list/2016-March/msg01224.html
>
> Just to give you a bit more info. This is running on:
> Linux ashehata-DellPE 3.19.0-51-generic #58-Ubuntu SMP Fri Feb 26
> 21:22:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>
> below is a dump of the working XML file I'm using
>
I notice you don't have any device with pci bus='1'. I'm assuming you
just
hadn't gotten to that part of the experiment yet, correct?
2:11 a.m.
so I was able to upgrade to the indicated versions. Had to go to Fedora 25,
since it natively supports these versions, and centos doesn't. upgrading
centos proved to be unreasonably difficult for me.
After upgrading I added the syntax as described in the docs. I have
configured 4 vcpus and 2 numa nodes. Node 0 has vcpus 0-1 and node 1 has
vcpus 2-3. Configured two pci-expander-buses: index 1 assigned to node 0
and index 2 assigned to node 1.
I use these indices for the bus values in the interfaces. But on booting up
the system, only the interfaces connected to node 1 are visible to the
guest OS.
Not sure why that would be.
If there are any ideas, that would be great.
Happy thanksgiving to the US folks.
On 22 November 2016 at 08:04, Amir Shehata <amir.shehata.whamcloud(a)gmail.com
wrote:
> Laine,
>
> Thanks. You're correct that I haven't gotten to use the new pci bus yet.
>
> I'm currently trying to upgrade to the versions you specified. Just having
> a bunch of problems... I'll let you know the results of the experiment once
> I sort them out.
>
> thanks
> amir
>
> On 21 November 2016 at 14:06, Laine Stump <laine(a)laine.org wrote:
>
>> On 11/21/2016 04:48 PM, Amir Shehata wrote:
>>
>>> Laine,
>>>
>>> Thank you for your response.
>>>
>>> I did use the numa element instead of using qemu commandline and that
>>> worked:
>>>
>>> 16 <cpu mode='custom' match='exact'>
>>> ...
>>> 34 <numa>
>>> 35 <cell id='0' cpus='0-1' memory='1048576'
unit='KiB'/>
>>> 36 <cell id='1' cpus='2-3' memory='1048576'
unit='KiB'/>
>>> 37 </numa>
>>> 38 </cpu>
>>>
>>> However, whenever I try to use the pci-expander-bus I get:
>>> error: XML document failed to validate against schema: Unable to
>>> validate doc against /usr/share/libvirt/schemas/domain.rng
>>> Extra element devices in interleave
>>> Element domain failed to validate content
>>>
>>
>> Sorry, I forgot to mention that pci-expander-bus is a fairly recent
>> addition to libvirt and qemu. For libvirt support of that controller,
>> you'll need at least libvirt 1.3.4, and qemu 2.4.0. Anything earlier than
>> that and the guest OS won't have any visibility to the NUMA node of
>> specific devices.
>>
>>
>>
>>> I tried the following
>>>
>>> 93 <controller type='pci' index='1'
model='pci-expander-bus'>
>>> 94 <address type='pci' domain='0x0000'
bus='0x00' slot='0x0b'
>>> function='0x0'/>
>>> 95 </controller>
>>>
>>> and
>>>
>>> 93 <controller type='pci' index='1'
model='pci-expander-bus'>
>>> 94 <target busNr='254'>
>>> 95 <node>1</node>
>>> 96 </target>
>>> 97 <address type='pci' domain='0x0000'
bus='0x00' slot='0x0b'
>>> function='0x0'/>
>>> 98 </controller>
>>>
>>>
>>> I got the syntax from: https://www.redhat.com/archive
>>> s/libvir-list/2016-March/msg01224.html
>>>
>>> Just to give you a bit more info. This is running on:
>>> Linux ashehata-DellPE 3.19.0-51-generic #58-Ubuntu SMP Fri Feb 26
>>> 21:22:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
>>>
>>
>>
>>> below is a dump of the working XML file I'm using
>>>
>>
>> I notice you don't have any device with pci bus='1'. I'm assuming
you
>> just hadn't gotten to that part of the experiment yet, correct?
>>
>>
>>
>
2799
days inactive
2804
days old
5 comments
2 participants
participants (2)
-
Amir Shehata -
Laine Stump