[libvirt] Non-contiguous NUMA cell numbers
Hi all, libvirt currently doesn't allow you to configure a guest with something like <cpu> <topology sockets='2' cores='5' threads='8'/> <numa> <cell id='0' cpus='0-39' memory='1048576' unit='KiB'/> <cell id='16' cpus='40-79' memory='1048576' unit='KiB'/> </numa> </cpu> with the following error: XML error: Exactly one 'cell' element per guest NUMA cell allowed, non-contiguous ranges or ranges not starting from 0 are not allowed The error message is very specific about not allowing gaps in NUMA cell numbering; however, on the very same host, I have $ numactl --hard | head -1 available: 4 nodes (0-1,16-17) so gaps in numbering are definitely possible, at least on ppc64. Should we consider relaxing this requirement and allow arbitrarily-numbered NUMA cells? Is there a specific reason why it was forbidden in the first place? Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Wed, Dec 23, 2015 at 05:14:16PM +0100, Andrea Bolognani wrote:
Hi all,
libvirt currently doesn't allow you to configure a guest with something like
<cpu> <topology sockets='2' cores='5' threads='8'/> <numa> <cell id='0' cpus='0-39' memory='1048576' unit='KiB'/> <cell id='16' cpus='40-79' memory='1048576' unit='KiB'/> </numa> </cpu>
with the following error:
XML error: Exactly one 'cell' element per guest NUMA cell allowed, non-contiguous ranges or ranges not starting from 0 are not allowed
The error message is very specific about not allowing gaps in NUMA cell numbering; however, on the very same host, I have
$ numactl --hard | head -1 available: 4 nodes (0-1,16-17)
so gaps in numbering are definitely possible, at least on ppc64.
Well that says Linux supports gaps, but does QEMU actually allow us to specify gaps ? Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 23.12.2015 17:14, Andrea Bolognani wrote:
Hi all,
libvirt currently doesn't allow you to configure a guest with something like
<cpu> <topology sockets='2' cores='5' threads='8'/> <numa> <cell id='0' cpus='0-39' memory='1048576' unit='KiB'/> <cell id='16' cpus='40-79' memory='1048576' unit='KiB'/> </numa>
This says you are configuring GUEST NUMA nodes, and last time I checked qemu did not support non-continuous NUMA nodes. So, whatever your HOST topology is, in GUEST you want NUMA nodes to be continuous.
</cpu>
with the following error:
XML error: Exactly one 'cell' element per guest NUMA cell allowed, non-contiguous ranges or ranges not starting from 0 are not allowed
The error message is very specific about not allowing gaps in NUMA cell numbering; however, on the very same host, I have
$ numactl --hard | head -1 available: 4 nodes (0-1,16-17)
so gaps in numbering are definitely possible, at least on ppc64.
Should we consider relaxing this requirement and allow arbitrarily-numbered NUMA cells? Is there a specific reason why it was forbidden in the first place?
Cheers.
Michal
n Mon, 2016-01-04 at 07:57 +0100, Michal Privoznik wrote:
On 23.12.2015 17:14, Andrea Bolognani wrote:
Hi all, libvirt currently doesn't allow you to configure a guest with something like <cpu> <topology sockets='2' cores='5' threads='8'/> <numa> <cell id='0' cpus='0-39' memory='1048576' unit='KiB'/> <cell id='16' cpus='40-79' memory='1048576' unit='KiB'/> </numa> This says you are configuring GUEST NUMA nodes, and last time I checked qemu did not support non-continuous NUMA nodes. So, whatever your HOST topology is, in GUEST you want NUMA nodes to be continuous.
I checked again and, as you and Daniel suggested, QEMU does not support non-contiguous NUMA nodes, eg. qemu-kvm \ -m 2048 \ -smp 80,sockets=2,cores=5,threads=8 \ -numa node,nodeid=0,cpus=0-39,mem=1024 \ -numa node,nodeid=16,cpus=40-79,mem=1024 outputs qemu-kvm: numa: Node ID missing: 15 so indeed there's a very good reason for libvirt to reject that NUMA configuration. Cheers. -- Andrea Bolognani Software Engineer - Virtualization Team
On Mon, Jan 4, 2016 at 12:18 PM, Andrea Bolognani <abologna@redhat.com> wrote:
n Mon, 2016-01-04 at 07:57 +0100, Michal Privoznik wrote:
On 23.12.2015 17:14, Andrea Bolognani wrote:
Hi all,
libvirt currently doesn't allow you to configure a guest with something like
<cpu> <topology sockets='2' cores='5' threads='8'/> <numa> <cell id='0' cpus='0-39' memory='1048576' unit='KiB'/> <cell id='16' cpus='40-79' memory='1048576' unit='KiB'/> </numa>
This says you are configuring GUEST NUMA nodes, and last time I checked qemu did not support non-continuous NUMA nodes. So, whatever your HOST topology is, in GUEST you want NUMA nodes to be continuous.
I checked again and, as you and Daniel suggested, QEMU does not support non-contiguous NUMA nodes, eg.
qemu-kvm \ -m 2048 \ -smp 80,sockets=2,cores=5,threads=8 \ -numa node,nodeid=0,cpus=0-39,mem=1024 \ -numa node,nodeid=16,cpus=40-79,mem=1024
outputs
qemu-kvm: numa: Node ID missing: 15
so indeed there's a very good reason for libvirt to reject that NUMA configuration.
Thanks you all for all the input. oVirt will follow this configuration limitation and enforce it then. Cheers.
-- Andrea Bolognani Software Engineer - Virtualization Team
participants (4)
-
Andrea Bolognani -
Daniel P. Berrange -
Michal Privoznik -
Roy Golan