Re: [Libvir] Extending libvirt to probe NUMA topology

On Wed, Jun 13, 2007 at 10:40:40AM -0500, Ryan Harper wrote: > Hello all, Hello Ryan, > I wanted to start a discussion on how we might get libvirt to be able to > probe the NUMA topology of Xen and Linux (for QEMU/KVM). In Xen, I've > recently posted patches for exporting topology into the [1]physinfo > hypercall, as well adding a [2]hypercall to probe the Xen heap. I > believe the topology and memory info is already available in Linux. > With these, we have enough information to be able to write some simple > policy above libvirt that can create guests in a NUMA-aware fashion. > > I'd like to suggest the following for discussion: > > (1) A function to discover topology > (2) A function to check available memory > (3) Specifying which cpus to use prior to domain start Okay, but let's start by defining the scope a bit. Historically NUMA have explored various paths, and I assume we are gonna work in a rather small subset of what NUMA (Non Uniform Memory Access) have meant over time. I assume the following, tell me if I'm wrong: - we are just considering memory and processor affinity - the topology, i.e. the affinity between the processors and the various memory areas is fixed and the kind of mapping is rather simple to get into more specifics: - we will need to expand the model of libvirt http://libvirt.org/intro.html to split the Node ressources into separate sets containing processors and memory areas which are highly connected together (assuming the model is a simple partition of the ressources between the equivalent of sub-Nodes) - the function (2) would for a given processor tell how much of its memory is already allocated (to existing running or paused domains) Right ? Is the partition model sufficient for the architectures ? If yes then we will need a new definition and terminology for those sub-Nodes.

For 3 we already have support for pinning the domain virtual CPUs to physical CPUs but I guess it's not sufficient because you want this to be activated from the definition of the domain: http://libvirt.org/html/libvirt-libvirt.html#virDomainPinVcpu So the XML format would have to be extended to allow specifying the subset of processors the domain is supposed to start on:

http://libvirt.org/format.html I would assume that if nothing is specified, the underlying Hypervisor (in libvirt terminology, that could be a linux kernel in practice) will by default try to do the optimal placement by itself, i.e. (3) is only useful if you want to override the default behaviour.