On Fri, Aug 19, 2011 at 12:55 PM, Osier Yang <jyang(a)redhat.com&gt; wrote: But this only specifies the host NUMA policy that should be used for guest VM processes. As I observed above, this controls the NUMA policy of the guest VM threads on host. vcpu/cpuset specifies how vcpu threads should be pinned on host. I think what we are addressing here is a bit different from the above two. Here we are actually trying to _define_ the NUMA topology of the guest, while via other capabilites (numatune, vcpu) we only control the cpu and memory bindings of vcpu threads on host. Hence I am not sure if if <numatune> is the right place for defining host NUMA topology which btw should be independent of the host topology. Thanks for your response. Regards, Bharata.

于 2011年08月19日 16:09, Bharata B Rao 写道: Yes. Yes, I known your meaning. Maybe something like: <numatune> <guest> ...... </guest> </numatune>

On 08/19/2011 01:35 AM, Bharata B Rao wrote: Option 2 (above) seems like the most logical interface to me. I would not support putting this under <numatune> because of the high risk of users confusing guest NUMA topology definition with host NUMA tuning. I like the idea of merging this into <topology> to prevent errors with specifying incompatible cpu and numa topologies but I think you can go a step further (assuming my following assertion is valid). Since cpus are assigned to numa nodes at the core level, and you are providing a 'nodeid' attribute, you can infer the 'cpus' attribute using 'cores' and 'nodeid' alone. For your example above: <topology sockets='1' cores='2' threads='1' nodeid='0' mem='size'> <topology sockets='1' cores='2' threads='1' nodeid='1' mem='size'> You have 4 cores total, each node is assigned 2. Assign cores to nodes starting with core 0 and node 0. -- Adam Litke IBM Linux Technology Center

On Fri, Aug 19, 2011 at 12:05:43PM +0530, Bharata B Rao wrote: I'm not sure it really makes sense having the NUMA memory config inside the <cpu> configuration, but i like the simplicity of of this specification. No matter what we do, libvirt is going to have todo some kind of semantic validation on the different info. This has the problem of redundancy of specification of the sockets, cores & threads, vs the new 'cpus' attribute. eg you can specify wierd configs like: <topology sockets='1' cores='2' threads='1' nodeid='0' cpus='0-1' mem='size'> <topology sockets='2' cores='1' threads='1' nodeid='1' cpus='2-3' mem='size'> Or even bogus configs <topology sockets='1' cores='2' threads='1' nodeid='0' cpus='0-1' mem='size'> <topology sockets='4' cores='1' threads='1' nodeid='1' cpus='2-3' mem='size'> That all said, given our current XML schema, it is inevitable that we will have some level of duplication of information. Some things that are important to consider are how this interacts with possible CPU / memory hotplug in the future, and how we will be able to pin guest NUMA nodes, to host NUMA nodes. For the first point, it might be desirable to create a NUMA topology which supports upto 8 logical CPUs, but only have 2 physical sockets actually plugged in at boot time. Also, I dread to question whether we want to be able to represent a multi-level NUMA topology, or just assume one level. If we want to be able to cope with multi-level topology, can we assume the levels are solely grouping at the socket, or will we have to consider the possibility of NUMA *inside* a socket. In other words, are we associating socket numbers with NUMA nodes, or are we associating logical CPU numbers with NUMA nodes. This is the difference between configuring something like: <vpus>16</vcpus> <cpu> <topology sockets='4' cores='4' threads='1'> </cpu> <numa> <node sockets='0-1' mem='0-1024'/> <node sockets='2-3' mem='1024-2048'/> </numa> vs <vpus>16</vcpus> <cpu> <topology sockets='4' cores='4' threads='1'> </cpu> <numa> <node cpus='0-7' mem='0-1024'/> <node cpus='8-15' mem='1024-2048'/> </numa> vs <vpus>16</vcpus> <cpu> <topology sockets='4' cores='4' threads='1'> </cpu> <numa> <node mems='0-1024'/> <node cpus='0-3'/> <node cpus='4-7'/> </node> <node mems='1024-2048'/> <node cpus='8-11'/> <node cpus='12-15'/> </node> </numa> vs ...more horrible examples... NB, QEMU's -numa argument may well not support some of the things I am talking about here, but we need to consider the real possibilty that QEMU's -numa arg will be extended, or replaced in the future. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

Hi, Here is another attempt at guest NUMA topology XML specification that should work for different NUMA topologies. We already specify the number of sockets, cores and threads a system has by using: <cpu> <topology sockets='2' cores='2' threads='2'> </cpu> For NUMA, we can add the following: <numa> <node cpus='0-3' mems='1024'> <node cpus='4-7' mems='1024'> </numa> Specifying only cpus in the NUMA node specification should be enough to represent most of the topologies. Based on the number of cpus specified in each node, we should be able to work out how many cores and sockets will be part of each node. Only other thing needed is explicit memory specification. I have taken a few example NUMA topologies here and shown how the above specification can help. Magny cours ----------------- Topology desc: http://code.google.com/p/likwid-topology/wiki/AMD_MagnyCours8 <cpu> <topology sockets='4' cores='4' threads='1'> <numa> <node cpus='0-3' mems='1024'> <node cpus='4-7' mems='1024'> <node cpus='8-11' mems='1024'> <node cpus='12-15'mems='1024'> </numa> <cpu> OR if we want to stick to how CPUs get enumerated in real hardware we can specify like this: <cpu> <topology sockets='4' cores='4' threads='1'> <numa> <node cpus='0,2,4,6' mems='1024> <node cpus='8,10,12,14' mems='1024> <node cpus='1,3,5,7' mems='1024'> <node cpus='9,11,13,15' mems='1024'> </numa> <cpu> The above two specifations for Magny Cours aren't perfect because we conveniently converted the multi-level NUMA into sigle level NUMA. System has 2 sockets and 2 NUMA domains consisting of 4 cores in each domain, but we aren't really reflecting this in the topology specification. But does this really matter ? We are still showing 4 distinct NUMA domains. Nehalem ------------ Topology desc: http://code.google.com/p/likwid-topology/wiki/Intel_Nehalem <cpu> <topology sockets='2' cores='4' threads='2'> <numa> <node cpus='0-7' mems='1024'> <node cpus='8-15' mems='1024'> </numa> </cpu> OR if we want to stick to how CPUs get enumerated in real hardware we can specify like this: <cpu> <topology sockets='2' cores='4' threads='2'> <numa> <node cpus='0-3,8-11' mems='1024'> <node cpus='4-7,12-15' mems='1024'> </numa> </cpu> However there is a problem here. The specification isn't granular enough to specify which CPU is part of which core. As you can see in the topology diagram, CPUs 0,8 belong to one core, CPUs 1,9 belong to one core etc. So the whole point of specifying all the CPUs explicitly in the specification gets defeated. Dunnington --------------- Topology desc: 2 nodes, 4 sockets in each node, 6 cores in each socket. <cpu> <topology sockets='8' cores='6 threads='1' <numa> <node cpus='0-23' mems=1024> <node cpus='24-47' mems=1024> </numa> </cpu> Here also there is the same problem. CPUs 0,4,8,12,16,,20 belong to a core but the specifcation doesn't allow for that. So here are some questions that we need to answer: - Can we just go with flat NUMA specification and convert multilevel NUMA into flat NUMA wherever possible (like in the above Magny cours eg) ? - Are there topologies where this doesn't work ? - Isn't it enough to enumerate CPUs serially among cores and sockets and not enumerate them exactly as in real hardware ? Regards, Bharata.