Re: [libvirt] [RFC PATCH] NUMA tuning support

On Thu, 2011-05-05 at 17:38 +0800, Osier Yang wrote: > Hi, All, > > This is a simple implenmentation for NUMA tuning support based on binary > program 'numactl', currently only supports to bind memory to specified nodes, > using option "--membind", perhaps it need to support more, but I'd like > send it early so that could make sure if the principle is correct. > > Ideally, NUMA tuning support should be added in qemu-kvm first, such > as they could provide command options, then what we need to do in libvirt > is just to pass the options to qemu-kvm, but unfortunately qemu-kvm doesn't > support it yet, what we could do currently is only to use numactl, > it forks process, a bit expensive than qemu-kvm supports NUMA tuning > inside with libnuma, but it shouldn't affects much I guess. > > The NUMA tuning XML is like: > > <numatune> > <membind nodeset='+0-4,8-12'/> > </numatune> > > Any thoughts/feedback is appreciated. Osier: A couple of thoughts/observations: 1) you can accomplish the same thing -- restricting a domain's memory to a specified set of nodes -- using the cpuset cgroup that is already associated with each domain. E.g., cgset -r cpuset.mems=<nodeset> /libvirt/qemu/<domain> Or the equivalent libcgroup call. However, numactl is more flexible; especially if you intend to support more policies: preferred, interleave. Which leads to the question: 2) Do you really want the full "membind" semantics as opposed to "preferred" by default? Membind policy will restrict the VMs pages to the specified nodeset and will initiate reclaim/stealing and wait for pages to become available or the task is OOM-killed because of mempolicy when all of the nodes in nodeset reach their minimum watermark. Membind works the same as cpuset.mems in this respect. Preferred policy will keep memory allocations [but not vcpu execution] local to the specified set of nodes as long as there is sufficient memory, and will silently "overflow" allocations to other nodes when necessary. I.e., it's a little more forgiving under memory pressure.