Re: [libvirt] cpuset / numa and qemu in TCG mode

On Mon, May 11, 2015 at 10:52:17AM +0200, Martin Kletzander wrote: > On Mon, May 11, 2015 at 10:52:46AM +1000, Tony Breeds wrote: > > > >Hello all, > > This is with reference to OpenStack (nova) bug 1439280 [1]. > > > >The symptom is when nova(-compute) tries to launch an instance/VM it errors out with: > > "libvirtError: Requested operation is not valid: cpu affinity is not supported" > >This only happens with using qemu in TCG mode. > > > > I can see where the error might have come from. QEMU doesn't have > vcpu threads if ran in TCG mode, so if libvirt tries to pin some of > them, this will happen. *But* upstream handles this perfectly. I > just tried and it works. I'll explain why a few lines down the road. Okay, that's good to know. > >After looking at the domain XML and the docs at [2] It seems to me the problem is either: > >1) the libvirt driver in nova is generating valid XML that is an invalid > > domain description ; or > >2) NUMA support in qemu (TCG) mode is broken. > > > > I don't get what this has to do with NUMA, but anyway I just think > this is a bug in older libvirt actually. The NUMA references are baddness on my part. They come from the fact that the code in nova that does the CPUpinning is tied up with NUMA support. So with that on mind I'd reword my summary as: "CPU Pinning in qemu (TCG) mode is broken."

> Yes, that's because we can't differentiate which threads do what with > TCG accel. Because of the you must only specify one pinning for all > threads (I/O threads, CPU threads and the emulator thread), which is > done exactly using the 'cpuset' attribute as that is valid for the > whole machine. Thatnsk to that we don't have to differentiate > anything and just use the cpuset for the whole machine -> no error > should occur there. Ok I think I follow that. > >Have I understood the documentation correctly? If so it would seem that the > >correct fix is in nova to teach the libvirt driver to generate the correct XML > >for this virtulisation type. > > > > You understood the documentation correctly, it was just libvirt acting > up a bit. Okay that good to know. So we can work around this in nova by either: a) Check for libvirt 1.2.15 in the approriate TCG code path and elide the cpuset ; or b) including 'emulatorpin' in the cputune node 'a' is easy to do and is basically what I have proposwed in the bug. Having said that which is the better way to work around libvirt acting up a bit in older versions?

Ther nova side will be pretty easy regardless. I'd say the best solution would be to back port the 'fix' but that seems like a lot of effort given the number of distros and libvirt versions potentiall involved.

> I'm still not sure why are you bringing NUMA into the mix as there is > nothing NUMA related in the bug or this mail. But to complete my > answer, even though there is no practical benefit to using NUMA on TCG > accelerated QEMU machine, it should still work. Okay so as I said above NUMA as such is a mistake. Thank tyou for confirming that when the correct XML is generated the instance/guest/VM works just fine. Yours Tony.