Re: [libvirt-users] VMs fail to start with NUMA configuration

On 2013年01月29日 00:17, Doug Goldstein wrote: > > On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang&lt;jyang(a)redhat.com&gt; wrote: >> >> On 2013年01月28日 11:47, Osier Yang wrote: >>> >>> >>> On 2013年01月28日 11:44, Osier Yang wrote: >>>> >>>> >>>> On 2013年01月26日 01:07, Doug Goldstein wrote: >>>>> >>>>> >>>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang&lt;jyang(a)redhat.com&gt; wrote: >>>>>> >>>>>> >>>>>> On 2013年01月24日 14:26, Doug Goldstein wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang&lt;jyang(a)redhat.com&gt; >>>>>>> wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein&lt;cardoe(a)gentoo.org&gt; >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + >>>>>>>>>> qemu >>>>>>>>>> 1.2.2 applied on top plus a number of stability patches). Having >>>>>>>>>> issue >>>>>>>>>> where my VMs fail to start with the following message: >>>>>>>>>> >>>>>>>>>> kvm_init_vcpu failed: Cannot allocate memory >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Smell likes we have problem on setting the NUMA policy (perhaps >>>>>>>> caused by the incorrect host NUMA topology), given that the system >>>>>>>> still has enough memory. Or numad (if it's installed) is doing >>>>>>>> something wrong. >>>>>>>> >>>>>>>> Can you see if there is something about the Nodeset used to set >>>>>>>> the policy in debug log? >>>>>>>> >>>>>>>> E.g. >>>>>>>> >>>>>>>> % cat libvirtd.debug | grep Nodeset >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> Well I don't see anything but its likely because I didn't do >>>>>>> something >>>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose >>>>>>> from the command line. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> If the process is in background, it's expected you can't see anything >>>>>> >>>>>> >>>>>> My /etc/libvirt/libvirtd.conf had: >>>>>>> >>>>>>> >>>>>>> >>>>>>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I >>>>>>> didn't >>>>>>> get any debug messages. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> log_level=1 has to be set. >>>>>> >>>>>> Anyway, let's simply do this: >>>>>> >>>>>> % service libvirtd stop >>>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug >>>>>> >>>>> >>>>> That's what I was doing, minus the tee just to the console and nothing >>>>> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log, >>>>> which also didn't get any debug messages. Turns out this instance must >>>>> have been built with --disable-debug, >>>>> >>>>> All I've got in the log is: >>>>> >>>>> # grep -i 'numa' libvirtd.debug >>>>> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 : >>>>> About to run /usr/bin/numad -w 2:2048 >>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 : >>>>> Nodeset returned from numad: 1 >>>> >>>> >>>> >>>> This looks right. >>>> >>>>> >>>>> Immediately below that is >>>>> >>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 : >>>>> Setting up domain cgroup (if required) >>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New >>>>> group /libvirt/qemu/bb-2.6.35.9-i686 >>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 : >>>>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in >>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 : >>>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 : >>>>> Make group /libvirt/qemu/bb-2.6.35.9-i686 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 : >>>>> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/ >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 : >>>>> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/ >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469 >>>>> : Setting up inheritance /libvirt/qemu -> >>>>> /libvirt/qemu/bb-2.6.35.9-i686 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 : >>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >>>>> fd 39 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482 >>>>> : Inherit cpuset.cpus = 0-63 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : >>>>> Set value >>>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus' >>>>> to '0-63' >>>> >>>> >>>> >>>> This looks not right, it should be 0-7 instead. >>>> >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >>>>> fd 39 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 : >>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >>>>> fd 39 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482 >>>>> : Inherit cpuset.mems = 0-7 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : >>>>> Set value >>>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems' >>>>> to '0-7' >>>> >>>> >>>> >>>> This is right. >>>> >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >>>>> fd 39 >>>>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 : >>>>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686 >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : >>>>> Set value >>>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems' >>>>> to '1' >>>> >>>> >>>> >>>> And it's strange that the cpuset.mems is changed to '1' here. >> >> >> >> Oh, actually this is right, cpuset.mems is about the memory nodes. >> >> >>>> >>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >>>>> fd 39 >>>>> >>>>> Could the RSS issue be related? Some kernel related option not playing >>>>> nice or enabled? >>> >>> >>> >>> Instead, I'm wondering if the problem is caused by the mismatch >>> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which >>> thus cause the problem for kernel memory management? >> >> >> >> So, the simple method to prove the guess is to use static placement >> like: >> >> <vcpu placement='static' cpuset='0-63'>2</vcpu> >> <numatune> >> <memory placement='static' nodeset='1'/> >> </numatune> >> >> Osier > > > Same error. Which I don't know if you expected or didn't expect. > It's expected. as "0-63" is the final result when using "auto" placement.