Re: [libvirt-users] VMs fail to start with NUMA configuration

On 2013年01月28日 11:44, Osier Yang wrote: > On 2013年01月26日 01:07, Doug Goldstein wrote: >> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang(a)redhat.com> wrote: >>> On 2013年01月24日 14:26, Doug Goldstein wrote: >>>> >>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang(a)redhat.com> wrote: >>>>> >>>>> On 2013年01月24日 12:11, Doug Goldstein wrote: >>>>>> >>>>>> >>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe(a)gentoo.org> >>>>>> wrote: >>>>>>> >>>>>>> >>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + >>>>>>> qemu >>>>>>> 1.2.2 applied on top plus a number of stability patches). Having >>>>>>> issue >>>>>>> where my VMs fail to start with the following message: >>>>>>> >>>>>>> kvm_init_vcpu failed: Cannot allocate memory >>>>> >>>>> >>>>> >>>>> Smell likes we have problem on setting the NUMA policy (perhaps >>>>> caused by the incorrect host NUMA topology), given that the system >>>>> still has enough memory. Or numad (if it's installed) is doing >>>>> something wrong. >>>>> >>>>> Can you see if there is something about the Nodeset used to set >>>>> the policy in debug log? >>>>> >>>>> E.g. >>>>> >>>>> % cat libvirtd.debug | grep Nodeset >>>> >>>> >>>> Well I don't see anything but its likely because I didn't do something >>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose >>>> from the command line. >>> >>> >>> If the process is in background, it's expected you can't see anything >>> >>> >>> My /etc/libvirt/libvirtd.conf had: >>>> >>>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I didn't >>>> get any debug messages. >>> >>> >>> log_level=1 has to be set. >>> >>> Anyway, let's simply do this: >>> >>> % service libvirtd stop >>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug >>> >> >> That's what I was doing, minus the tee just to the console and nothing >> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log, >> which also didn't get any debug messages. Turns out this instance must >> have been built with --disable-debug, >> >> All I've got in the log is: >> >> # grep -i 'numa' libvirtd.debug >> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 : >> About to run /usr/bin/numad -w 2:2048 >> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 : >> Nodeset returned from numad: 1 > > This looks right. > >> >> Immediately below that is >> >> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 : >> Setting up domain cgroup (if required) >> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New >> group /libvirt/qemu/bb-2.6.35.9-i686 >> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 : >> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in >> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 : >> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 : >> Make group /libvirt/qemu/bb-2.6.35.9-i686 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 : >> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/ >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 : >> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/ >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469 >> : Setting up inheritance /libvirt/qemu -> >> /libvirt/qemu/bb-2.6.35.9-i686 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 : >> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus >> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >> fd 39 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482 >> : Inherit cpuset.cpus = 0-63 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : >> Set value >> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus' >> to '0-63' > > This looks not right, it should be 0-7 instead. > >> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >> fd 39 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 : >> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems >> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >> fd 39 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482 >> : Inherit cpuset.mems = 0-7 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : >> Set value >> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems' >> to '0-7' > > This is right. > >> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >> fd 39 >> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 : >> Could not autoset a RSS limit for domain bb-2.6.35.9-i686 >> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : >> Set value >> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems' >> to '1' > > And it's strange that the cpuset.mems is changed to '1' here.

> >> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed >> fd 39 >> >> Could the RSS issue be related? Some kernel related option not playing >> nice or enabled? Instead, I'm wondering if the problem is caused by the mismatch (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which thus cause the problem for kernel memory management?