On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang <jyang@redhat.com> wrote:
On 2013年01月24日 14:26, Doug Goldstein wrote:
On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang@redhat.com> wrote:
On 2013年01月24日 12:11, Doug Goldstein wrote:
On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe@gentoo.org> wrote:
I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu 1.2.2 applied on top plus a number of stability patches). Having issue where my VMs fail to start with the following message:
kvm_init_vcpu failed: Cannot allocate memory
Smell likes we have problem on setting the NUMA policy (perhaps caused by the incorrect host NUMA topology), given that the system still has enough memory. Or numad (if it's installed) is doing something wrong.
Can you see if there is something about the Nodeset used to set the policy in debug log?
E.g.
% cat libvirtd.debug | grep Nodeset
Well I don't see anything but its likely because I didn't do something correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose from the command line.
If the process is in background, it's expected you can't see anything
My /etc/libvirt/libvirtd.conf had:
log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I didn't get any debug messages.
log_level=1 has to be set.
Anyway, let's simply do this:
% service libvirtd stop % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
That's what I was doing, minus the tee just to the console and nothing was coming out. Which is why I added the 1:file:/tmp/libvirtd.log, which also didn't get any debug messages. Turns out this instance must have been built with --disable-debug, All I've got in the log is: # grep -i 'numa' libvirtd.debug 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 : About to run /usr/bin/numad -w 2:2048 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 : Nodeset returned from numad: 1 Immediately below that is 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 : Setting up domain cgroup (if required) 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New group /libvirt/qemu/bb-2.6.35.9-i686 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 : Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 : Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 : Make group /libvirt/qemu/bb-2.6.35.9-i686 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 : Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/ 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 : Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/ 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469 : Setting up inheritance /libvirt/qemu -> /libvirt/qemu/bb-2.6.35.9-i686 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 : Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482 : Inherit cpuset.cpus = 0-63 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus' to '0-63' 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 : Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482 : Inherit cpuset.mems = 0-7 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems' to '0-7' 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 : Could not autoset a RSS limit for domain bb-2.6.35.9-i686 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 : Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems' to '1' 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39 Could the RSS issue be related? Some kernel related option not playing nice or enabled? -- Doug Goldstein