3:45 p.m.
I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
1.2.2 applied on top plus a number of stability patches). Having issue
where my VMs fail to start with the following message:
kvm_init_vcpu failed: Cannot allocate memory
Following the instructions at
http://libvirt.org/formatdomain.html#elementsNUMATuning I've added the
following to my VCPU configuration:
<vcpu placement='auto'>2</vcpu>
Which libvirt expands out as I'd expect per the documentation to:
<vcpu placement='auto'>2</vcpu>
<numatune>
<memory tune='strict' placement='auto'/>
</numatune>
However, the VMs won't start and the system is no low on memory.
# numactl --hardware
available: 8 nodes (0-7)
node 0 cpus: 0 4 8 12 16 20 24 28
node 0 size: 16374 MB
node 0 free: 11899 MB
node 1 cpus: 32 36 40 44 48 52 56 60
node 1 size: 16384 MB
node 1 free: 15318 MB
node 2 cpus: 2 6 10 14 18 22 26 30
node 2 size: 16384 MB
node 2 free: 15766 MB
node 3 cpus: 34 38 42 46 50 54 58 62
node 3 size: 16384 MB
node 3 free: 15347 MB
node 4 cpus: 3 7 11 15 19 23 27 31
node 4 size: 16384 MB
node 4 free: 15041 MB
node 5 cpus: 35 39 43 47 51 55 59 63
node 5 size: 16384 MB
node 5 free: 15202 MB
node 6 cpus: 1 5 9 13 17 21 25 29
node 6 size: 16384 MB
node 6 free: 15197 MB
node 7 cpus: 33 37 41 45 49 53 57 61
node 7 size: 16368 MB
node 7 free: 15669 MB
The system has 4 Opteron 6272 which add up to a total of 64 cores, 16
cores per socket. These are the CPUs that Dan B noticed issues with in
the past regarding topology but I thought this was resolved. But the
capabilities are posted below, you'll notice the topology is
incorrect.
# virsh capabilities
<capabilities>
<host>
<uuid>44454c4c-5300-1038-8031-c4c04f545331</uuid>
<cpu>
<arch>x86_64</arch>
<model>Opteron_G4</model>
<vendor>AMD</vendor>
<topology sockets='1' cores='8' threads='2'/>
<feature name='nodeid_msr'/>
<feature name='wdt'/>
<feature name='skinit'/>
<feature name='ibs'/>
<feature name='osvw'/>
<feature name='cr8legacy'/>
<feature name='extapic'/>
<feature name='cmp_legacy'/>
<feature name='fxsr_opt'/>
<feature name='mmxext'/>
<feature name='osxsave'/>
<feature name='monitor'/>
<feature name='ht'/>
<feature name='vme'/>
</cpu>
<power_management>
<suspend_disk/>
</power_management>
<migration_features>
<live/>
<uri_transports>
<uri_transport>tcp</uri_transport>
</uri_transports>
</migration_features>
<topology>
<cells num='8'>
<cell id='0'>
<cpus num='8'>
<cpu id='0'/>
<cpu id='4'/>
<cpu id='8'/>
<cpu id='12'/>
<cpu id='16'/>
<cpu id='20'/>
<cpu id='24'/>
<cpu id='28'/>
</cpus>
</cell>
<cell id='1'>
<cpus num='8'>
<cpu id='32'/>
<cpu id='36'/>
<cpu id='40'/>
<cpu id='44'/>
<cpu id='48'/>
<cpu id='52'/>
<cpu id='56'/>
<cpu id='60'/>
</cpus>
</cell>
<cell id='2'>
<cpus num='8'>
<cpu id='2'/>
<cpu id='6'/>
<cpu id='10'/>
<cpu id='14'/>
<cpu id='18'/>
<cpu id='22'/>
<cpu id='26'/>
<cpu id='30'/>
</cpus>
</cell>
<cell id='3'>
<cpus num='8'>
<cpu id='34'/>
<cpu id='38'/>
<cpu id='42'/>
<cpu id='46'/>
<cpu id='50'/>
<cpu id='54'/>
<cpu id='58'/>
<cpu id='62'/>
</cpus>
</cell>
<cell id='4'>
<cpus num='8'>
<cpu id='3'/>
<cpu id='7'/>
<cpu id='11'/>
<cpu id='15'/>
<cpu id='19'/>
<cpu id='23'/>
<cpu id='27'/>
<cpu id='31'/>
</cpus>
</cell>
<cell id='5'>
<cpus num='8'>
<cpu id='35'/>
<cpu id='39'/>
<cpu id='43'/>
<cpu id='47'/>
<cpu id='51'/>
<cpu id='55'/>
<cpu id='59'/>
<cpu id='63'/>
</cpus>
</cell>
<cell id='6'>
<cpus num='8'>
<cpu id='1'/>
<cpu id='5'/>
<cpu id='9'/>
<cpu id='13'/>
<cpu id='17'/>
<cpu id='21'/>
<cpu id='25'/>
<cpu id='29'/>
</cpus>
</cell>
<cell id='7'>
<cpus num='8'>
<cpu id='33'/>
<cpu id='37'/>
<cpu id='41'/>
<cpu id='45'/>
<cpu id='49'/>
<cpu id='53'/>
<cpu id='57'/>
<cpu id='61'/>
</cpus>
</cell>
</cells>
</topology>
<secmodel>
<model>none</model>
<doi>0</doi>
</secmodel>
<secmodel>
<model>dac</model>
<doi>0</doi>
</secmodel>
</host>
<guest>
<os_type>hvm</os_type>
<arch name='i686'>
<wordsize>32</wordsize>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
<domain type='qemu'>
</domain>
<domain type='kvm'>
<emulator>/usr/bin/qemu-kvm</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<pae/>
<nonpae/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
</features>
</guest>
<guest>
<os_type>hvm</os_type>
<arch name='x86_64'>
<wordsize>64</wordsize>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
<domain type='qemu'>
</domain>
<domain type='kvm'>
<emulator>/usr/bin/qemu-kvm</emulator>
<machine>pc-1.2</machine>
<machine canonical='pc-1.2'>pc</machine>
<machine>pc-1.1</machine>
<machine>pc-1.0</machine>
<machine>pc-0.15</machine>
<machine>pc-0.14</machine>
<machine>pc-0.13</machine>
<machine>pc-0.12</machine>
<machine>pc-0.11</machine>
<machine>pc-0.10</machine>
<machine>isapc</machine>
<machine>none</machine>
</domain>
</arch>
<features>
<cpuselection/>
<deviceboot/>
<acpi default='on' toggle='yes'/>
<apic default='on' toggle='no'/>
</features>
</guest>
</capabilities>
Any suggestions are appreciated.
--
Doug Goldstein
10:11 p.m.
On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein <cardoe(a)gentoo.org> wrote:
I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 +
qemu
1.2.2 applied on top plus a number of stability patches). Having issue
where my VMs fail to start with the following message:
kvm_init_vcpu failed: Cannot allocate memory
I've tried libvirt-1.0.1 on this machine and had the same results. The
capabilities have been fixed on that machine however and now display:
<topology sockets='1' cores='64' threads='1'/>
I also forgot to mention that LIBVIRT_DEBUG=1 and running the daemon
with --verbose didn't have more details than what I originally posted.
--
Doug Goldstein
11:02 p.m.
On 2013年01月24日 12:11, Doug Goldstein wrote:
On Wed, Jan 23, 2013 at 3:45 PM, Doug
Goldstein<cardoe(a)gentoo.org> wrote:
> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
> 1.2.2 applied on top plus a number of stability patches). Having issue
> where my VMs fail to start with the following message:
>
> kvm_init_vcpu failed: Cannot allocate memory
Smell likes we have problem on setting the NUMA policy (perhaps
caused by the incorrect host NUMA topology), given that the system
still has enough memory. Or numad (if it's installed) is doing
something wrong.
Can you see if there is something about the Nodeset used to set
the policy in debug log?
E.g.
% cat libvirtd.debug | grep Nodeset
% cat /var/log/libvirt/qemu/$guest.log
On the other hand, I'm wondering if there is a chance to see the
full error msg kernel throws out.
I've tried libvirt-1.0.1 on this machine and had the same results. The
capabilities have been fixed on that machine however and now display:
<topology sockets='1' cores='64' threads='1'/>
I also forgot to mention that LIBVIRT_DEBUG=1 and running the daemon
with --verbose didn't have more details than what I originally posted.
12:26 a.m.
On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang <jyang(a)redhat.com> wrote:
On 2013年01月24日 12:11, Doug Goldstein wrote:
>
> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe(a)gentoo.org> wrote:
>>
>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
>> 1.2.2 applied on top plus a number of stability patches). Having issue
>> where my VMs fail to start with the following message:
>>
>> kvm_init_vcpu failed: Cannot allocate memory
Smell likes we have problem on setting the NUMA policy (perhaps
caused by the incorrect host NUMA topology), given that the system
still has enough memory. Or numad (if it's installed) is doing
something wrong.
Can you see if there is something about the Nodeset used to set
the policy in debug log?
E.g.
% cat libvirtd.debug | grep Nodeset
Well I don't see anything but its likely because I didn't do something
correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
from the command line. My /etc/libvirt/libvirtd.conf had:
log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I didn't
get any debug messages.
% cat /var/log/libvirt/qemu/$guest.log
Nothing in here. Just the command line:
2013-01-24 04:00:49.989+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/
bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3 HOME=/root USER=root LOGNAME=root QEM
U_AUDIO_DRV=none /usr/bin/qemu-kvm -name bb-2.6.18-128.el5.x86_64 -S -M pc-1.2 -
cpu Opteron_G4,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+lwp,+wdt,+skinit,
+ibs,+osvw,+cr8legacy,+extapic,+cmp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+
ht,+vme -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -uuid
2b5990f0-649f-ae25-99f0-dc4b05f682e1 -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/bb-2.6.18-128.el5.x86_64.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -no-shutdown -boot menu=off -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/dev/disk/by-path/ip-192.168.200.20:3260-iscsi-iqn.2011-07.lab.san-1:2.6.18-128.el5-x86_64-lun-0,if=none,id=drive-ide0-0-0,format=raw,cache=none,werror=stop,rerror=stop,aio=native
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-netdev tap,fd=31,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7c:b2:81,bus=pci.0,addr=0x7
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:10,password
-vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device
i6300esb,id=watchdog0,bus=pci.0,addr=0x6 -watchdog-action reset
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/12
CPU feature perfctr_nb not found
CPU feature perfctr_core not found
CPU feature topoext not found
CPU feature lwp not found
kvm_init_vcpu failed: Cannot allocate memory
2013-01-24 04:00:50.194+0000: shutting down
On the other hand, I'm wondering if there is a chance to see the
full error msg kernel throws out.
That's what I've been trying to figure out. Trying to get some more
info out of libvirt as well. If you've got any ideas I'll give them a
shot.
--
Doug Goldstein
12:58 a.m.
On 2013年01月24日 14:26, Doug Goldstein wrote:
On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang(a)redhat.com>
wrote:
> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>
>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe(a)gentoo.org>
wrote:
>>>
>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
>>> 1.2.2 applied on top plus a number of stability patches). Having issue
>>> where my VMs fail to start with the following message:
>>>
>>> kvm_init_vcpu failed: Cannot allocate memory
>
>
> Smell likes we have problem on setting the NUMA policy (perhaps
> caused by the incorrect host NUMA topology), given that the system
> still has enough memory. Or numad (if it's installed) is doing
> something wrong.
>
> Can you see if there is something about the Nodeset used to set
> the policy in debug log?
>
> E.g.
>
> % cat libvirtd.debug | grep Nodeset
Well I don't see anything but its likely because I didn't do something
correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
from the command line.
If the process is in background, it's expected you can't see anything
My /etc/libvirt/libvirtd.conf had:
log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log"
But I didn't
get any debug messages.
log_level=1 has to be set.
Anyway, let's simply do this:
% service libvirtd stop
% LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
> % cat /var/log/libvirt/qemu/$guest.log
Nothing in here. Just the command line:
2013-01-24 04:00:49.989+0000: starting up
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/
bin:/usr/x86_64-pc-linux-gnu/gcc-bin/4.6.3 HOME=/root USER=root LOGNAME=root QEM
U_AUDIO_DRV=none /usr/bin/qemu-kvm -name bb-2.6.18-128.el5.x86_64 -S -M pc-1.2 -
cpu Opteron_G4,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+lwp,+wdt,+skinit,
+ibs,+osvw,+cr8legacy,+extapic,+cmp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+
ht,+vme -enable-kvm -m 2048 -smp 2,sockets=1,cores=2,threads=1 -uuid
2b5990f0-649f-ae25-99f0-dc4b05f682e1 -no-user-config -nodefaults
-chardev
socket,id=charmonitor,path=/var/lib/libvirt/qemu/bb-2.6.18-128.el5.x86_64.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=utc,driftfix=slew -no-shutdown -boot menu=off -device
piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive
file=/dev/disk/by-path/ip-192.168.200.20:3260-iscsi-iqn.2011-07.lab.san-1:2.6.18-128.el5-x86_64-lun-0,if=none,id=drive-ide0-0-0,format=raw,cache=none,werror=stop,rerror=stop,aio=native
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-netdev tap,fd=31,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7c:b2:81,bus=pci.0,addr=0x7
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -vnc 0.0.0.0:10,password
-vga cirrus -device AC97,id=sound0,bus=pci.0,addr=0x4 -device
i6300esb,id=watchdog0,bus=pci.0,addr=0x6 -watchdog-action reset
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
char device redirected to /dev/pts/12
CPU feature perfctr_nb not found
CPU feature perfctr_core not found
CPU feature topoext not found
CPU feature lwp not found
kvm_init_vcpu failed: Cannot allocate memory
2013-01-24 04:00:50.194+0000: shutting down
>
> On the other hand, I'm wondering if there is a chance to see the
> full error msg kernel throws out.
>
That's what I've been trying to figure out. Trying to get some more
info out of libvirt as well. If you've got any ideas I'll give them a
shot.
11:07 a.m.
On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang <jyang(a)redhat.com> wrote:
On 2013年01月24日 14:26, Doug Goldstein wrote:
>
> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang(a)redhat.com> wrote:
>>
>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>
>>>
>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe(a)gentoo.org>
>>> wrote:
>>>>
>>>>
>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 + qemu
>>>> 1.2.2 applied on top plus a number of stability patches). Having issue
>>>> where my VMs fail to start with the following message:
>>>>
>>>> kvm_init_vcpu failed: Cannot allocate memory
>>
>>
>>
>> Smell likes we have problem on setting the NUMA policy (perhaps
>> caused by the incorrect host NUMA topology), given that the system
>> still has enough memory. Or numad (if it's installed) is doing
>> something wrong.
>>
>> Can you see if there is something about the Nodeset used to set
>> the policy in debug log?
>>
>> E.g.
>>
>> % cat libvirtd.debug | grep Nodeset
>
>
> Well I don't see anything but its likely because I didn't do something
> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
> from the command line.
If the process is in background, it's expected you can't see anything
My /etc/libvirt/libvirtd.conf had:
>
> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I didn't
> get any debug messages.
log_level=1 has to be set.
Anyway, let's simply do this:
% service libvirtd stop
% LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
That's what I was doing, minus the tee just to the console and nothing
was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
which also didn't get any debug messages. Turns out this instance must
have been built with --disable-debug,
All I've got in the log is:
# grep -i 'numa' libvirtd.debug
2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
About to run /usr/bin/numad -w 2:2048
2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
Nodeset returned from numad: 1
Immediately below that is
2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
Setting up domain cgroup (if required)
2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
group /libvirt/qemu/bb-2.6.35.9-i686
2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
Make group /libvirt/qemu/bb-2.6.35.9-i686
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
: Setting up inheritance /libvirt/qemu ->
/libvirt/qemu/bb-2.6.35.9-i686
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
: Inherit cpuset.cpus = 0-63
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
to '0-63'
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
: Inherit cpuset.mems = 0-7
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
to '0-7'
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
Could not autoset a RSS limit for domain bb-2.6.35.9-i686
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
to '1'
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
Could the RSS issue be related? Some kernel related option not playing
nice or enabled?
--
Doug Goldstein
9:44 p.m.
On 2013年01月26日 01:07, Doug Goldstein wrote:
On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang(a)redhat.com>
wrote:
> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>
>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang(a)redhat.com> wrote:
>>>
>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>
>>>>
>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug Goldstein<cardoe(a)gentoo.org>
>>>> wrote:
>>>>>
>>>>>
>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 +
qemu
>>>>> 1.2.2 applied on top plus a number of stability patches). Having
issue
>>>>> where my VMs fail to start with the following message:
>>>>>
>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>
>>>
>>>
>>> Smell likes we have problem on setting the NUMA policy (perhaps
>>> caused by the incorrect host NUMA topology), given that the system
>>> still has enough memory. Or numad (if it's installed) is doing
>>> something wrong.
>>>
>>> Can you see if there is something about the Nodeset used to set
>>> the policy in debug log?
>>>
>>> E.g.
>>>
>>> % cat libvirtd.debug | grep Nodeset
>>
>>
>> Well I don't see anything but its likely because I didn't do something
>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
>> from the command line.
>
>
> If the process is in background, it's expected you can't see anything
>
>
> My /etc/libvirt/libvirtd.conf had:
>>
>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I
didn't
>> get any debug messages.
>
>
> log_level=1 has to be set.
>
> Anyway, let's simply do this:
>
> % service libvirtd stop
> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
>
That's what I was doing, minus the tee just to the console and nothing
was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
which also didn't get any debug messages. Turns out this instance must
have been built with --disable-debug,
All I've got in the log is:
# grep -i 'numa' libvirtd.debug
2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
About to run /usr/bin/numad -w 2:2048
2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
Nodeset returned from numad: 1
This looks right.
Immediately below that is
2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
Setting up domain cgroup (if required)
2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
group /libvirt/qemu/bb-2.6.35.9-i686
2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
Make group /libvirt/qemu/bb-2.6.35.9-i686
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
: Setting up inheritance /libvirt/qemu ->
/libvirt/qemu/bb-2.6.35.9-i686
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
: Inherit cpuset.cpus = 0-63
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
to '0-63'
This looks not right, it should be 0-7 instead.
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
fd 39
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
: Inherit cpuset.mems = 0-7
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
to '0-7'
This is right.
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
fd 39
2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
Could not autoset a RSS limit for domain bb-2.6.35.9-i686
2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
Set value '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
to '1'
And it's strange that the cpuset.mems is changed to '1' here.
2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
fd 39
Could the RSS issue be related? Some kernel related option not playing
nice or enabled?
9:47 p.m.
On 2013年01月28日 11:44, Osier Yang wrote:
On 2013年01月26日 01:07, Doug Goldstein wrote:
> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang(a)redhat.com> wrote:
>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>
>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang(a)redhat.com> wrote:
>>>>
>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
Goldstein<cardoe(a)gentoo.org>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm 1.2.0 +
>>>>>> qemu
>>>>>> 1.2.2 applied on top plus a number of stability patches). Having
>>>>>> issue
>>>>>> where my VMs fail to start with the following message:
>>>>>>
>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>
>>>>
>>>>
>>>> Smell likes we have problem on setting the NUMA policy (perhaps
>>>> caused by the incorrect host NUMA topology), given that the system
>>>> still has enough memory. Or numad (if it's installed) is doing
>>>> something wrong.
>>>>
>>>> Can you see if there is something about the Nodeset used to set
>>>> the policy in debug log?
>>>>
>>>> E.g.
>>>>
>>>> % cat libvirtd.debug | grep Nodeset
>>>
>>>
>>> Well I don't see anything but its likely because I didn't do
something
>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
>>> from the command line.
>>
>>
>> If the process is in background, it's expected you can't see anything
>>
>>
>> My /etc/libvirt/libvirtd.conf had:
>>>
>>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I
didn't
>>> get any debug messages.
>>
>>
>> log_level=1 has to be set.
>>
>> Anyway, let's simply do this:
>>
>> % service libvirtd stop
>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
>>
>
> That's what I was doing, minus the tee just to the console and nothing
> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
> which also didn't get any debug messages. Turns out this instance must
> have been built with --disable-debug,
>
> All I've got in the log is:
>
> # grep -i 'numa' libvirtd.debug
> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
> About to run /usr/bin/numad -w 2:2048
> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
> Nodeset returned from numad: 1
This looks right.
>
> Immediately below that is
>
> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
> Setting up domain cgroup (if required)
> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
> group /libvirt/qemu/bb-2.6.35.9-i686
> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
> Make group /libvirt/qemu/bb-2.6.35.9-i686
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
> : Setting up inheritance /libvirt/qemu ->
> /libvirt/qemu/bb-2.6.35.9-i686
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
> : Inherit cpuset.cpus = 0-63
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
> Set value
> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
> to '0-63'
This looks not right, it should be 0-7 instead.
> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
> : Inherit cpuset.mems = 0-7
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
> Set value
> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
> to '0-7'
This is right.
> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
> Set value
> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
> to '1'
And it's strange that the cpuset.mems is changed to '1' here.
> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed fd 39
>
> Could the RSS issue be related? Some kernel related option not playing
> nice or enabled?
Instead, I'm wondering if the problem is caused by the mismatch
(from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
thus cause the problem for kernel memory management?
Osier
10:46 p.m.
On 2013年01月28日 11:47, Osier Yang wrote:
On 2013年01月28日 11:44, Osier Yang wrote:
> On 2013年01月26日 01:07, Doug Goldstein wrote:
>> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang(a)redhat.com> wrote:
>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>
>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang(a)redhat.com>
wrote:
>>>>>
>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
Goldstein<cardoe(a)gentoo.org>
>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm
1.2.0 +
>>>>>>> qemu
>>>>>>> 1.2.2 applied on top plus a number of stability patches).
Having
>>>>>>> issue
>>>>>>> where my VMs fail to start with the following message:
>>>>>>>
>>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>>
>>>>>
>>>>>
>>>>> Smell likes we have problem on setting the NUMA policy (perhaps
>>>>> caused by the incorrect host NUMA topology), given that the system
>>>>> still has enough memory. Or numad (if it's installed) is doing
>>>>> something wrong.
>>>>>
>>>>> Can you see if there is something about the Nodeset used to set
>>>>> the policy in debug log?
>>>>>
>>>>> E.g.
>>>>>
>>>>> % cat libvirtd.debug | grep Nodeset
>>>>
>>>>
>>>> Well I don't see anything but its likely because I didn't do
something
>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
>>>> from the command line.
>>>
>>>
>>> If the process is in background, it's expected you can't see
anything
>>>
>>>
>>> My /etc/libvirt/libvirtd.conf had:
>>>>
>>>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log" But I
didn't
>>>> get any debug messages.
>>>
>>>
>>> log_level=1 has to be set.
>>>
>>> Anyway, let's simply do this:
>>>
>>> % service libvirtd stop
>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
>>>
>>
>> That's what I was doing, minus the tee just to the console and nothing
>> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
>> which also didn't get any debug messages. Turns out this instance must
>> have been built with --disable-debug,
>>
>> All I've got in the log is:
>>
>> # grep -i 'numa' libvirtd.debug
>> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
>> About to run /usr/bin/numad -w 2:2048
>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
>> Nodeset returned from numad: 1
>
> This looks right.
>
>>
>> Immediately below that is
>>
>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
>> Setting up domain cgroup (if required)
>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
>> group /libvirt/qemu/bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
>> : Setting up inheritance /libvirt/qemu ->
>> /libvirt/qemu/bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>> fd 39
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>> : Inherit cpuset.cpus = 0-63
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>> Set value
>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>> to '0-63'
>
> This looks not right, it should be 0-7 instead.
>
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>> fd 39
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>> fd 39
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>> : Inherit cpuset.mems = 0-7
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>> Set value
>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>> to '0-7'
>
> This is right.
>
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>> fd 39
>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>> Set value
>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>> to '1'
>
> And it's strange that the cpuset.mems is changed to '1' here.
Oh, actually this is right, cpuset.mems is about the memory nodes.
>
>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>> fd 39
>>
>> Could the RSS issue be related? Some kernel related option not playing
>> nice or enabled?
Instead, I'm wondering if the problem is caused by the mismatch
(from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
thus cause the problem for kernel memory management?
So, the simple method to prove the guess is to use static placement
like:
<vcpu placement='static' cpuset='0-63'>2</vcpu>
<numatune>
<memory placement='static' nodeset='1'/>
</numatune>
Osier
10:17 a.m.
On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang <jyang(a)redhat.com> wrote:
On 2013年01月28日 11:47, Osier Yang wrote:
>
> On 2013年01月28日 11:44, Osier Yang wrote:
>>
>> On 2013年01月26日 01:07, Doug Goldstein wrote:
>>>
>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang(a)redhat.com> wrote:
>>>>
>>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>>
>>>>>
>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier Yang<jyang(a)redhat.com>
wrote:
>>>>>>
>>>>>>
>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
Goldstein<cardoe(a)gentoo.org>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2 (qemu-kvm
1.2.0 +
>>>>>>>> qemu
>>>>>>>> 1.2.2 applied on top plus a number of stability patches).
Having
>>>>>>>> issue
>>>>>>>> where my VMs fail to start with the following message:
>>>>>>>>
>>>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Smell likes we have problem on setting the NUMA policy (perhaps
>>>>>> caused by the incorrect host NUMA topology), given that the
system
>>>>>> still has enough memory. Or numad (if it's installed) is
doing
>>>>>> something wrong.
>>>>>>
>>>>>> Can you see if there is something about the Nodeset used to set
>>>>>> the policy in debug log?
>>>>>>
>>>>>> E.g.
>>>>>>
>>>>>> % cat libvirtd.debug | grep Nodeset
>>>>>
>>>>>
>>>>>
>>>>> Well I don't see anything but its likely because I didn't do
something
>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd --verbose
>>>>> from the command line.
>>>>
>>>>
>>>>
>>>> If the process is in background, it's expected you can't see
anything
>>>>
>>>>
>>>> My /etc/libvirt/libvirtd.conf had:
>>>>>
>>>>>
>>>>> log_outputs="3:syslog:libvirtd 1:file:/tmp/libvirtd.log"
But I didn't
>>>>> get any debug messages.
>>>>
>>>>
>>>>
>>>> log_level=1 has to be set.
>>>>
>>>> Anyway, let's simply do this:
>>>>
>>>> % service libvirtd stop
>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a libvirtd.debug
>>>>
>>>
>>> That's what I was doing, minus the tee just to the console and nothing
>>> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
>>> which also didn't get any debug messages. Turns out this instance must
>>> have been built with --disable-debug,
>>>
>>> All I've got in the log is:
>>>
>>> # grep -i 'numa' libvirtd.debug
>>> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
>>> About to run /usr/bin/numad -w 2:2048
>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
>>> Nodeset returned from numad: 1
>>
>>
>> This looks right.
>>
>>>
>>> Immediately below that is
>>>
>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
>>> Setting up domain cgroup (if required)
>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
>>> group /libvirt/qemu/bb-2.6.35.9-i686
>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
>>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
>>> : Setting up inheritance /libvirt/qemu ->
>>> /libvirt/qemu/bb-2.6.35.9-i686
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>> fd 39
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>>> : Inherit cpuset.cpus = 0-63
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>> Set value
>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>>> to '0-63'
>>
>>
>> This looks not right, it should be 0-7 instead.
>>
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>> fd 39
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>> fd 39
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>>> : Inherit cpuset.mems = 0-7
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>> Set value
>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>> to '0-7'
>>
>>
>> This is right.
>>
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>> fd 39
>>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
>>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>> Set value
>>> '/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>> to '1'
>>
>>
>> And it's strange that the cpuset.mems is changed to '1' here.
Oh, actually this is right, cpuset.mems is about the memory nodes.
>>
>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>> fd 39
>>>
>>> Could the RSS issue be related? Some kernel related option not playing
>>> nice or enabled?
>
>
> Instead, I'm wondering if the problem is caused by the mismatch
> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
> thus cause the problem for kernel memory management?
So, the simple method to prove the guess is to use static placement
like:
<vcpu placement='static' cpuset='0-63'>2</vcpu>
<numatune>
<memory placement='static' nodeset='1'/>
</numatune>
Osier
Same error. Which I don't know if you expected or didn't expect.
--
Doug Goldstein
10:23 a.m.
On 2013年01月29日 00:17, Doug Goldstein wrote:
On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang<jyang(a)redhat.com>
wrote:
> On 2013年01月28日 11:47, Osier Yang wrote:
>>
>> On 2013年01月28日 11:44, Osier Yang wrote:
>>>
>>> On 2013年01月26日 01:07, Doug Goldstein wrote:
>>>>
>>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang(a)redhat.com>
wrote:
>>>>>
>>>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>>>
>>>>>>
>>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier
Yang<jyang(a)redhat.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
Goldstein<cardoe(a)gentoo.org>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2
(qemu-kvm 1.2.0 +
>>>>>>>>> qemu
>>>>>>>>> 1.2.2 applied on top plus a number of stability
patches). Having
>>>>>>>>> issue
>>>>>>>>> where my VMs fail to start with the following
message:
>>>>>>>>>
>>>>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Smell likes we have problem on setting the NUMA policy
(perhaps
>>>>>>> caused by the incorrect host NUMA topology), given that the
system
>>>>>>> still has enough memory. Or numad (if it's installed) is
doing
>>>>>>> something wrong.
>>>>>>>
>>>>>>> Can you see if there is something about the Nodeset used to
set
>>>>>>> the policy in debug log?
>>>>>>>
>>>>>>> E.g.
>>>>>>>
>>>>>>> % cat libvirtd.debug | grep Nodeset
>>>>>>
>>>>>>
>>>>>>
>>>>>> Well I don't see anything but its likely because I didn't
do something
>>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd
--verbose
>>>>>> from the command line.
>>>>>
>>>>>
>>>>>
>>>>> If the process is in background, it's expected you can't see
anything
>>>>>
>>>>>
>>>>> My /etc/libvirt/libvirtd.conf had:
>>>>>>
>>>>>>
>>>>>> log_outputs="3:syslog:libvirtd
1:file:/tmp/libvirtd.log" But I didn't
>>>>>> get any debug messages.
>>>>>
>>>>>
>>>>>
>>>>> log_level=1 has to be set.
>>>>>
>>>>> Anyway, let's simply do this:
>>>>>
>>>>> % service libvirtd stop
>>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a
libvirtd.debug
>>>>>
>>>>
>>>> That's what I was doing, minus the tee just to the console and
nothing
>>>> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
>>>> which also didn't get any debug messages. Turns out this instance
must
>>>> have been built with --disable-debug,
>>>>
>>>> All I've got in the log is:
>>>>
>>>> # grep -i 'numa' libvirtd.debug
>>>> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
>>>> About to run /usr/bin/numad -w 2:2048
>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
>>>> Nodeset returned from numad: 1
>>>
>>>
>>> This looks right.
>>>
>>>>
>>>> Immediately below that is
>>>>
>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
>>>> Setting up domain cgroup (if required)
>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
>>>> group /libvirt/qemu/bb-2.6.35.9-i686
>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
>>>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>>> Make controller /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>>> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:469
>>>> : Setting up inheritance /libvirt/qemu ->
>>>> /libvirt/qemu/bb-2.6.35.9-i686
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>> fd 39
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>>>> : Inherit cpuset.cpus = 0-63
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>>> Set value
>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>>>> to '0-63'
>>>
>>>
>>> This looks not right, it should be 0-7 instead.
>>>
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>> fd 39
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361 :
>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>> fd 39
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupCpuSetInherit:482
>>>> : Inherit cpuset.mems = 0-7
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>>> Set value
>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>> to '0-7'
>>>
>>>
>>> This is right.
>>>
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>> fd 39
>>>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
>>>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331 :
>>>> Set value
>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>> to '1'
>>>
>>>
>>> And it's strange that the cpuset.mems is changed to '1' here.
>
>
> Oh, actually this is right, cpuset.mems is about the memory nodes.
>
>
>>>
>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>> fd 39
>>>>
>>>> Could the RSS issue be related? Some kernel related option not playing
>>>> nice or enabled?
>>
>>
>> Instead, I'm wondering if the problem is caused by the mismatch
>> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
>> thus cause the problem for kernel memory management?
>
>
> So, the simple method to prove the guess is to use static placement
> like:
>
> <vcpu placement='static' cpuset='0-63'>2</vcpu>
> <numatune>
> <memory placement='static' nodeset='1'/>
> </numatune>
>
> Osier
Same error. Which I don't know if you expected or didn't expect.
It's expected. as "0-63" is the final result when using "auto"
placement.
11:25 p.m.
On Mon, Jan 28, 2013 at 10:23 AM, Osier Yang <jyang(a)redhat.com> wrote:
On 2013年01月29日 00:17, Doug Goldstein wrote:
>
> On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang<jyang(a)redhat.com> wrote:
>>
>> On 2013年01月28日 11:47, Osier Yang wrote:
>>>
>>>
>>> On 2013年01月28日 11:44, Osier Yang wrote:
>>>>
>>>>
>>>> On 2013年01月26日 01:07, Doug Goldstein wrote:
>>>>>
>>>>>
>>>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier Yang<jyang(a)redhat.com>
wrote:
>>>>>>
>>>>>>
>>>>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier
Yang<jyang(a)redhat.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
Goldstein<cardoe(a)gentoo.org>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm 1.2.2
(qemu-kvm 1.2.0 +
>>>>>>>>>> qemu
>>>>>>>>>> 1.2.2 applied on top plus a number of stability
patches). Having
>>>>>>>>>> issue
>>>>>>>>>> where my VMs fail to start with the following
message:
>>>>>>>>>>
>>>>>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Smell likes we have problem on setting the NUMA policy
(perhaps
>>>>>>>> caused by the incorrect host NUMA topology), given that
the system
>>>>>>>> still has enough memory. Or numad (if it's installed)
is doing
>>>>>>>> something wrong.
>>>>>>>>
>>>>>>>> Can you see if there is something about the Nodeset used
to set
>>>>>>>> the policy in debug log?
>>>>>>>>
>>>>>>>> E.g.
>>>>>>>>
>>>>>>>> % cat libvirtd.debug | grep Nodeset
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Well I don't see anything but its likely because I
didn't do
>>>>>>> something
>>>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd
--verbose
>>>>>>> from the command line.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> If the process is in background, it's expected you can't
see anything
>>>>>>
>>>>>>
>>>>>> My /etc/libvirt/libvirtd.conf had:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> log_outputs="3:syslog:libvirtd
1:file:/tmp/libvirtd.log" But I
>>>>>>> didn't
>>>>>>> get any debug messages.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> log_level=1 has to be set.
>>>>>>
>>>>>> Anyway, let's simply do this:
>>>>>>
>>>>>> % service libvirtd stop
>>>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a
libvirtd.debug
>>>>>>
>>>>>
>>>>> That's what I was doing, minus the tee just to the console and
nothing
>>>>> was coming out. Which is why I added the 1:file:/tmp/libvirtd.log,
>>>>> which also didn't get any debug messages. Turns out this instance
must
>>>>> have been built with --disable-debug,
>>>>>
>>>>> All I've got in the log is:
>>>>>
>>>>> # grep -i 'numa' libvirtd.debug
>>>>> 2013-01-25 16:50:15.287+0000: 417: debug : virCommandRunAsync:2200 :
>>>>> About to run /usr/bin/numad -w 2:2048
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614 :
>>>>> Nodeset returned from numad: 1
>>>>
>>>>
>>>>
>>>> This looks right.
>>>>
>>>>>
>>>>> Immediately below that is
>>>>>
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622 :
>>>>> Setting up domain cgroup (if required)
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 : New
>>>>> group /libvirt/qemu/bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537 :
>>>>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>>>> Make controller
/sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562 :
>>>>> Make controller /sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupCpuSetInherit:469
>>>>> : Setting up inheritance /libvirt/qemu ->
>>>>> /libvirt/qemu/bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361
:
>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupCpuSetInherit:482
>>>>> : Inherit cpuset.cpus = 0-63
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331
:
>>>>> Set value
>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>>>>> to '0-63'
>>>>
>>>>
>>>>
>>>> This looks not right, it should be 0-7 instead.
>>>>
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupGetValueStr:361
:
>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupCpuSetInherit:482
>>>>> : Inherit cpuset.mems = 0-7
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331
:
>>>>> Set value
>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>> to '0-7'
>>>>
>>>>
>>>>
>>>> This is right.
>>>>
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388 :
>>>>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupSetValueStr:331
:
>>>>> Set value
>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>> to '1'
>>>>
>>>>
>>>>
>>>> And it's strange that the cpuset.mems is changed to '1'
here.
>>
>>
>>
>> Oh, actually this is right, cpuset.mems is about the memory nodes.
>>
>>
>>>>
>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 : Closed
>>>>> fd 39
>>>>>
>>>>> Could the RSS issue be related? Some kernel related option not
playing
>>>>> nice or enabled?
>>>
>>>
>>>
>>> Instead, I'm wondering if the problem is caused by the mismatch
>>> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
>>> thus cause the problem for kernel memory management?
>>
>>
>>
>> So, the simple method to prove the guess is to use static placement
>> like:
>>
>> <vcpu placement='static' cpuset='0-63'>2</vcpu>
>> <numatune>
>> <memory placement='static' nodeset='1'/>
>> </numatune>
>>
>> Osier
>
>
> Same error. Which I don't know if you expected or didn't expect.
>
It's expected. as "0-63" is the final result when using "auto"
placement.
Since there's another user on the libvirt-list asking about the exact
same CPU I've got, I figured I'd do some poking. Oddly enough him and
I had different outputs from virsh nodeinfo. Just as background its
AMD 6272 CPUs. I've for 4 of them in the box but they're organized as
follows:
Sockets: 4
Cores: 16
Threads: 1 per core (16)
NUMA nodes: 8
Mem per node: 16GB
Total: 128GB
# virsh nodeinfo
CPU model: x86_64
CPU(s): 64
CPU frequency: 2100 MHz
CPU socket(s): 1
Core(s) per socket: 64
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 132013200 KiB
# virsh capabilities
<snip>
<topology sockets='1' cores='64' threads='1'/>
<snip>
<topology>
<cells num='8'>
<snip>
I've hand verified all the values in
/sys/devices/system/nodeX/cpuX/topology/physical_package_id to show
that the physical package is oriented in pairs (0&1, 2&3, 4&5, 6&7)
for the NUMA nodes.
Need to give git a whirl as I know that's got a bit different code
than 1.0.1 but I'll report back.
--
Doug Goldstein
1:21 a.m.
On 01/30/2013 01:25 PM, Doug Goldstein wrote:
On Mon, Jan 28, 2013 at 10:23 AM, Osier Yang<jyang(a)redhat.com>
wrote:
> On 2013年01月29日 00:17, Doug Goldstein wrote:
>> On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang<jyang(a)redhat.com> wrote:
>>> On 2013年01月28日 11:47, Osier Yang wrote:
>>>>
>>>> On 2013年01月28日 11:44, Osier Yang wrote:
>>>>>
>>>>> On 2013年01月26日 01:07, Doug Goldstein wrote:
>>>>>>
>>>>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier
Yang<jyang(a)redhat.com> wrote:
>>>>>>>
>>>>>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier
Yang<jyang(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
Goldstein<cardoe(a)gentoo.org>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm
1.2.2 (qemu-kvm 1.2.0 +
>>>>>>>>>>> qemu
>>>>>>>>>>> 1.2.2 applied on top plus a number of
stability patches). Having
>>>>>>>>>>> issue
>>>>>>>>>>> where my VMs fail to start with the following
message:
>>>>>>>>>>>
>>>>>>>>>>> kvm_init_vcpu failed: Cannot allocate memory
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Smell likes we have problem on setting the NUMA
policy (perhaps
>>>>>>>>> caused by the incorrect host NUMA topology), given
that the system
>>>>>>>>> still has enough memory. Or numad (if it's
installed) is doing
>>>>>>>>> something wrong.
>>>>>>>>>
>>>>>>>>> Can you see if there is something about the Nodeset
used to set
>>>>>>>>> the policy in debug log?
>>>>>>>>>
>>>>>>>>> E.g.
>>>>>>>>>
>>>>>>>>> % cat libvirtd.debug | grep Nodeset
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Well I don't see anything but its likely because I
didn't do
>>>>>>>> something
>>>>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran libvirtd
--verbose
>>>>>>>> from the command line.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> If the process is in background, it's expected you
can't see anything
>>>>>>>
>>>>>>>
>>>>>>> My /etc/libvirt/libvirtd.conf had:
>>>>>>>>
>>>>>>>>
>>>>>>>> log_outputs="3:syslog:libvirtd
1:file:/tmp/libvirtd.log" But I
>>>>>>>> didn't
>>>>>>>> get any debug messages.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> log_level=1 has to be set.
>>>>>>>
>>>>>>> Anyway, let's simply do this:
>>>>>>>
>>>>>>> % service libvirtd stop
>>>>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a
libvirtd.debug
>>>>>>>
>>>>>> That's what I was doing, minus the tee just to the console
and nothing
>>>>>> was coming out. Which is why I added the
1:file:/tmp/libvirtd.log,
>>>>>> which also didn't get any debug messages. Turns out this
instance must
>>>>>> have been built with --disable-debug,
>>>>>>
>>>>>> All I've got in the log is:
>>>>>>
>>>>>> # grep -i 'numa' libvirtd.debug
>>>>>> 2013-01-25 16:50:15.287+0000: 417: debug :
virCommandRunAsync:2200 :
>>>>>> About to run /usr/bin/numad -w 2:2048
>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3614
:
>>>>>> Nodeset returned from numad: 1
>>>>>
>>>>>
>>>>> This looks right.
>>>>>
>>>>>> Immediately below that is
>>>>>>
>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : qemuProcessStart:3622
:
>>>>>> Setting up domain cgroup (if required)
>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 :
New
>>>>>> group /libvirt/qemu/bb-2.6.35.9-i686
>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>>>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct in
>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupDetect:273 :
>>>>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:537
:
>>>>>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562
:
>>>>>> Make controller
/sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virCgroupMakeGroup:562
:
>>>>>> Make controller
/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupCpuSetInherit:469
>>>>>> : Setting up inheritance /libvirt/qemu ->
>>>>>> /libvirt/qemu/bb-2.6.35.9-i686
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupGetValueStr:361 :
>>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>> fd 39
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupCpuSetInherit:482
>>>>>> : Inherit cpuset.cpus = 0-63
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331 :
>>>>>> Set value
>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>>>>>> to '0-63'
>>>>>
>>>>>
>>>>> This looks not right, it should be 0-7 instead.
>>>>>
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>> fd 39
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupGetValueStr:361 :
>>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>> fd 39
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupCpuSetInherit:482
>>>>>> : Inherit cpuset.mems = 0-7
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331 :
>>>>>> Set value
>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>>> to '0-7'
>>>>>
>>>>>
>>>>> This is right.
>>>>>
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>> fd 39
>>>>>> 2013-01-25 16:50:17.296+0000: 417: warning : qemuSetupCgroup:388
:
>>>>>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331 :
>>>>>> Set value
>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>>> to '1'
>>>>>
>>>>>
>>>>> And it's strange that the cpuset.mems is changed to '1'
here.
>>>
>>>
>>> Oh, actually this is right, cpuset.mems is about the memory nodes.
>>>
>>>
>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>> fd 39
>>>>>>
>>>>>> Could the RSS issue be related? Some kernel related option not
playing
>>>>>> nice or enabled?
>>>>
>>>>
>>>> Instead, I'm wondering if the problem is caused by the mismatch
>>>> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
>>>> thus cause the problem for kernel memory management?
>>>
>>>
>>> So, the simple method to prove the guess is to use static placement
>>> like:
>>>
>>> <vcpu placement='static' cpuset='0-63'>2</vcpu>
>>> <numatune>
>>> <memory placement='static' nodeset='1'/>
>>> </numatune>
>>>
>>> Osier
>>
>> Same error. Which I don't know if you expected or didn't expect.
>>
> It's expected. as "0-63" is the final result when using
"auto"
> placement.
Since there's another user on the libvirt-list asking about the exact
same CPU I've got, I figured I'd do some poking. Oddly enough him and
I had different outputs from virsh nodeinfo. Just as background its
AMD 6272 CPUs. I've for 4 of them in the box but they're organized as
follows:
Sockets: 4
Cores: 16
Threads: 1 per core (16)
NUMA nodes: 8
Mem per node: 16GB
Total: 128GB
# virsh nodeinfo
CPU model: x86_64
CPU(s): 64
CPU frequency: 2100 MHz
CPU socket(s): 1
Core(s) per socket: 64
Thread(s) per core: 1
NUMA cell(s): 1
Memory size: 132013200 KiB
# virsh capabilities
<snip>
<topology sockets='1' cores='64' threads='1'/>
<snip>
<topology>
<cells num='8'>
<snip>
I've hand verified all the values in
/sys/devices/system/nodeX/cpuX/topology/physical_package_id to show
that the physical package is oriented in pairs (0&1, 2&3, 4&5, 6&7)
for the NUMA nodes.
Need to give git a whirl as I know that's got a bit different code
than 1.0.1 but I'll report back.
For AMD 62xx CPUs, the output is expected.
Check out this bug:
virsh nodeinfo can't get the right info on AMD Bulldozer cpu
https://bugzilla.redhat.com/show_bug.cgi?id=874050
Wayne Sun
2013-01-30
4:01 p.m.
On Wed, Jan 30, 2013 at 1:21 AM, Wayne Sun <gsun(a)redhat.com> wrote:
On 01/30/2013 01:25 PM, Doug Goldstein wrote:
>
> On Mon, Jan 28, 2013 at 10:23 AM, Osier Yang<jyang(a)redhat.com> wrote:
>>
>> On 2013年01月29日 00:17, Doug Goldstein wrote:
>>>
>>> On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang<jyang(a)redhat.com>
wrote:
>>>>
>>>> On 2013年01月28日 11:47, Osier Yang wrote:
>>>>>
>>>>>
>>>>> On 2013年01月28日 11:44, Osier Yang wrote:
>>>>>>
>>>>>>
>>>>>> On 2013年01月26日 01:07, Doug Goldstein wrote:
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier
Yang<jyang(a)redhat.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier
Yang<jyang(a)redhat.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
>>>>>>>>>>> Goldstein<cardoe(a)gentoo.org>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> I am using libvirt 0.10.2.2 and qemu-kvm
1.2.2 (qemu-kvm 1.2.0
>>>>>>>>>>>> +
>>>>>>>>>>>> qemu
>>>>>>>>>>>> 1.2.2 applied on top plus a number of
stability patches).
>>>>>>>>>>>> Having
>>>>>>>>>>>> issue
>>>>>>>>>>>> where my VMs fail to start with the
following message:
>>>>>>>>>>>>
>>>>>>>>>>>> kvm_init_vcpu failed: Cannot allocate
memory
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Smell likes we have problem on setting the NUMA
policy (perhaps
>>>>>>>>>> caused by the incorrect host NUMA topology),
given that the
>>>>>>>>>> system
>>>>>>>>>> still has enough memory. Or numad (if it's
installed) is doing
>>>>>>>>>> something wrong.
>>>>>>>>>>
>>>>>>>>>> Can you see if there is something about the
Nodeset used to set
>>>>>>>>>> the policy in debug log?
>>>>>>>>>>
>>>>>>>>>> E.g.
>>>>>>>>>>
>>>>>>>>>> % cat libvirtd.debug | grep Nodeset
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Well I don't see anything but its likely because
I didn't do
>>>>>>>>> something
>>>>>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran
libvirtd --verbose
>>>>>>>>> from the command line.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> If the process is in background, it's expected you
can't see
>>>>>>>> anything
>>>>>>>>
>>>>>>>>
>>>>>>>> My /etc/libvirt/libvirtd.conf had:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> log_outputs="3:syslog:libvirtd
1:file:/tmp/libvirtd.log" But I
>>>>>>>>> didn't
>>>>>>>>> get any debug messages.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> log_level=1 has to be set.
>>>>>>>>
>>>>>>>> Anyway, let's simply do this:
>>>>>>>>
>>>>>>>> % service libvirtd stop
>>>>>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 | tee -a
libvirtd.debug
>>>>>>>>
>>>>>>> That's what I was doing, minus the tee just to the
console and
>>>>>>> nothing
>>>>>>> was coming out. Which is why I added the
1:file:/tmp/libvirtd.log,
>>>>>>> which also didn't get any debug messages. Turns out this
instance
>>>>>>> must
>>>>>>> have been built with --disable-debug,
>>>>>>>
>>>>>>> All I've got in the log is:
>>>>>>>
>>>>>>> # grep -i 'numa' libvirtd.debug
>>>>>>> 2013-01-25 16:50:15.287+0000: 417: debug :
virCommandRunAsync:2200 :
>>>>>>> About to run /usr/bin/numad -w 2:2048
>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
qemuProcessStart:3614 :
>>>>>>> Nodeset returned from numad: 1
>>>>>>
>>>>>>
>>>>>>
>>>>>> This looks right.
>>>>>>
>>>>>>> Immediately below that is
>>>>>>>
>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
qemuProcessStart:3622 :
>>>>>>> Setting up domain cgroup (if required)
>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug : virCgroupNew:619 :
New
>>>>>>> group /libvirt/qemu/bb-2.6.35.9-i686
>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
virCgroupDetect:273 :
>>>>>>> Detected mount/mapping 1:cpuacct at /sys/fs/cgroup/cpuacct
in
>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
virCgroupDetect:273 :
>>>>>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupMakeGroup:537 :
>>>>>>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupMakeGroup:562 :
>>>>>>> Make controller
>>>>>>> /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupMakeGroup:562 :
>>>>>>> Make controller
/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
>>>>>>> virCgroupCpuSetInherit:469
>>>>>>> : Setting up inheritance /libvirt/qemu ->
>>>>>>> /libvirt/qemu/bb-2.6.35.9-i686
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupGetValueStr:361
>>>>>>> :
>>>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>>> fd 39
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
>>>>>>> virCgroupCpuSetInherit:482
>>>>>>> : Inherit cpuset.cpus = 0-63
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331
>>>>>>> :
>>>>>>> Set value
>>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>>>>>>> to '0-63'
>>>>>>
>>>>>>
>>>>>>
>>>>>> This looks not right, it should be 0-7 instead.
>>>>>>
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>>> fd 39
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupGetValueStr:361
>>>>>>> :
>>>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>>> fd 39
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
>>>>>>> virCgroupCpuSetInherit:482
>>>>>>> : Inherit cpuset.mems = 0-7
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331
>>>>>>> :
>>>>>>> Set value
>>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>>>> to '0-7'
>>>>>>
>>>>>>
>>>>>>
>>>>>> This is right.
>>>>>>
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>>> fd 39
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: warning :
qemuSetupCgroup:388 :
>>>>>>> Could not autoset a RSS limit for domain bb-2.6.35.9-i686
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331
>>>>>>> :
>>>>>>> Set value
>>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>>>> to '1'
>>>>>>
>>>>>>
>>>>>>
>>>>>> And it's strange that the cpuset.mems is changed to
'1' here.
>>>>
>>>>
>>>>
>>>> Oh, actually this is right, cpuset.mems is about the memory nodes.
>>>>
>>>>
>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug : virFileClose:72 :
Closed
>>>>>>> fd 39
>>>>>>>
>>>>>>> Could the RSS issue be related? Some kernel related option
not
>>>>>>> playing
>>>>>>> nice or enabled?
>>>>>
>>>>>
>>>>>
>>>>> Instead, I'm wondering if the problem is caused by the mismatch
>>>>> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
>>>>> thus cause the problem for kernel memory management?
>>>>
>>>>
>>>>
>>>> So, the simple method to prove the guess is to use static placement
>>>> like:
>>>>
>>>> <vcpu placement='static'
cpuset='0-63'>2</vcpu>
>>>> <numatune>
>>>> <memory placement='static' nodeset='1'/>
>>>> </numatune>
>>>>
>>>> Osier
>>>
>>>
>>> Same error. Which I don't know if you expected or didn't expect.
>>>
>> It's expected. as "0-63" is the final result when using
"auto"
>> placement.
>
> Since there's another user on the libvirt-list asking about the exact
> same CPU I've got, I figured I'd do some poking. Oddly enough him and
> I had different outputs from virsh nodeinfo. Just as background its
> AMD 6272 CPUs. I've for 4 of them in the box but they're organized as
> follows:
>
> Sockets: 4
> Cores: 16
> Threads: 1 per core (16)
> NUMA nodes: 8
> Mem per node: 16GB
> Total: 128GB
>
> # virsh nodeinfo
> CPU model: x86_64
> CPU(s): 64
> CPU frequency: 2100 MHz
> CPU socket(s): 1
> Core(s) per socket: 64
> Thread(s) per core: 1
> NUMA cell(s): 1
> Memory size: 132013200 KiB
>
> # virsh capabilities
> <snip>
> <topology sockets='1' cores='64' threads='1'/>
> <snip>
> <topology>
> <cells num='8'>
> <snip>
>
> I've hand verified all the values in
> /sys/devices/system/nodeX/cpuX/topology/physical_package_id to show
> that the physical package is oriented in pairs (0&1, 2&3, 4&5, 6&7)
> for the NUMA nodes.
>
> Need to give git a whirl as I know that's got a bit different code
> than 1.0.1 but I'll report back.
>
For AMD 62xx CPUs, the output is expected.
Check out this bug:
virsh nodeinfo can't get the right info on AMD Bulldozer cpu
https://bugzilla.redhat.com/show_bug.cgi?id=874050
Wayne Sun
2013-01-30
Wayne,
I'd argue we need to determine what format we really need the data in.
Do we actually really care about physical sockets? Or should we care
about packages? Because with this specific CPU there are 2 packages in
1 physical socket to form 2 NUMA nodes per package.
The reason I say this is that we went from NUMA being defined for the
domain working to the domain failing to start up with a cryptic error
message which IMHO is worse.
The flip side of the coin is that we can just strip out all the NUMA
settings when starting the domain up if we know it won't work.
--
Doug Goldstein
10:30 p.m.
[ CC Peter ]
On 2013年01月31日 06:01, Doug Goldstein wrote:
On Wed, Jan 30, 2013 at 1:21 AM, Wayne Sun<gsun(a)redhat.com>
wrote:
> On 01/30/2013 01:25 PM, Doug Goldstein wrote:
>>
>> On Mon, Jan 28, 2013 at 10:23 AM, Osier Yang<jyang(a)redhat.com> wrote:
>>>
>>> On 2013年01月29日 00:17, Doug Goldstein wrote:
>>>>
>>>> On Sun, Jan 27, 2013 at 10:46 PM, Osier Yang<jyang(a)redhat.com>
wrote:
>>>>>
>>>>> On 2013年01月28日 11:47, Osier Yang wrote:
>>>>>>
>>>>>>
>>>>>> On 2013年01月28日 11:44, Osier Yang wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 2013年01月26日 01:07, Doug Goldstein wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jan 24, 2013 at 12:58 AM, Osier
Yang<jyang(a)redhat.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2013年01月24日 14:26, Doug Goldstein wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jan 23, 2013 at 11:02 PM, Osier
Yang<jyang(a)redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2013年01月24日 12:11, Doug Goldstein wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jan 23, 2013 at 3:45 PM, Doug
>>>>>>>>>>>> Goldstein<cardoe(a)gentoo.org>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am using libvirt 0.10.2.2 and
qemu-kvm 1.2.2 (qemu-kvm 1.2.0
>>>>>>>>>>>>> +
>>>>>>>>>>>>> qemu
>>>>>>>>>>>>> 1.2.2 applied on top plus a number of
stability patches).
>>>>>>>>>>>>> Having
>>>>>>>>>>>>> issue
>>>>>>>>>>>>> where my VMs fail to start with the
following message:
>>>>>>>>>>>>>
>>>>>>>>>>>>> kvm_init_vcpu failed: Cannot allocate
memory
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Smell likes we have problem on setting the
NUMA policy (perhaps
>>>>>>>>>>> caused by the incorrect host NUMA topology),
given that the
>>>>>>>>>>> system
>>>>>>>>>>> still has enough memory. Or numad (if
it's installed) is doing
>>>>>>>>>>> something wrong.
>>>>>>>>>>>
>>>>>>>>>>> Can you see if there is something about the
Nodeset used to set
>>>>>>>>>>> the policy in debug log?
>>>>>>>>>>>
>>>>>>>>>>> E.g.
>>>>>>>>>>>
>>>>>>>>>>> % cat libvirtd.debug | grep Nodeset
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Well I don't see anything but its likely
because I didn't do
>>>>>>>>>> something
>>>>>>>>>> correct. I had LIBVIRT_DEBUG=1 exported and ran
libvirtd --verbose
>>>>>>>>>> from the command line.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> If the process is in background, it's expected
you can't see
>>>>>>>>> anything
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> My /etc/libvirt/libvirtd.conf had:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> log_outputs="3:syslog:libvirtd
1:file:/tmp/libvirtd.log" But I
>>>>>>>>>> didn't
>>>>>>>>>> get any debug messages.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> log_level=1 has to be set.
>>>>>>>>>
>>>>>>>>> Anyway, let's simply do this:
>>>>>>>>>
>>>>>>>>> % service libvirtd stop
>>>>>>>>> % LIBVIRT_DEBUG=1 /usr/sbin/libvirtd 2>&1 |
tee -a libvirtd.debug
>>>>>>>>>
>>>>>>>> That's what I was doing, minus the tee just to the
console and
>>>>>>>> nothing
>>>>>>>> was coming out. Which is why I added the
1:file:/tmp/libvirtd.log,
>>>>>>>> which also didn't get any debug messages. Turns out
this instance
>>>>>>>> must
>>>>>>>> have been built with --disable-debug,
>>>>>>>>
>>>>>>>> All I've got in the log is:
>>>>>>>>
>>>>>>>> # grep -i 'numa' libvirtd.debug
>>>>>>>> 2013-01-25 16:50:15.287+0000: 417: debug :
virCommandRunAsync:2200 :
>>>>>>>> About to run /usr/bin/numad -w 2:2048
>>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
qemuProcessStart:3614 :
>>>>>>>> Nodeset returned from numad: 1
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This looks right.
>>>>>>>
>>>>>>>> Immediately below that is
>>>>>>>>
>>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
qemuProcessStart:3622 :
>>>>>>>> Setting up domain cgroup (if required)
>>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
virCgroupNew:619 : New
>>>>>>>> group /libvirt/qemu/bb-2.6.35.9-i686
>>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
virCgroupDetect:273 :
>>>>>>>> Detected mount/mapping 1:cpuacct at
/sys/fs/cgroup/cpuacct in
>>>>>>>> 2013-01-25 16:50:17.295+0000: 417: debug :
virCgroupDetect:273 :
>>>>>>>> Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset
in
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupMakeGroup:537 :
>>>>>>>> Make group /libvirt/qemu/bb-2.6.35.9-i686
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupMakeGroup:562 :
>>>>>>>> Make controller
>>>>>>>> /sys/fs/cgroup/cpuacct/libvirt/qemu/bb-2.6.35.9-i686/
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupMakeGroup:562 :
>>>>>>>> Make controller
/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
>>>>>>>> virCgroupCpuSetInherit:469
>>>>>>>> : Setting up inheritance /libvirt/qemu ->
>>>>>>>> /libvirt/qemu/bb-2.6.35.9-i686
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupGetValueStr:361
>>>>>>>> :
>>>>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.cpus
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virFileClose:72 : Closed
>>>>>>>> fd 39
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
>>>>>>>> virCgroupCpuSetInherit:482
>>>>>>>> : Inherit cpuset.cpus = 0-63
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331
>>>>>>>> :
>>>>>>>> Set value
>>>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.cpus'
>>>>>>>> to '0-63'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This looks not right, it should be 0-7 instead.
>>>>>>>
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virFileClose:72 : Closed
>>>>>>>> fd 39
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupGetValueStr:361
>>>>>>>> :
>>>>>>>> Get value /sys/fs/cgroup/cpuset/libvirt/qemu/cpuset.mems
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virFileClose:72 : Closed
>>>>>>>> fd 39
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
>>>>>>>> virCgroupCpuSetInherit:482
>>>>>>>> : Inherit cpuset.mems = 0-7
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331
>>>>>>>> :
>>>>>>>> Set value
>>>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>>>>> to '0-7'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> This is right.
>>>>>>>
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virFileClose:72 : Closed
>>>>>>>> fd 39
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: warning :
qemuSetupCgroup:388 :
>>>>>>>> Could not autoset a RSS limit for domain
bb-2.6.35.9-i686
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virCgroupSetValueStr:331
>>>>>>>> :
>>>>>>>> Set value
>>>>>>>>
'/sys/fs/cgroup/cpuset/libvirt/qemu/bb-2.6.35.9-i686/cpuset.mems'
>>>>>>>> to '1'
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> And it's strange that the cpuset.mems is changed to
'1' here.
>>>>>
>>>>>
>>>>>
>>>>> Oh, actually this is right, cpuset.mems is about the memory nodes.
>>>>>
>>>>>
>>>>>>>> 2013-01-25 16:50:17.296+0000: 417: debug :
virFileClose:72 : Closed
>>>>>>>> fd 39
>>>>>>>>
>>>>>>>> Could the RSS issue be related? Some kernel related
option not
>>>>>>>> playing
>>>>>>>> nice or enabled?
>>>>>>
>>>>>>
>>>>>>
>>>>>> Instead, I'm wondering if the problem is caused by the
mismatch
>>>>>> (from libvirt p.o.v) between cpuset.cpus and cpuset.mems, which
>>>>>> thus cause the problem for kernel memory management?
>>>>>
>>>>>
>>>>>
>>>>> So, the simple method to prove the guess is to use static placement
>>>>> like:
>>>>>
>>>>> <vcpu placement='static'
cpuset='0-63'>2</vcpu>
>>>>> <numatune>
>>>>> <memory placement='static' nodeset='1'/>
>>>>> </numatune>
>>>>>
>>>>> Osier
>>>>
>>>>
>>>> Same error. Which I don't know if you expected or didn't expect.
>>>>
>>> It's expected. as "0-63" is the final result when using
"auto"
>>> placement.
>>
>> Since there's another user on the libvirt-list asking about the exact
>> same CPU I've got, I figured I'd do some poking. Oddly enough him and
>> I had different outputs from virsh nodeinfo. Just as background its
>> AMD 6272 CPUs. I've for 4 of them in the box but they're organized as
>> follows:
>>
>> Sockets: 4
>> Cores: 16
>> Threads: 1 per core (16)
>> NUMA nodes: 8
>> Mem per node: 16GB
>> Total: 128GB
>>
>> # virsh nodeinfo
>> CPU model: x86_64
>> CPU(s): 64
>> CPU frequency: 2100 MHz
>> CPU socket(s): 1
>> Core(s) per socket: 64
>> Thread(s) per core: 1
>> NUMA cell(s): 1
>> Memory size: 132013200 KiB
>>
>> # virsh capabilities
>> <snip>
>> <topology sockets='1' cores='64'
threads='1'/>
>> <snip>
>> <topology>
>> <cells num='8'>
>> <snip>
>>
>> I've hand verified all the values in
>> /sys/devices/system/nodeX/cpuX/topology/physical_package_id to show
>> that the physical package is oriented in pairs (0&1, 2&3, 4&5,
6&7)
>> for the NUMA nodes.
>>
>> Need to give git a whirl as I know that's got a bit different code
>> than 1.0.1 but I'll report back.
As far as I see, Peter committed more patches to fix the CPU toplogy
parsing on AMD platfroms. Perhaps he will known if this is fixed
in new release.
>>
> For AMD 62xx CPUs, the output is expected.
>
> Check out this bug:
> virsh nodeinfo can't get the right info on AMD Bulldozer cpu
> https://bugzilla.redhat.com/show_bug.cgi?id=874050
>
> Wayne Sun
> 2013-01-30
>
Wayne,
I'd argue we need to determine what format we really need the data in.
Do we actually really care about physical sockets? Or should we care
about packages? Because with this specific CPU there are 2 packages in
1 physical socket to form 2 NUMA nodes per package.
I agreed. Though the total number of CPUs is correct, which guarantees
most of the stuffs related with CPU topology work. But it still should
be fixed.
The reason I say this is that we went from NUMA being defined for the
domain working to the domain failing to start up with a cryptic error
message which IMHO is worse.
The flip side of the coin is that we can just strip out all the NUMA
settings when starting the domain up if we know it won't work.
4194
days inactive
4202
days old
14 comments
3 participants
participants (3)
-
Doug Goldstein -
Osier Yang -
Wayne Sun