12:49 p.m.
Hello,
I noticed that libvirtd sometimes crashes immediately after start..
here's the backtrace:
Core was generated by `libvirtd --daemon --listen'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x000000000042eb21 in qemuDomainObjEnterMonitor (obj=0xe417d0) at
qemu/qemu_driver.c:478
#2 0x0000000000448e7e in qemudDomainGetInfo (dom=<value optimized out>,
info=0x43392e10) at qemu/qemu_driver.c:5165
#3 0x00007fee054f29f2 in virDomainGetInfo (domain=0xe3f610, info=0x43392e10) at
libvirt.c:3177
#4 0x000000000042554e in remoteDispatchDomainGetInfo (server=<value optimized out>,
client=<value optimized out>, conn=0xdd8a20, hdr=<value optimized out>,
rerr=0x43392eb0,
args=<value optimized out>, ret=0x43392f90) at remote.c:1503
#5 0x0000000000427444 in remoteDispatchClientCall (server=0xdb84d0,
client=0x7fedfc040bc0, msg=0x7fedfc0410a0) at dispatch.c:529
#6 remoteDispatchClientRequest (server=0xdb84d0, client=0x7fedfc040bc0,
msg=0x7fedfc0410a0) at dispatch.c:407
#7 0x00000000004194c7 in qemudWorker (data=<value optimized out>) at
libvirtd.c:1587
#8 0x00007fee038be73d in start_thread () from /lib64/libpthread.so.0
#9 0x00007fee03430f6d in clone () from /lib64/libc.so.6
Could somebody please look at it?
If I could provide any additional information, please let me know.
Thanks in advance
BR
nik
--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis(a)linuxbox.cz
-------------------------------------
1:04 p.m.
libvir-list-bounces(a)redhat.com wrote on 11/23/2010 01:49:46 PM:
Hello,
I noticed that libvirtd sometimes crashes immediately after start..
here's the backtrace:
Core was generated by `libvirtd --daemon --listen'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from
/lib64/libpthread.so.0
(gdb) bt
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from
/lib64/libpthread.so.0
#1 0x000000000042eb21 in qemuDomainObjEnterMonitor (obj=0xe417d0)
at qemu/qemu_driver.c:478
#2 0x0000000000448e7e in qemudDomainGetInfo (dom=<value optimized
out>, info=0x43392e10) at qemu/qemu_driver.c:5165
#3 0x00007fee054f29f2 in virDomainGetInfo (domain=0xe3f610,
info=0x43392e10) at libvirt.c:3177
#4 0x000000000042554e in remoteDispatchDomainGetInfo (server=<value
optimized out>, client=<value optimized out>, conn=0xdd8a20,
hdr=<value optimized out>, rerr=0x43392eb0,
args=<value optimized out>, ret=0x43392f90) at remote.c:1503
#5 0x0000000000427444 in remoteDispatchClientCall (server=0xdb84d0,
client=0x7fedfc040bc0, msg=0x7fedfc0410a0) at dispatch.c:529
#6 remoteDispatchClientRequest (server=0xdb84d0,
client=0x7fedfc040bc0, msg=0x7fedfc0410a0) at dispatch.c:407
#7 0x00000000004194c7 in qemudWorker (data=<value optimized out>)
at libvirtd.c:1587
#8 0x00007fee038be73d in start_thread () from /lib64/libpthread.so.0
#9 0x00007fee03430f6d in clone () from /lib64/libc.so.6
Could somebody please look at it?
If I could provide any additional information, please let me know.
Thanks in advance
BR
nik
I also saw crashes recently, though those were resolved with a recent
patch from ~3 hours ago.
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=149c49213799bc0fb1b9...
Hope this one also resolves your crashes.
Regards,
Stefan
12:13 p.m.
I also saw crashes recently, though those were resolved with a recent
patch from ~3 hours ago.
http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=149c49213799bc0fb1b9...
Hope this one also resolves your crashes.
Regards,
Stefan
Hello Stefan,
Thanks for reply! but I'm afraid this is unrelated problem, seems there is something
wrong with locking.. (like Eric pointed)
cheers!
n.
1:09 p.m.
On 11/23/2010 11:49 AM, Nikola Ciprich wrote:
Hello,
I noticed that libvirtd sometimes crashes immediately after start..
here's the backtrace:
Core was generated by `libvirtd --daemon --listen'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x000000000042eb21 in qemuDomainObjEnterMonitor (obj=0xe417d0) at
qemu/qemu_driver.c:478
#2 0x0000000000448e7e in qemudDomainGetInfo (dom=<value optimized out>,
info=0x43392e10) at qemu/qemu_driver.c:5165
#3 0x00007fee054f29f2 in virDomainGetInfo (domain=0xe3f610, info=0x43392e10) at
libvirt.c:3177
#4 0x000000000042554e in remoteDispatchDomainGetInfo (server=<value optimized
out>, client=<value optimized out>, conn=0xdd8a20, hdr=<value optimized
out>, rerr=0x43392eb0,
args=<value optimized out>, ret=0x43392f90) at remote.c:1503
#5 0x0000000000427444 in remoteDispatchClientCall (server=0xdb84d0,
client=0x7fedfc040bc0, msg=0x7fedfc0410a0) at dispatch.c:529
#6 remoteDispatchClientRequest (server=0xdb84d0, client=0x7fedfc040bc0,
msg=0x7fedfc0410a0) at dispatch.c:407
#7 0x00000000004194c7 in qemudWorker (data=<value optimized out>) at
libvirtd.c:1587
#8 0x00007fee038be73d in start_thread () from /lib64/libpthread.so.0
#9 0x00007fee03430f6d in clone () from /lib64/libc.so.6
Could somebody please look at it?
If I could provide any additional information, please let me know.
Thanks in advance
Hmm; this looks similar to the crash that was supposed to be fixed by
commit 054d43f57 which made it into 0.8.5; maybe I missed a case where
we aren't grabbing locks in the correct order?
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
12:15 p.m.
Hmm; this looks similar to the crash that was supposed to be fixed
by
commit 054d43f57 which made it into 0.8.5; maybe I missed a case where
we aren't grabbing locks in the correct order?
Hello Eric,
might be.. I see those crashes quite often on my testing boxes, actually every third or
fourth
boot it crashes..
Do You think You might have time to revise the patch?
cheers
nik
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis(a)linuxbox.cz
-------------------------------------
5:20 p.m.
On 11/23/2010 11:49 AM, Nikola Ciprich wrote:
Hello,
I noticed that libvirtd sometimes crashes immediately after start..
here's the backtrace:
Core was generated by `libvirtd --daemon --listen'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from /lib64/libpthread.so.0
(gdb) bt
#0 0x00007fee038c0ca0 in pthread_mutex_lock () from /lib64/libpthread.so.0
#1 0x000000000042eb21 in qemuDomainObjEnterMonitor (obj=0xe417d0) at
qemu/qemu_driver.c:478
#2 0x0000000000448e7e in qemudDomainGetInfo (dom=<value optimized out>,
info=0x43392e10) at qemu/qemu_driver.c:5165
Unfortunately, I'm not seeing a local race:
} else if (!priv->jobActive) {
if (qemuDomainObjBeginJob(vm) < 0)
goto cleanup;
if (!virDomainObjIsActive(vm))
err = 0;
else {
qemuDomainObjEnterMonitor(vm);
err = qemuMonitorGetBalloonInfo(priv->mon, &balloon);
qemuDomainObjExitMonitor(vm);
}
That properly grabs the job lock, verifies that the vm is still active,
and then uses the monitor. The only other thing I can think of is a
non-local race that I'm not seeing in the immediate vicinity; perhaps
one thread is able to see the vm object prior to another thread
finishing the creation of the monitor lock, so that querying the domain
info ends up calling pthread_mutex_lock on an invalid mutex? But that
shouldn't be possible - object creation is also done under a lock, so
nothing should be able to see a partially initialized object.
Is this something you can easily repeat? Can you reproduce it while a
debugger is attached, or have you been limited to post-mortem debugging
so far? I'm a bit stumped on what to look for next.
--
Eric Blake eblake(a)redhat.com +1-801-349-2682
Libvirt virtualization library http://libvirt.org
2:16 p.m.
Unfortunately, I'm not seeing a local race:
} else if (!priv->jobActive) {
if (qemuDomainObjBeginJob(vm) < 0)
goto cleanup;
if (!virDomainObjIsActive(vm))
err = 0;
else {
qemuDomainObjEnterMonitor(vm);
err = qemuMonitorGetBalloonInfo(priv->mon, &balloon);
qemuDomainObjExitMonitor(vm);
}
That properly grabs the job lock, verifies that the vm is still active,
and then uses the monitor. The only other thing I can think of is a
non-local race that I'm not seeing in the immediate vicinity; perhaps
one thread is able to see the vm object prior to another thread
finishing the creation of the monitor lock, so that querying the domain
info ends up calling pthread_mutex_lock on an invalid mutex? But that
shouldn't be possible - object creation is also done under a lock, so
nothing should be able to see a partially initialized object.
Is this something you can easily repeat? Can you reproduce it while a
debugger is attached, or have you been limited to post-mortem debugging
so far? I'm a bit stumped on what to look for next.
Hello Eric,
crashes always occur when libvirt is starting, no machines are running at that time..
Dunno if this information can help...?
Maybe it might be possible to edit initscript to start libvirt in gdb?
n.
--
-------------------------------------
Ing. Nikola CIPRICH
LinuxBox.cz, s.r.o.
28. rijna 168, 709 01 Ostrava
tel.: +420 596 603 142
fax: +420 596 621 273
mobil: +420 777 093 799
www.linuxbox.cz
mobil servis: +420 737 238 656
email servis: servis(a)linuxbox.cz
-------------------------------------
5136
days inactive
5143
days old
6 comments
3 participants
participants (3)
-
Eric Blake -
Nikola Ciprich -
Stefan Berger