Re: [libvirt] libvirtd segfault

On 12/29/2012 04:09 AM, Michal Privoznik wrote: Michal, I have a faster way to reproduce the crash (~10 minutes). Continue to read for new (easier) steps. This test was done with the standard v1.0.0 libvirtd code source, with no other patches applied. Here is the end result from the standard libvirt.log after the crash was reproduced, outside gdb: 2013-01-02 14:05:40.775+0000: 20383: error : qemuProcessWaitForMonitor:1700 : internal error process exited while connecting to monitor: char device redirected to /dev/pts/18 qemu: terminating on signal 15 from pid 20198 Caught Segmentation violation dumping internal log buffer: ====== start of log ===== ^@: OBJECT_UNREF: obj=0x754200 2013-01-02 14:05:40.655+000020198: debug : virEventPollDispatchHandles:484 : i=3 w=4 2013-01-02 14:05:40.655+000020198: debug : virEventPollDispatchHandles:484 : i=4 w=5 2013-01-02 14:05:40.655+000020198: debug : virEventPollDispatchHandles:484 : i=5 w=6 ........ <snip> Here's the script I used: #!/usr/bin/env perl use strict; use warnings; use threads; use threads::shared; my $upper_bound = $ARGV[0] || die "No upper bound specified!\n"; my $id = $ARGV[1] || die "No id specified!\n"; sub _shutdown_lvm { print "\tvirsh destroy $id\n"; system("virsh destroy $id"); } sub _start_lvm { print "\tvirsh create /xen/configs/$id.cfg\n"; system("virsh create /xen/configs/$id.cfg"); } sub _sequence { my $num = shift; print "started thread $num\n"; _shutdown_lvm(); _start_lvm(); print "done with thread $num\n"; return $num; } my @threads; for ( my $count = 1; $count <= $upper_bound; $count++) { my $t = threads->new(\&_sequence, $count); push(@threads,$t); } foreach (@threads) { my $num = $_->join; print "done with $num\n"; } I then ran it, with these args: perl libvirt_crash.pl 90000 zzueyjvh I then got the segfault within 10 minutes: 2013-01-02 14:16:39.820+0000: 4627: error : qemuProcessWaitForMonitor:1700 : internal error process exited while connecting to monitor: char device redirected to /dev/pts/22 qemu: terminating on signal 15 from pid 3501 Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fffa17fb700 (LWP 17310)] qemuDomainObjBeginJobInternal (driver=0x7fffe4018c70, driver_locked=true, obj=0x7fffb0001b80, job=QEMU_JOB_DESTROY, asyncJob=QEMU_ASYNC_JOB_NONE) at qemu/qemu_domain.c:780 780 priv->jobs_queued++; (gdb) (gdb) bt #0 qemuDomainObjBeginJobInternal (driver=0x7fffe4018c70, driver_locked=true, obj=0x7fffb0001b80, job=QEMU_JOB_DESTROY, asyncJob=QEMU_ASYNC_JOB_NONE) at qemu/qemu_domain.c:780 #1 0x00007fffeac0bf46 in qemuDomainDestroyFlags (dom=<value optimized out>, flags=<value optimized out>) at qemu/qemu_driver.c:2189 #2 0x0000003a35302587 in virDomainDestroy (domain=0x7fff90002600) at libvirt.c:2215 #3 0x00000000004296e2 in remoteDispatchDomainDestroy (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7fffa17fabc0, args=<value optimized out>, ret=<value optimized out>) at remote_dispatch.h:1277 #4 remoteDispatchDomainDestroyHelper (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7fffa17fabc0, args=<value optimized out>, ret=<value optimized out>) at remote_dispatch.h:1255 #5 0x0000003a3534fd02 in virNetServerProgramDispatchCall (prog=0x6894a0, server=0x680e60, client=0x129bf90, msg=0xe99130) at rpc/virnetserverprogram.c:431 #6 virNetServerProgramDispatch (prog=0x6894a0, server=0x680e60, client=0x129bf90, msg=0xe99130) at rpc/virnetserverprogram.c:304 #7 0x0000003a3534daa6 in virNetServerProcessMsg (srv=<value optimized out>, client=0x129bf90, prog=<value optimized out>, msg=0xe99130) at rpc/virnetserver.c:173 #8 0x0000003a3534e5e3 in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x680e60) at rpc/virnetserver.c:194 #9 0x0000003a35267fdc in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144 #10 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #11 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #12 0x0000003009ae570d in clone () from /lib64/libc.so.6 (gdb) Thread 36 (Thread 0x7fff8a1fc700 (LWP 17626)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 35 (Thread 0x7fff8abfd700 (LWP 17624)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 34 (Thread 0x7fff8b5fe700 (LWP 17622)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 33 (Thread 0x7fff8bfff700 (LWP 17596)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 32 (Thread 0x7fffa0dfa700 (LWP 17595)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 31 (Thread 0x7fffa17fb700 (LWP 17310)): #0 qemuDomainObjBeginJobInternal (driver=0x7fffe4018c70, driver_locked=true, obj=0x7fffb0001b80, job=QEMU_JOB_DESTROY, asyncJob=QEMU_ASYNC_JOB_NONE) at qemu/qemu_domain.c:780 #1 0x00007fffeac0bf46 in qemuDomainDestroyFlags (dom=<value optimized out>, flags=<value optimized out>) at qemu/qemu_driver.c:2189 #2 0x0000003a35302587 in virDomainDestroy (domain=0x7fff90002600) at libvirt.c:2215 #3 0x00000000004296e2 in remoteDispatchDomainDestroy (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7fffa17fabc0, args=<value optimized out>, ret=<value optimized out>) at remote_dispatch.h:1277 #4 remoteDispatchDomainDestroyHelper (server=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=0x7fffa17fabc0, args=<value optimized out>, ret=<value optimized out>) at remote_dispatch.h:1255 #5 0x0000003a3534fd02 in virNetServerProgramDispatchCall (prog=0x6894a0, server=0x680e60, client=0x129bf90, msg=0xe99130) at rpc/virnetserverprogram.c:431 #6 virNetServerProgramDispatch (prog=0x6894a0, server=0x680e60, client=0x129bf90, msg=0xe99130) at rpc/virnetserverprogram.c:304 #7 0x0000003a3534daa6 in virNetServerProcessMsg (srv=<value optimized out>, client=0x129bf90, prog=<value optimized out>, msg=0xe99130) at rpc/virnetserver.c:173 #8 0x0000003a3534e5e3 in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x680e60) at rpc/virnetserver.c:194 #9 0x0000003a35267fdc in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144 #10 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #11 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #12 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 30 (Thread 0x7fffa21fc700 (LWP 17309)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 29 (Thread 0x7fffa2bfd700 (LWP 17307)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 28 (Thread 0x7fffa35fe700 (LWP 17306)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 27 (Thread 0x7fffa3fff700 (LWP 17139)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 26 (Thread 0x7fffc4dfa700 (LWP 4654)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 25 (Thread 0x7fffc57fb700 (LWP 4630)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 24 (Thread 0x7fffc61fc700 (LWP 4627)): #0 0x000000300a20dff4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x000000300a209328 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x000000300a2091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x0000003a3525443b in virEventPollUpdateHandle (watch=3518, events=3) at util/event_poll.c:154 #4 0x0000003a353597a4 in virNetSocketUpdateIOCallback (sock=0xe992a0, events=3) at rpc/virnetsocket.c:1707 #5 0x0000003a3534994b in virNetServerClientUpdateEvent (client=0x68e4e0) at rpc/virnetserverclient.c:225 #6 0x0000003a35349a8c in virNetServerClientSendMessageLocked (client=0x68e4e0, msg=0xe99c70) at rpc/virnetserverclient.c:1262 #7 0x0000003a35349ae0 in virNetServerClientSendMessage (client=0x68e4e0, msg=0xe99c70) at rpc/virnetserverclient.c:1275 #8 0x0000003a3534f576 in virNetServerProgramSendError (program=<value optimized out>, version=<value optimized out>, client=0x68e4e0, msg=0xe99c70, rerr=0x7fffc61fbbc0, procedure=10, type=1, serial=4) at rpc/virnetserverprogram.c:170 #9 0x0000003a3534f66d in virNetServerProgramSendReplyError (prog=<value optimized out>, client=<value optimized out>, msg=<value optimized out>, rerr=<value optimized out>, req=<value optimized out>) at rpc/virnetserverprogram.c:201 #10 0x0000003a3534fb94 in virNetServerProgramDispatchCall (prog=0x6894a0, server=<value optimized out>, client=0x68e4e0, msg=0xe99c70) at rpc/virnetserverprogram.c:486 #11 virNetServerProgramDispatch (prog=0x6894a0, server=<value optimized out>, client=0x68e4e0, msg=0xe99c70) at rpc/virnetserverprogram.c:304 #12 0x0000003a3534daa6 in virNetServerProcessMsg (srv=<value optimized out>, client=0x68e4e0, prog=<value optimized out>, msg=0xe99c70) at rpc/virnetserver.c:173 #13 0x0000003a3534e5e3 in virNetServerHandleJob (jobOpaque=<value optimized out>, opaque=0x680e60) at rpc/virnetserver.c:194 #14 0x0000003a35267fdc in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:144 #15 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #16 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #17 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 23 (Thread 0x7fffc6bfd700 (LWP 4622)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 22 (Thread 0x7fffe979e700 (LWP 4619)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fffec73f700 (LWP 3513)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fffed140700 (LWP 3512)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fffedb41700 (LWP 3511)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7fffee542700 (LWP 3510)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7fffeef43700 (LWP 3509)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7fffef944700 (LWP 3508)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7ffff0345700 (LWP 3507)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7ffff0d46700 (LWP 3506)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7ffff1747700 (LWP 3505)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7ffff2148700 (LWP 3504)): #0 0x000000300a20b3dc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000003a35267aa6 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:117 #2 0x0000003a35268073 in virThreadPoolWorker (opaque=<value optimized out>) at util/threadpool.c:103 #3 0x0000003a352678c9 in virThreadHelper (data=<value optimized out>) at util/threads-pthread.c:161 #4 0x000000300a2077f1 in start_thread () from /lib64/libpthread.so.0 #5 0x0000003009ae570d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7ffff7fda800 (LWP 3501)): #0 0x0000003009a74808 in _IO_str_overflow_internal () from /lib64/libc.so.6 #1 0x0000003009a73899 in _IO_default_xsputn_internal () from /lib64/libc.so.6 #2 0x0000003009a47516 in vfprintf () from /lib64/libc.so.6 #3 0x0000003009afed5c in __vasprintf_chk () from /lib64/libc.so.6 #4 0x0000003a3526a2f4 in vasprintf (strp=<value optimized out>, fmt=<value optimized out>, list=<value optimized out>) at /usr/include/bits/stdio2.h:199 #5 virVasprintf (strp=<value optimized out>, fmt=<value optimized out>, list=<value optimized out>) at util/util.c:1990 #6 0x0000003a3526a398 in virAsprintf (strp=<value optimized out>, fmt=<value optimized out>) at util/util.c:2008 #7 0x0000003a3525b4f5 in virLogFormatString (msg=0x7fffffffd440, linenr=353, funcname=0x3a353cddb0 "virEventPollCalculateTimeout", priority=<value optimized out>, str=0x129c410 "Schedule timeout then=1357136203671 now=1357136199821") at util/logging.c:692 #8 0x0000003a3525b99b in virLogVMessage (source=VIR_LOG_FROM_FILE, priority=<value optimized out>, filename=0x3a353cd705 "util/event_poll.c", linenr=353, funcname=0x3a353cddb0 "virEventPollCalculateTimeout", metadata=0x0, fmt=0x3a353cdb10 "Schedule timeout then=%llu now=%llu", vargs=0x7fffffffd4d0) at util/logging.c:818 #9 0x0000003a3525bda9 in virLogMessage (source=<value optimized out>, priority=<value optimized out>, filename=<value optimized out>, linenr=<value optimized out>, funcname=<value optimized out>, metadata=<value optimized out>, fmt=0x3a353cdb10 "Schedule timeout then=%llu now=%llu") at util/logging.c:751 #10 0x0000003a35255368 in virEventPollCalculateTimeout () at util/event_poll.c:353 #11 virEventPollRunOnce () at util/event_poll.c:621 #12 0x0000003a35253ecd in virEventRunDefaultImpl () at util/event.c:247 #13 0x0000003a3534d275 in virNetServerRun (srv=0x680e60) at rpc/virnetserver.c:1121 #14 0x000000000040c76a in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:1449 (gdb)