[libvirt] libvirt 3.5.0 locks
Hi. I'm testing libvirt 3.5.0 and have many issues like this: Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: error : qemuDomainObjBeginJobInternal:3872 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: warning : qemuDomainObjBeginJobInternal:3860 : Cannot start job (query, none) for domain 147565; current job is (query, none) owned by (18828 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (53670s, 0s) Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.576+0000: 18836: error : virNetDevTapInterfaceStats:751 : internal error: /proc/net/dev: Interface not found Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.542+0000: 18832: warning : qemuGetProcessInfo:1428 : cannot parse process status data Jul 14 09:46:31 cn25 libvirtd[18820]: 2017-07-14 06:46:31.050+0000: 18838: warning : qemuGetProcessInfo:1428 : cannot parse process status data Domain in status in shutdown as i see tap device already removed from the system, so virNetDevTapInterfaceStats says right info. My question - why lock now released and why only libvirt restart can helps in this case -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
On Fri, Jul 14, 2017 at 10:09:00AM +0300, Vasiliy Tolstov wrote:
Hi. I'm testing libvirt 3.5.0 and have many issues like this:
Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: error : qemuDomainObjBeginJobInternal:3872 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: warning : qemuDomainObjBeginJobInternal:3860 : Cannot start job (query, none) for domain 147565; current job is (query, none) owned by (18828 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (53670s, 0s) Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.576+0000: 18836: error : virNetDevTapInterfaceStats:751 : internal error: /proc/net/dev: Interface not found Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.542+0000: 18832: warning : qemuGetProcessInfo:1428 : cannot parse process status data Jul 14 09:46:31 cn25 libvirtd[18820]: 2017-07-14 06:46:31.050+0000: 18838: warning : qemuGetProcessInfo:1428 : cannot parse process status data
Domain in status in shutdown as i see tap device already removed from the system, so virNetDevTapInterfaceStats says right info.
My question - why lock now released and why only libvirt restart can helps in this case
What's be stacktrace of libvirtd in this case?
-- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
2017-07-14 10:27 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
On Fri, Jul 14, 2017 at 10:09:00AM +0300, Vasiliy Tolstov wrote:
Hi. I'm testing libvirt 3.5.0 and have many issues like this:
Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: error : qemuDomainObjBeginJobInternal:3872 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: warning : qemuDomainObjBeginJobInternal:3860 : Cannot start job (query, none) for domain 147565; current job is (query, none) owned by (18828 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (53670s, 0s) Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.576+0000: 18836: error : virNetDevTapInterfaceStats:751 : internal error: /proc/net/dev: Interface not found Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.542+0000: 18832: warning : qemuGetProcessInfo:1428 : cannot parse process status data Jul 14 09:46:31 cn25 libvirtd[18820]: 2017-07-14 06:46:31.050+0000: 18838: warning : qemuGetProcessInfo:1428 : cannot parse process status data
Domain in status in shutdown as i see tap device already removed from the system, so virNetDevTapInterfaceStats says right info.
My question - why lock now released and why only libvirt restart can helps in this case
What's be stacktrace of libvirtd in this case?
How can i get it when new lock appears? -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
On Fri, Jul 14, 2017 at 11:53:15AM +0300, Vasiliy Tolstov wrote:
2017-07-14 10:27 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
On Fri, Jul 14, 2017 at 10:09:00AM +0300, Vasiliy Tolstov wrote:
Hi. I'm testing libvirt 3.5.0 and have many issues like this:
Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: error : qemuDomainObjBeginJobInternal:3872 : Timed out during operation: cannot acquire state change lock (held by remoteDispatchConnectGetAllDomainStats) Jul 14 09:47:25 cn25 libvirtd[18820]: 2017-07-14 06:47:25.598+0000: 18841: warning : qemuDomainObjBeginJobInternal:3860 : Cannot start job (query, none) for domain 147565; current job is (query, none) owned by (18828 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (53670s, 0s) Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.576+0000: 18836: error : virNetDevTapInterfaceStats:751 : internal error: /proc/net/dev: Interface not found Jul 14 09:46:55 cn25 libvirtd[18820]: 2017-07-14 06:46:55.542+0000: 18832: warning : qemuGetProcessInfo:1428 : cannot parse process status data Jul 14 09:46:31 cn25 libvirtd[18820]: 2017-07-14 06:46:31.050+0000: 18838: warning : qemuGetProcessInfo:1428 : cannot parse process status data
Domain in status in shutdown as i see tap device already removed from the system, so virNetDevTapInterfaceStats says right info.
My question - why lock now released and why only libvirt restart can helps in this case
What's be stacktrace of libvirtd in this case?
How can i get it when new lock appears?
gdb -batch -p $(pgrep libvirtd) -ex 't a a bt'
-- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
2017-07-14 12:07 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
What's be stacktrace of libvirtd in this case?
How can i get it when new lock appears?
gdb -batch -p $(pgrep libvirtd) -ex 't a a bt'
I'm try, does i need to have debugging symbols on this host ? -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
On Fri, Jul 14, 2017 at 12:18:06PM +0300, Vasiliy Tolstov wrote:
2017-07-14 12:07 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
What's be stacktrace of libvirtd in this case?
How can i get it when new lock appears?
gdb -batch -p $(pgrep libvirtd) -ex 't a a bt'
I'm try, does i need to have debugging symbols on this host ?
That helps a lot
-- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
2017-07-14 12:21 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
On Fri, Jul 14, 2017 at 12:18:06PM +0300, Vasiliy Tolstov wrote:
2017-07-14 12:07 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
What's be stacktrace of libvirtd in this case?
How can i get it when new lock appears?
gdb -batch -p $(pgrep libvirtd) -ex 't a a bt'
I'm try, does i need to have debugging symbols on this host ?
That helps a lot
I can't now install dbg symbols but can get backtraces: [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fe2d463eaed in poll () from /lib/x86_64-linux-gnu/libc.so.6 Thread 47 (Thread 0x7fe2967fc700 (LWP 23281)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 46 (Thread 0x7fe2b2f99700 (LWP 18710)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 45 (Thread 0x7fe2b379a700 (LWP 18709)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 44 (Thread 0x7fe2b3f9b700 (LWP 18708)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 43 (Thread 0x7fe2b479c700 (LWP 18707)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 42 (Thread 0x7fe2b4f9d700 (LWP 18706)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 41 (Thread 0x7fe2bc7b2700 (LWP 18705)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 40 (Thread 0x7fe2bcfb3700 (LWP 18704)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 39 (Thread 0x7fe2bd7b4700 (LWP 18703)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 38 (Thread 0x7fe2bdfb5700 (LWP 18702)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 37 (Thread 0x7fe2be7b6700 (LWP 18701)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 36 (Thread 0x7fe2befb7700 (LWP 18700)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 35 (Thread 0x7fe2bf7b8700 (LWP 18699)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 34 (Thread 0x7fe2bffb9700 (LWP 18698)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 33 (Thread 0x7fe2c07ba700 (LWP 18697)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 32 (Thread 0x7fe2c0fbb700 (LWP 18696)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 31 (Thread 0x7fe2c17bc700 (LWP 18695)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 30 (Thread 0x7fe2c1fbd700 (LWP 18694)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 29 (Thread 0x7fe2c27be700 (LWP 18693)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 28 (Thread 0x7fe2c2fbf700 (LWP 18692)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 27 (Thread 0x7fe2c37c0700 (LWP 18691)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 26 (Thread 0x7fe2c3fc1700 (LWP 18690)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 25 (Thread 0x7fe2c47c2700 (LWP 18689)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 24 (Thread 0x7fe2c4fc3700 (LWP 18688)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 23 (Thread 0x7fe2c57c4700 (LWP 18687)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 22 (Thread 0x7fe2c5fc5700 (LWP 18686)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 21 (Thread 0x7fe2c67c6700 (LWP 18685)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 20 (Thread 0x7fe2c6fc7700 (LWP 18684)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 19 (Thread 0x7fe2c77c8700 (LWP 18683)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 18 (Thread 0x7fe2c7fc9700 (LWP 18682)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 17 (Thread 0x7fe2c87ca700 (LWP 18681)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 16 (Thread 0x7fe2c8fcb700 (LWP 18680)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 15 (Thread 0x7fe2c97cc700 (LWP 18679)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 14 (Thread 0x7fe2c9fcd700 (LWP 18678)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 13 (Thread 0x7fe2ca7ce700 (LWP 18677)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 12 (Thread 0x7fe2cafcf700 (LWP 18676)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 11 (Thread 0x7fe2cb7d0700 (LWP 18675)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 10 (Thread 0x7fe2cbfd1700 (LWP 18674)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 9 (Thread 0x7fe2cc7d2700 (LWP 18673)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 8 (Thread 0x7fe2ccfd3700 (LWP 18672)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 7 (Thread 0x7fe2cd7d4700 (LWP 18671)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 6 (Thread 0x7fe2cdfd5700 (LWP 18670)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2b5973d5b in qemuMonitorSend () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #3 0x00007fe2b5987ab5 in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007fe2b5989ff1 in qemuMonitorJSONGetMemoryStats () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007fe2b599adb7 in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007fe2b599af7a in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007fe2b599b54d in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007fe2d826e0d5 in virConnectGetAllDomainStats () from /usr/lib/libvirt.so.0 #9 0x00005556cea0ec70 in ?? () #10 0x00007fe2d82ca0d9 in virNetServerProgramDispatch () from /usr/lib/libvirt.so.0 #11 0x00005556cea3fbe8 in ?? () #12 0x00007fe2d81a7598 in ?? () from /usr/lib/libvirt.so.0 #13 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #14 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #15 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 5 (Thread 0x7fe2ce7d6700 (LWP 18669)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 4 (Thread 0x7fe2cefd7700 (LWP 18668)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 3 (Thread 0x7fe2cf7d8700 (LWP 18667)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 2 (Thread 0x7fe2cffd9700 (LWP 18666)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6 Thread 1 (Thread 0x7fe2d8a198c0 (LWP 18664)): #0 0x00007fe2d463eaed in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe2d81514d7 in virEventPollRunOnce () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81500b1 in virEventRunDefaultImpl () from /usr/lib/libvirt.so.0 #3 0x00007fe2d82c446d in virNetDaemonRun () from /usr/lib/libvirt.so.0 #4 0x00005556cea06282 in ?? () #5 0x00007fe2d4580b45 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x00005556cea06c37 in ?? () -- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
Another lock with libvirt 3.5.0 root@cn12:~# gdb -batch -p $(pgrep libvirtd) -ex 't a a bt' warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. 0x00007f95b5d4bd3d in internal_fallocate (fd=-1177453080, offset=4294967295, len=121) at ../sysdeps/posix/posix_fallocate.c:88 88 ../sysdeps/posix/posix_fallocate.c: No such file or directory. Thread 1 (process 16629): #0 0x00007f95b5d4bd3d in internal_fallocate (fd=-1177453080, offset=4294967295, len=121) at ../sysdeps/posix/posix_fallocate.c:88 #1 0x00007f95bc0e4530 in ?? () #2 0x00007f95bc0e4c40 in ?? () #3 0x00007f95bc0e4bd0 in ?? () #4 0x00007f95bc0e4b60 in ?? () #5 0x00007f95bc0e44e0 in ?? () #6 0x0000000000000000 in ?? () root@cn12:~# 2017-07-26 15:15 GMT+03:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:
2017-07-14 12:21 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
On Fri, Jul 14, 2017 at 12:18:06PM +0300, Vasiliy Tolstov wrote:
2017-07-14 12:07 GMT+03:00 Martin Kletzander <mkletzan@redhat.com>:
What's be stacktrace of libvirtd in this case?
How can i get it when new lock appears?
gdb -batch -p $(pgrep libvirtd) -ex 't a a bt'
I'm try, does i need to have debugging symbols on this host ?
That helps a lot
I can't now install dbg symbols but can get backtraces:
[Thread debugging using libthread_db enabled] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". 0x00007fe2d463eaed in poll () from /lib/x86_64-linux-gnu/libc.so.6
Thread 47 (Thread 0x7fe2967fc700 (LWP 23281)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 46 (Thread 0x7fe2b2f99700 (LWP 18710)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 45 (Thread 0x7fe2b379a700 (LWP 18709)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 44 (Thread 0x7fe2b3f9b700 (LWP 18708)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 43 (Thread 0x7fe2b479c700 (LWP 18707)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 42 (Thread 0x7fe2b4f9d700 (LWP 18706)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 41 (Thread 0x7fe2bc7b2700 (LWP 18705)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 40 (Thread 0x7fe2bcfb3700 (LWP 18704)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 39 (Thread 0x7fe2bd7b4700 (LWP 18703)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 38 (Thread 0x7fe2bdfb5700 (LWP 18702)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 37 (Thread 0x7fe2be7b6700 (LWP 18701)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 36 (Thread 0x7fe2befb7700 (LWP 18700)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 35 (Thread 0x7fe2bf7b8700 (LWP 18699)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 34 (Thread 0x7fe2bffb9700 (LWP 18698)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 33 (Thread 0x7fe2c07ba700 (LWP 18697)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 32 (Thread 0x7fe2c0fbb700 (LWP 18696)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 31 (Thread 0x7fe2c17bc700 (LWP 18695)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 30 (Thread 0x7fe2c1fbd700 (LWP 18694)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 29 (Thread 0x7fe2c27be700 (LWP 18693)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 28 (Thread 0x7fe2c2fbf700 (LWP 18692)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 27 (Thread 0x7fe2c37c0700 (LWP 18691)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 26 (Thread 0x7fe2c3fc1700 (LWP 18690)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 25 (Thread 0x7fe2c47c2700 (LWP 18689)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 24 (Thread 0x7fe2c4fc3700 (LWP 18688)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 23 (Thread 0x7fe2c57c4700 (LWP 18687)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 22 (Thread 0x7fe2c5fc5700 (LWP 18686)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a7664 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 21 (Thread 0x7fe2c67c6700 (LWP 18685)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 20 (Thread 0x7fe2c6fc7700 (LWP 18684)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 19 (Thread 0x7fe2c77c8700 (LWP 18683)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 18 (Thread 0x7fe2c7fc9700 (LWP 18682)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 17 (Thread 0x7fe2c87ca700 (LWP 18681)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 16 (Thread 0x7fe2c8fcb700 (LWP 18680)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 15 (Thread 0x7fe2c97cc700 (LWP 18679)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 14 (Thread 0x7fe2c9fcd700 (LWP 18678)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 13 (Thread 0x7fe2ca7ce700 (LWP 18677)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 12 (Thread 0x7fe2cafcf700 (LWP 18676)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 11 (Thread 0x7fe2cb7d0700 (LWP 18675)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 10 (Thread 0x7fe2cbfd1700 (LWP 18674)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 9 (Thread 0x7fe2cc7d2700 (LWP 18673)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 8 (Thread 0x7fe2ccfd3700 (LWP 18672)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 7 (Thread 0x7fe2cd7d4700 (LWP 18671)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 6 (Thread 0x7fe2cdfd5700 (LWP 18670)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2b5973d5b in qemuMonitorSend () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #3 0x00007fe2b5987ab5 in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007fe2b5989ff1 in qemuMonitorJSONGetMemoryStats () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007fe2b599adb7 in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007fe2b599af7a in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007fe2b599b54d in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007fe2d826e0d5 in virConnectGetAllDomainStats () from /usr/lib/libvirt.so.0 #9 0x00005556cea0ec70 in ?? () #10 0x00007fe2d82ca0d9 in virNetServerProgramDispatch () from /usr/lib/libvirt.so.0 #11 0x00005556cea3fbe8 in ?? () #12 0x00007fe2d81a7598 in ?? () from /usr/lib/libvirt.so.0 #13 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #14 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #15 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 5 (Thread 0x7fe2ce7d6700 (LWP 18669)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 4 (Thread 0x7fe2cefd7700 (LWP 18668)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 3 (Thread 0x7fe2cf7d8700 (LWP 18667)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 2 (Thread 0x7fe2cffd9700 (LWP 18666)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81a76a3 in ?? () from /usr/lib/libvirt.so.0 #3 0x00007fe2d81a6968 in ?? () from /usr/lib/libvirt.so.0 #4 0x00007fe2d4912064 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fe2d464762d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Thread 1 (Thread 0x7fe2d8a198c0 (LWP 18664)): #0 0x00007fe2d463eaed in poll () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fe2d81514d7 in virEventPollRunOnce () from /usr/lib/libvirt.so.0 #2 0x00007fe2d81500b1 in virEventRunDefaultImpl () from /usr/lib/libvirt.so.0 #3 0x00007fe2d82c446d in virNetDaemonRun () from /usr/lib/libvirt.so.0 #4 0x00005556cea06282 in ?? () #5 0x00007fe2d4580b45 in __libc_start_main () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x00005556cea06c37 in ?? ()
-- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
-- Vasiliy Tolstov, e-mail: v.tolstov@selfip.ru
On Thu, Aug 24, 2017 at 02:31:05PM +0300, Vasiliy Tolstov wrote:
Another lock with libvirt 3.5.0
root@cn12:~# gdb -batch -p $(pgrep libvirtd) -ex 't a a bt'
warning: Unable to find libthread_db matching inferior's thread library, thread debugging will not be available. 0x00007f95b5d4bd3d in internal_fallocate (fd=-1177453080, offset=4294967295, len=121) at ../sysdeps/posix/posix_fallocate.c:88 88 ../sysdeps/posix/posix_fallocate.c: No such file or directory.
I think you have some problem with your gdb/libthread or something. It showed only one thread.
Thread 1 (process 16629): #0 0x00007f95b5d4bd3d in internal_fallocate (fd=-1177453080, offset=4294967295, len=121) at ../sysdeps/posix/posix_fallocate.c:88 #1 0x00007f95bc0e4530 in ?? () #2 0x00007f95bc0e4c40 in ?? () #3 0x00007f95bc0e4bd0 in ?? () #4 0x00007f95bc0e4b60 in ?? () #5 0x00007f95bc0e44e0 in ?? () #6 0x0000000000000000 in ?? () root@cn12:~#
This is not helpful at all. What do you think I can gather from this? Please, install debugging symbols or run libvirt built with debugging info directly from git.
2017-07-26 15:15 GMT+03:00 Vasiliy Tolstov <v.tolstov@selfip.ru>:
Everything above was pointless until here:
Thread 6 (Thread 0x7fe2cdfd5700 (LWP 18670)): #0 0x00007fe2d491604f in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fe2d81a6bc6 in virCondWait () from /usr/lib/libvirt.so.0
We are waiting...
#2 0x00007fe2b5973d5b in qemuMonitorSend () from
for a reply from qemu monitor...
/usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #3 0x00007fe2b5987ab5 in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007fe2b5989ff1 in qemuMonitorJSONGetMemoryStats () from
after asked about memory statistics...
/usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007fe2b599adb7 in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007fe2b599af7a in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007fe2b599b54d in ?? () from /usr/lib/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007fe2d826e0d5 in virConnectGetAllDomainStats () from
using virConnectGetAllDomainStats(). That's the absolute maximum I can gather from this one. The rest was pointless as well.
participants (2)
-
Martin Kletzander -
Vasiliy Tolstov