[libvirt] deadlock while updating VM network filter

Hi, I've encountered deadlock while updating VM network filter and I'm trying to debug it. However I don't have any luck with reproducing it... Here's a backtrace of libvirtd daemon. Any hints on how to fix that would be appreciated:) Libvirt version is 1.2.3. Thread 12 (Thread 0x7fd4550b5700 (LWP 6414)): #0 0x00007fd4585b5d0c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd45887e1ba in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fd45887e89b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fd45887dcf6 in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fd4585b1f3a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd4582ebdad in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x7fd4548b4700 (LWP 6415)): #0 0x00007fd4585b87a4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fd4585b419c in _L_lock_518 () from /lib64/libpthread.so.0 #2 0x00007fd4585b3feb in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fd4588a96c8 in virDomainListPopulate () from /usr/lib64/libvirt.so.0 #4 0x00007fd45884c349 in virHashForEach () from /usr/lib64/libvirt.so.0 #5 0x00007fd4588bc486 in virDomainObjListExport () from /usr/lib64/libvirt.so.0 #6 0x00007fd45891b601 in virConnectListAllDomains () from /usr/lib64/libvirt.so.0 #7 0x00007fd4593b1d2f in remoteDispatchConnectListAllDomainsHelper () #8 0x00007fd4589844d9 in virNetServerProgramDispatch () from /usr/lib64/libvirt.so.0 #9 0x00007fd4593dc4ed in virNetServerHandleJob () #10 0x00007fd45887e7fe in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #11 0x00007fd45887dcf6 in virThreadHelper () from /usr/lib64/libvirt.so.0 #12 0x00007fd4585b1f3a in start_thread () from /lib64/libpthread.so.0 #13 0x00007fd4582ebdad in clone () from /lib64/libc.so.6 Thread 10 (Thread 0x7fd4540b3700 (LWP 6416)): #0 0x00007fd4585b87a4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fd4585b41b7 in _L_lock_565 () from /lib64/libpthread.so.0 #2 0x00007fd4585b4049 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fd44cd54100 in virNWFilterLockIface () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so #4 0x00007fd44cd47925 in _virNWFilterTeardownFilter () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so #5 0x00007fd44cd48910 in virNWFilterTeardownFilter () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so #6 0x00007fd44c63d203 in qemuDomainChangeNet () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007fd44c69b88f in qemuDomainUpdateDeviceFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #8 0x00007fd458920b2b in virDomainUpdateDeviceFlags () from /usr/lib64/libvirt.so.0 #9 0x00007fd4593bc084 in remoteDispatchDomainUpdateDeviceFlagsHelper () #10 0x00007fd4589844d9 in virNetServerProgramDispatch () from /usr/lib64/libvirt.so.0 #11 0x00007fd4593dc4ed in virNetServerHandleJob () #12 0x00007fd45887e7fe in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #13 0x00007fd45887dcf6 in virThreadHelper () from /usr/lib64/libvirt.so.0 #14 0x00007fd4585b1f3a in start_thread () from /lib64/libpthread.so.0 #15 0x00007fd4582ebdad in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x7fd4538b2700 (LWP 6417)): #0 0x00007fd4585b5d0c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00007fd45887e1ba in virCondWait () from /usr/lib64/libvirt.so.0 #2 0x00007fd45887e89b in virThreadPoolWorker () from /usr/lib64/libvirt.so.0 #3 0x00007fd45887dcf6 in virThreadHelper () from /usr/lib64/libvirt.so.0 #4 0x00007fd4585b1f3a in start_thread () from /lib64/libpthread.so.0 #5 0x00007fd4582ebdad in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7fd41b7fe700 (LWP 22858)): #0 0x00007fd4585b87a4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fd4585b41b7 in _L_lock_565 () from /lib64/libpthread.so.0 #2 0x00007fd4585b4049 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fd44cd4873b in virNWFilterInstantiateFilterLate () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so #4 0x00007fd44cd54690 in learnIPAddressThread () from /usr/lib64/libvirt/connection-driver/libvirt_driver_nwfilter.so #5 0x00007fd4585b1f3a in start_thread () from /lib64/libpthread.so.0 #6 0x00007fd4582ebdad in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7fd459372740 (LWP 6413)): #0 0x00007fd4585b87a4 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00007fd4585b419c in _L_lock_518 () from /lib64/libpthread.so.0 #2 0x00007fd4585b3feb in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00007fd44c6452b5 in qemuProcessHandleEvent () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #4 0x00007fd44c65c50d in qemuMonitorEmitEvent () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #5 0x00007fd44c66fc73 in qemuMonitorJSONIOProcess () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #6 0x00007fd44c65ae08 in qemuMonitorIO () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #7 0x00007fd4588476a5 in virEventPollRunOnce () from /usr/lib64/libvirt.so.0 #8 0x00007fd458845b90 in virEventRunDefaultImpl () from /usr/lib64/libvirt.so.0 -- mg

On 21.05.2014 11:40, Marcin Gibuła wrote:
I think deadlock is between thread 2: 1. learnIPAddressThread - virNWFilterLockIface(req->ifname) 2. virNWFilterInstantiateFilterLate - virMutexLock(&updateMutex); and thread 10: 1. virNWFilterTeardownFilter - virMutexLock(&updateMutex) 2. _virNWFilterTeardownFilter - virNWFilterLockIface(ifname) But I don't know call chains well enough to fix it... any ideas? -- mg
participants (1)
-
Marcin Gibuła