Re: [libvirt] [PATCH] [RFC] nwfilter: resolve deadlock between VM operations and filter update

On 10/07/2010 09:06 AM, Soren Hansen wrote: Great. I will prepare a V3. I am also shooting a kill -SIGHUP at libvirt once in a while to see what happens (while creating / destroying 2 VMs and modifying their filters). Most of the time all goes well, but occasionally things do get stuck. I get the following debugging output from libvirt and attaching gdb to libvirt I see the following stack traces. Maybe Daniel can interpret this... To me it looks like some of the conditions need to be 'tickled'... 09:47:25.000: error : qemuAutostartDomain:822 : Failed to start job on VM 'dummy-vm1': Timed out during operation: cannot acquire state change lock (gdb) thr ap all bt Thread 9 (Thread 0x7f49bf592710 (LWP 17464)): #0 0x000000327680b729 in pthread_cond_timedwait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000435312 in virCondWaitUntil (c=<value optimized out>, m=<value optimized out>, whenms=<value optimized out>) at util/threads-pthread.c:115 #2 0x000000000043d0ab in qemuDomainObjBeginJobWithDriver (driver=0x1f9c010, obj=0x7f49a00011b0) at qemu/qemu_driver.c:409 #3 0x0000000000458abf in qemuAutostartDomain (payload=<value optimized out>, name=<value optimized out>, opaque=0x7f49bf591320) at qemu/qemu_driver.c:818 #4 0x00007f49c040ab6a in virHashForEach (table=0x1f9be20, iter=0x458a90 <qemuAutostartDomain>, data=0x7f49bf591320) at util/hash.c:495 #5 0x000000000043cdac in qemudAutostartConfigs (driver=0x1f9c010) at qemu/qemu_driver.c:855 #6 0x000000000043ce2a in qemudReload () at qemu/qemu_driver.c:2003 #7 0x00007f49c0450a3e in virStateReload () at libvirt.c:1017 #8 0x00000000004189e1 in qemudDispatchSignalEvent ( watch=<value optimized out>, fd=<value optimized out>, events=<value optimized out>, opaque=0x1f6f830) at libvirtd.c:388 ---Type <return> to continue, or q <return> to quit--- #9 0x00000000004186a9 in virEventDispatchHandles () at event.c:479 #10 virEventRunOnce () at event.c:608 #11 0x000000000041a346 in qemudOneLoop () at libvirtd.c:2217 #12 0x000000000041a613 in qemudRunLoop (opaque=0x1f6f830) at libvirtd.c:2326 #13 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #14 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x7f49beb91710 (LWP 17465)): #0 0x000000327680b3bc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000435336 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:100 #2 0x000000000041b2e5 in qemudWorker (data=0x7f49b80008c0) at libvirtd.c:1549 #3 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #4 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x7f49be190710 (LWP 17466)): #0 0x000000327680b3bc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000435336 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:100 #2 0x00000000004716f6 in qemuMonitorSend (mon=0x7f49a00040b0, ---Type <return> to continue, or q <return> to quit--- msg=<value optimized out>) at qemu/qemu_monitor.c:726 #3 0x0000000000473673 in qemuMonitorCommandWithHandler (mon=0x7f49a00040b0, cmd=0x4c316f "info cpus", passwordHandler=0, passwordOpaque=0x0, scm_fd=-1, reply=0x7f49be18e400) at qemu/qemu_monitor_text.c:233 #4 0x000000000047845e in qemuMonitorTextGetCPUInfo ( mon=<value optimized out>, pids=0x7f49be18fa18) at qemu/qemu_monitor_text.c:401 #5 0x0000000000455db2 in qemuDetectVcpuPIDs (conn=0x7f49ac036100, driver=0x1f9c010, vm=<value optimized out>, migrateFrom=0x0, start_paused=false, stdin_fd=-1, stdin_path=0x0) at qemu/qemu_driver.c:2413 #6 qemudStartVMDaemon (conn=0x7f49ac036100, driver=0x1f9c010, vm=<value optimized out>, migrateFrom=0x0, start_paused=false, stdin_fd=-1, stdin_path=0x0) at qemu/qemu_driver.c:3925 #7 0x0000000000457f6b in qemudDomainCreate (conn=0x7f49ac036100, xml=<value optimized out>, flags=0) at qemu/qemu_driver.c:4559 #8 0x00007f49c04564ab in virDomainCreateXML (conn=0x7f49ac036100, xmlDesc=0x7f49a02b8bf0 " <domain type='kvm'>\n <name>dummy-vm2</name>\n <memory>32768</memory>\n <currentMemory>32768</currentMemory>\n <vcpu>1</vcpu>\n <os>\n <type>hvm</type>\n <boot dev='hd'/>\n </os>"..., flags=0) at libvirt.c:1984 #9 0x0000000000427da8 in remoteDispatchDomainCreateXml ( server=<value optimized out>, client=<value optimized out>, ---Type <return> to continue, or q <return> to quit--- conn=0x7f49ac036100, hdr=<value optimized out>, rerr=0x7f49be18fc70, args=<value optimized out>, ret=0x7f49be18fbc0) at remote.c:1271 #10 0x000000000042a657 in remoteDispatchClientCall (server=0x1f6f830, client=0x7f49b83f03c0, msg=0x7f49b8453800) at dispatch.c:529 #11 remoteDispatchClientRequest (server=0x1f6f830, client=0x7f49b83f03c0, msg=0x7f49b8453800) at dispatch.c:407 #12 0x000000000041b378 in qemudWorker (data=0x7f49b80008d8) at libvirtd.c:1570 #13 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #14 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x7f49bd78f710 (LWP 17467)): #0 0x000000327680b3bc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000435336 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:100 #2 0x000000000041b2e5 in qemudWorker (data=0x7f49b80008f0) at libvirtd.c:1549 #3 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #4 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x7f49bcd8e710 (LWP 17468)): #0 0x000000327680b3bc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 ---Type <return> to continue, or q <return> to quit--- #1 0x0000000000435336 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:100 #2 0x00000000004716f6 in qemuMonitorSend (mon=0x7f49ac099740, msg=<value optimized out>) at qemu/qemu_monitor.c:726 #3 0x0000000000473673 in qemuMonitorCommandWithHandler (mon=0x7f49ac099740, cmd=0x7f49ac381620 "balloon 32", passwordHandler=0, passwordOpaque=0x0, scm_fd=-1, reply=0x7f49bcd8c420) at qemu/qemu_monitor_text.c:233 #4 0x0000000000475f45 in qemuMonitorTextSetBalloon (mon=0x7f49ac099740, newmem=<value optimized out>) at qemu/qemu_monitor_text.c:791 #5 0x0000000000455743 in qemudStartVMDaemon (conn=0x7f49a811f9c0, driver=0x1f9c010, vm=<value optimized out>, migrateFrom=0x0, start_paused=false, stdin_fd=-1, stdin_path=0x0) at qemu/qemu_driver.c:3942 #6 0x0000000000457f6b in qemudDomainCreate (conn=0x7f49a811f9c0, xml=<value optimized out>, flags=0) at qemu/qemu_driver.c:4559 #7 0x00007f49c04564ab in virDomainCreateXML (conn=0x7f49a811f9c0, xmlDesc=0x7f49ac2f9f20 " <domain type='kvm'>\n <name>dummy-vm1</name>\n <memory>32768</memory>\n <currentMemory>32768</currentMemory>\n <vcpu>1</vcpu>\n <os>\n <type>hvm</type>\n <boot dev='hd'/>\n </os>"..., flags=0) at libvirt.c:1984 #8 0x0000000000427da8 in remoteDispatchDomainCreateXml ( server=<value optimized out>, client=<value optimized out>, conn=0x7f49a811f9c0, hdr=<value optimized out>, rerr=0x7f49bcd8dc70, ---Type <return> to continue, or q <return> to quit--- args=<value optimized out>, ret=0x7f49bcd8dbc0) at remote.c:1271 #9 0x000000000042a657 in remoteDispatchClientCall (server=0x1f6f830, client=0x7f49b89b41d0, msg=0x7f49b8306460) at dispatch.c:529 #10 remoteDispatchClientRequest (server=0x1f6f830, client=0x7f49b89b41d0, msg=0x7f49b8306460) at dispatch.c:407 #11 0x000000000041b378 in qemudWorker (data=0x7f49b8000908) at libvirtd.c:1570 #12 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #13 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x7f49b7fff710 (LWP 17469)): #0 0x000000327680b3bc in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x0000000000435336 in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/threads-pthread.c:100 #2 0x000000000041b2e5 in qemudWorker (data=0x7f49b8000920) at libvirtd.c:1549 #3 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #4 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 3 (Thread 0x7f486e22b710 (LWP 4599)): #0 0x00000032760d7de3 in poll () from /lib64/libc.so.6 #1 0x0000003279c07a02 in ?? () from /usr/lib64/libpcap.so.1 #2 0x0000003279c0c65b in pcap_next () from /usr/lib64/libpcap.so.1 ---Type <return> to continue, or q <return> to quit--- #3 0x00000000004afce4 in learnIPAddressThread (arg=0x7f49b82de290) at nwfilter/nwfilter_learnipaddr.c:496 #4 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #5 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x7f486d82a710 (LWP 4897)): #0 0x00000032760d7de3 in poll () from /lib64/libc.so.6 #1 0x0000003279c07a02 in ?? () from /usr/lib64/libpcap.so.1 #2 0x0000003279c0c65b in pcap_next () from /usr/lib64/libpcap.so.1 #3 0x00000000004afce4 in learnIPAddressThread (arg=0x7f49b8802310) at nwfilter/nwfilter_learnipaddr.c:496 #4 0x0000003276807761 in start_thread () from /lib64/libpthread.so.0 #5 0x00000032760e14ed in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x7f49c03c27e0 (LWP 17460)): #0 0x0000003276807fbd in pthread_join () from /lib64/libpthread.so.0 #1 0x000000000041e434 in main (argc=<value optimized out>, argv=<value optimized out>) at libvirtd.c:3272 (gdb)