Re: [libvirt] libvirt 3.9.0 locks (was libvirt 3.5.0 locks)

On Fri, Dec 01, 2017 at 15:26:35 +0300, Vasiliy Tolstov wrote: > Hi again. I'm still try to determine why my libvirtd locks, this is > another portion of gdb stuff: > https://gist.github.com/vtolstov/ae8c4a67e15b2fbd14bbb95c226fb427 > > error looks in logs like: > Dec 01 12:35:49 cn04 libvirtd[1171]: 2017-12-01 09:35:49.637+0000: > 28063: error : qemuDomainObjBeginJobInternal:4403 : Timed out during > operation: cannot acquire state change lock (held by > remoteDispatchConnectGetAllDomainStats) > Dec 01 12:35:49 cn04 libvirtd[1171]: 2017-12-01 09:35:49.637+0000: > 28063: warning : qemuDomainObjBeginJobInternal:4391 : Cannot start job > (destroy, none) for domain 225560-20001; current job is (query, none) > owned by (1180 remoteDispatchConnectGetAllDomainStats, 0 <null>) for > (30s, 0s) > > and virsh list says that domain in shutdown state (but qemu process > already exited). > > Please, help me to solve this issue. Looks like qemu did not respond while trying to update vCPU halted state: hread 42 (Thread 0x7f84f96fc700 (LWP 1180)): #0 pthread_cond_wait@(a)GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f8503bf62c6 in virCondWait (c=<optimized out>, m=<optimized out>) at ../../../src/util/virthread.c:154 #2 0x00007f84e13c309b in qemuMonitorSend (mon=0x7f8498000da0, msg=0x80) at ../../../src/qemu/qemu_monitor.c:1067 #3 0x00007f84e13d7045 in qemuMonitorJSONCommandWithFd (mon=0x7f8498000da0, cmd=0x7f84b0001d40, scm_fd=-1, reply=0x7f84f96fb908) at ../../../src/qemu/qemu_monitor_json.c:300 #4 0x00007f84e13d8f67 in qemuMonitorJSONCommand (reply=<optimized out>, cmd=<optimized out>, mon=<optimized out>) at ../../../src/qemu/qemu_monitor_json.c:330 #5 qemuMonitorJSONQueryCPUs (mon=0x7f8498000da0, entries=0x80, nentries=0x6dfa1, force=79) at ../../../src/qemu/qemu_monitor_json.c:1481 #6 0x00007f84e13c5712 in qemuMonitorGetCpuHalted (mon=0x7f8498000da0, maxvcpus=4) at ../../../src/qemu/qemu_monitor.c:2054 #7 0x00007f84e138876c in qemuDomainRefreshVcpuHalted (driver=0x7f84d8151b40, vm=0x7f84d81fe080, asyncJob=0) at ../../../src/qemu/qemu_domain.c:7613 #8 0x00007f84e1406ab0 in qemuDomainGetStatsVcpu (driver=0x7f84d8151b40, dom=0x7f84d81fe080, record=0x7f84b0002248, maxparams=0x7f84f96fbb04, privflags=1) at ../../../src/qemu/qemu_driver.c:19508 Is the guest working at this point?