On 01/17/2018 03:45 PM, Serhii Kharchenko wrote:
Hello libvirt-users list,
We're catching the same bug since 3.4.0 version (3.3.0 works OK).
So, we have process that is permanently connected to libvirtd via socket
and it is collecting stats, listening to events and control the VPSes.
When we try to 'shutdown' a number of VPSes we often catch the bug. One of
VPSes sticks in 'in shutdown' state, no related 'qemu' process is
present,
and there is the next error in the log:
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.005+0000:
20438: warning : qemuGetProcessInfo:1460 : cannot parse process status data
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
20441: error : virFileReadAll:1420 : Failed to open file
'/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage':
No such file or directory
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
20441: error : virCgroupGetValueStr:844 : Unable to read from
'/sys/fs/cgroup/cpu,cpuacct/machine.slice/machine-qemu\x2d36\x2dDOMAIN1.scope/cpuacct.usage':
No such file or directory
Jan 17 13:54:20 server1 libvirtd[20437]: 2018-01-17 13:54:20.006+0000:
20441: error : virCgroupGetDomainTotalCpuStats:3319 : unable to get cpu
account: Operation not permitted
Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000:
20522: warning : qemuDomainObjBeginJobInternal:4862 : Cannot start job
(destroy, none) for domain DOMAIN1; current job is (query, none) owned by
(20440 remoteDispatchConnectGetAllDomainStats, 0 <null>) for (30s, 0s)
Jan 17 13:54:23 server1 libvirtd[20437]: 2018-01-17 13:54:23.805+0000:
20522: error : qemuDomainObjBeginJobInternal:4874 : Timed out during
operation: cannot acquire state change lock (held by
remoteDispatchConnectGetAllDomainStats)
Sounds like qemuConnectGetAllDomainStats() forgot to unset the job on a
domain. Can you please attach gdb to libvirtd and run 't a a bt' and
share the output?
Michal