Re: investigate locks on a domain
On Tue, Jul 20, 2021 at 15:52:58 +0800, Jiatong Shen wrote:
Hello community,
I am seeing following log in production,
2021-07-20 07:43:49.417+0000: 3918294: error :
qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
acquire state change lock (held by qemuProcessReconnect)
2021-07-20 07:44:19.424+0000: 3918296: warning :
qemuDomainObjBeginJobInternal:4933 : Cannot start job (modify, none) for
domain instance-0000074e; current job is (modify, none) owned by (3919429
qemuProcessReconnect, 0 <null>) for (2183193s, 0s)
2021-07-20 07:44:19.424+0000: 3918296: error :
qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
acquire state change lock (held by qemuProcessReconnect)
2021-07-20 07:44:49.428+0000: 3918296: warning :
qemuDomainObjBeginJobInternal:4933 : Cannot start job (query, none) for
domain instance-0000074e; current job is (modify, none) owned by (3919429
qemuProcessReconnect, 0 <null>) for (2183223s, 0s)
2021-07-20 07:44:49.428+0000: 3918296: error :
qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot
acquire state change lock (held by qemuProcessReconnect)
2021-07-20 07:45:19.429+0000: 3918298: warning :
qemuDomainObjBeginJobInternal:4933 :
I am confused about what is qemuProcessReconnect and why it acquires a
domain state lock..
qemuProcessReconnect is an operation that is executed in a separate
thread which re-establishes connection to a qemu process once you
restart libvirtd.
This usually means that the reconnection process got stuck for some
reason. Unfortunately your log doesn't show why and unless you've got a
debug log prior to that happening it won't be possible.
Theoretically seeing which function the thread doing the reconnect is
stuck at could perhaps show why.
Usually the only fix for this is to destroy the VM which is stuck.