Re: investigate locks on a domain

On Tue, Jul 20, 2021 at 15:52:58 +0800, Jiatong Shen wrote: > Hello community, > > I am seeing following log in production, > > 2021-07-20 07:43:49.417+0000: 3918294: error : > qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot > acquire state change lock (held by qemuProcessReconnect) > 2021-07-20 07:44:19.424+0000: 3918296: warning : > qemuDomainObjBeginJobInternal:4933 : Cannot start job (modify, none) for > domain instance-0000074e; current job is (modify, none) owned by (3919429 > qemuProcessReconnect, 0 <null>) for (2183193s, 0s) > 2021-07-20 07:44:19.424+0000: 3918296: error : > qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot > acquire state change lock (held by qemuProcessReconnect) > 2021-07-20 07:44:49.428+0000: 3918296: warning : > qemuDomainObjBeginJobInternal:4933 : Cannot start job (query, none) for > domain instance-0000074e; current job is (modify, none) owned by (3919429 > qemuProcessReconnect, 0 <null>) for (2183223s, 0s) > 2021-07-20 07:44:49.428+0000: 3918296: error : > qemuDomainObjBeginJobInternal:4945 : Timed out during operation: cannot > acquire state change lock (held by qemuProcessReconnect) > 2021-07-20 07:45:19.429+0000: 3918298: warning : > qemuDomainObjBeginJobInternal:4933 : > > I am confused about what is qemuProcessReconnect and why it acquires a > domain state lock.. qemuProcessReconnect is an operation that is executed in a separate thread which re-establishes connection to a qemu process once you restart libvirtd. This usually means that the reconnection process got stuck for some reason. Unfortunately your log doesn't show why and unless you've got a debug log prior to that happening it won't be possible. Theoretically seeing which function the thread doing the reconnect is stuck at could perhaps show why.