[libvirt-users] virsh does no longer respond to commands
Hi, after a recent live backup done with a snapshot, blockcommit failed and a subsequent (manual) blockjob abort got stuck. Now we are unable to do basic operations on the virtual machine like if it was frozen and when issuing another command we see the following: $ virsh shutdown $domain error: Failed to shutdown domain $domain error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort) $ virsh blockcommit $domain --active --pivot error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort) $ virsh blockjob $domain --abort error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort) Houston, we've got a problem. We have roughly 16 hours to fix this. Any help is appreciated! -- Dominik Psenner
Luckily, we were able to sort this problem out by rebooting the hosting machine and restoring the guest machine from a backup. Did we stumble upon a bug? "blockjob abort" should not leave behind orphaned locks when it completed, should it? On 2017-10-22 14:04, Dominik Psenner wrote:
Hi,
after a recent live backup done with a snapshot, blockcommit failed and a subsequent (manual) blockjob abort got stuck. Now we are unable to do basic operations on the virtual machine like if it was frozen and when issuing another command we see the following:
$ virsh shutdown $domain error: Failed to shutdown domain $domain error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
$ virsh blockcommit $domain --active --pivot error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
$ virsh blockjob $domain --abort error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
Houston, we've got a problem. We have roughly 16 hours to fix this. Any help is appreciated!
On 10/22/2017 02:04 PM, Dominik Psenner wrote:
Hi,
after a recent live backup done with a snapshot, blockcommit failed and a subsequent (manual) blockjob abort got stuck. Now we are unable to do basic operations on the virtual machine like if it was frozen and when issuing another command we see the following:
$ virsh shutdown $domain error: Failed to shutdown domain $domain error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
$ virsh blockcommit $domain --active --pivot error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
$ virsh blockjob $domain --abort error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
This looks like some API forgot to unset the job before returning. In that case, restarting libvirtd is the only option. If this happened on the latest release please do file a bug. Otherwise try with the latest release. Michal
It has been the version available on ubuntu 16.04 lts: $ virsh --version 1.3.1 Please file a bug for me if you think it is one. On 25 Oct 2017 2:34 p.m., "Michal Privoznik" <mprivozn@redhat.com> wrote:
On 10/22/2017 02:04 PM, Dominik Psenner wrote:
Hi,
after a recent live backup done with a snapshot, blockcommit failed and a subsequent (manual) blockjob abort got stuck. Now we are unable to do basic operations on the virtual machine like if it was frozen and when issuing another command we see the following:
$ virsh shutdown $domain error: Failed to shutdown domain $domain error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
$ virsh blockcommit $domain --active --pivot error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
$ virsh blockjob $domain --abort error: Timed out during operation: cannot acquire state change lock (held by remoteDispatchDomainBlockJobAbort)
This looks like some API forgot to unset the job before returning. In that case, restarting libvirtd is the only option. If this happened on the latest release please do file a bug. Otherwise try with the latest release.
Michal
On 10/25/2017 07:36 PM, Dominik Psenner wrote:
It has been the version available on ubuntu 16.04 lts:
$ virsh --version 1.3.1
That is ancient libvirt.
Please file a bug for me if you think it is one.
You should file a bug against your distro so that they can backport fixes. I don't use ubuntu, so I can't do it. Michal
participants (2)
-
Dominik Psenner -
Michal Privoznik