libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params)
Hi everyone, I have a question that you may be able to help me with. I had a live block migration of a qemu-kvm guest fail (initiated via nova), in which the guest remained running on the source, but if I try to re-initiate the live migration it returns libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params). Looking at blockjob --info I see there is a block copy that has been stuck at 23 % for a few hours now. Is it safe to issue a —abort to the blockjob without impacting the vm and will that resolve the lock? Source: libvirt-4.5.0-23.el7_7.5.x86_64 qemu-kvm-ev-2.10.0-21.el7_5.7.1.x86_64 Destination: libvirt-4.5.0-23.el7_7.5.x86_64 qemu-kvm-ev-2.10.0-21.el7_5.7.1.x86_64 Thanks, Blake
On Thu, Feb 27, 2020 at 18:28:44 -0700, Blake Anderson wrote:
Hi everyone, I have a question that you may be able to help me with. I had a live block migration of a qemu-kvm guest fail (initiated via nova), in which the guest remained running on the source, but if I try to re-initiate the live migration it returns libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params). Looking at blockjob --info I see there is a block copy that has been stuck at 23 % for a few hours now. Is it safe to issue a —abort to the blockjob without impacting the vm and will that resolve the lock?
The domain job lock is held by the whole migration API not just the blockjob. Aborting the blockjob or the migration thus should have the same results after everything cleans up since the failed blockjob should trigger abort of the whole migration anyways. So either of these should do the trick: virsh blockjob --abort $VM $DISK virsh domjobabort $VM Aborting a migration is safe, everything should just continue running at the source.
Hey Peter, Thanks for the advice. It looks like neither of these methods worked. virsh block job —abort $domain $disk returned error: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params) I checked virsh domjobinfo $domain prior to attempted the domjobabort and it just lists Job type: Cancelled The domjobabort returned no output. Do you have any ideas of how to clear the lock short of restarting the qemu process?
On Feb 27, 2020, at 11:49 PM, Peter Krempa <pkrempa@redhat.com> wrote:
On Thu, Feb 27, 2020 at 18:28:44 -0700, Blake Anderson wrote:
Hi everyone, I have a question that you may be able to help me with. I had a live block migration of a qemu-kvm guest fail (initiated via nova), in which the guest remained running on the source, but if I try to re-initiate the live migration it returns libvirtError: Timed out during operation: cannot acquire state change lock (held by monitor=remoteDispatchDomainMigratePerform3Params). Looking at blockjob --info I see there is a block copy that has been stuck at 23 % for a few hours now. Is it safe to issue a —abort to the blockjob without impacting the vm and will that resolve the lock?
The domain job lock is held by the whole migration API not just the blockjob. Aborting the blockjob or the migration thus should have the same results after everything cleans up since the failed blockjob should trigger abort of the whole migration anyways. So either of these should do the trick:
virsh blockjob --abort $VM $DISK
virsh domjobabort $VM
Aborting a migration is safe, everything should just continue running at the source.
participants (2)
-
Blake Anderson -
Peter Krempa