[libvirt-users] libvirtError: block copy still active: disk not ready for pivot yet
Hi guys, When performing live merge, in few cases, we see the following exception: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 736, in wrapper return f(*a, **kw) File "/usr/share/vdsm/virt/vm.py", line 5278, in run self.tryPivot() File "/usr/share/vdsm/virt/vm.py", line 5247, in tryPivot ret = self.vm._dom.blockJobAbort(self.drive.name, flags) File "/usr/share/vdsm/virt/virdomain.py", line 68, in f ret = attr(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper ret = f(*args, **kwargs) File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper return func(inst, *args, **kwargs) File "/usr/lib64/python2.7/site-packages/libvirt.py", line 733, in blockJobAbort if ret == -1: raise libvirtError ('virDomainBlockJobAbort() failed', dom=self) libvirtError: block copy still active: disk 'vdb' not ready for pivot yet That exception observed in following BZs: https://bugzilla.redhat.com/1376580 https://bugzilla.redhat.com/1397122 I am trying to understand what this exception indicates in order to handle it appropriately when thrown by libvirt. Thanks, Ala
On Wed, Dec 28, 2016 at 01:26:12PM +0200, Ala Hino wrote:
Hi guys,
When performing live merge, in few cases, we see the following exception:
[...]
libvirtError: block copy still active: disk 'vdb' not ready for pivot yet
You can see if the block operation is still in progress or not by doing: `sudo virsh blockjob vm1 vdb --info`.
That exception observed in following BZs:
https://bugzilla.redhat.com/1376580 https://bugzilla.redhat.com/1397122
[I see, that the above are oVirt bugs.]
I am trying to understand what this exception indicates in order to handle it appropriately when thrown by libvirt.
Here's the commit that adds the above error message. In short: Until the 'query-block-jobs' QMP command emits the flag BLOCK_JOB_READY, the copy / mirror block operation is truly not complete. commit eae59247c59aa02147b2b4a50177e8e877fdb218 Author: Peter Krempa <pkrempa@redhat.com> Date: Wed Jul 15 15:11:02 2015 +0200 qemu: Update state of block job to READY only if it actually is ready Few parts of the code looked at the current progress of and assumed that a two phase blockjob is in the _READY state as soon as the progress reached 100% (info.cur == info.end). In current versions of qemu this assumption is invalid and qemu exposes a new flag 'ready' in the query-block-jobs output that is set to true if the job is actually finished. This patch adds internal data handling for reading the 'ready' flag and acting appropriately as long as the flag is present. While this still doesn't fix the virsh client problem with two phase block jobs and the --pivot option, it at least improves the error message: $ virsh blockcommit --wait --verbose vm vda --base vda[1] --active --pivot Block commit: [100 %]error: failed to pivot job for disk vda error: internal error: unable to execute QEMU command 'block-job-complete': The active block job for device 'drive-virtio-disk0' cannot be completed to $ virsh blockcommit --wait --verbose VM vda --base vda[1] --active --pivot Block commit: [100 %]error: failed to pivot job for disk vda error: block copy still active: disk 'vda' not ready for pivot yet [...] -- /kashyap
I understand that libvirt provides async events that are triggered once the block job reaches ready state and successfully pivots as it is reported by qemu. Whats the event that is triggered and what's the handler that we have to implement? To be more specific, I am looking for the onEVENT callback method that is invoked when the event is fired. On Tue, Jan 3, 2017 at 6:28 PM, Kashyap Chamarthy <kchamart@redhat.com> wrote:
On Wed, Dec 28, 2016 at 01:26:12PM +0200, Ala Hino wrote:
Hi guys,
When performing live merge, in few cases, we see the following exception:
[...]
libvirtError: block copy still active: disk 'vdb' not ready for pivot yet
You can see if the block operation is still in progress or not by doing: `sudo virsh blockjob vm1 vdb --info`.
That exception observed in following BZs:
https://bugzilla.redhat.com/1376580 https://bugzilla.redhat.com/1397122
[I see, that the above are oVirt bugs.]
I am trying to understand what this exception indicates in order to handle it appropriately when thrown by libvirt.
Here's the commit that adds the above error message. In short: Until the 'query-block-jobs' QMP command emits the flag BLOCK_JOB_READY, the copy / mirror block operation is truly not complete.
commit eae59247c59aa02147b2b4a50177e8e877fdb218 Author: Peter Krempa <pkrempa@redhat.com> Date: Wed Jul 15 15:11:02 2015 +0200
qemu: Update state of block job to READY only if it actually is ready
Few parts of the code looked at the current progress of and assumed that a two phase blockjob is in the _READY state as soon as the progress reached 100% (info.cur == info.end). In current versions of qemu this assumption is invalid and qemu exposes a new flag 'ready' in the query-block-jobs output that is set to true if the job is actually finished.
This patch adds internal data handling for reading the 'ready' flag and acting appropriately as long as the flag is present.
While this still doesn't fix the virsh client problem with two phase block jobs and the --pivot option, it at least improves the error message:
$ virsh blockcommit --wait --verbose vm vda --base vda[1] --active --pivot Block commit: [100 %]error: failed to pivot job for disk vda error: internal error: unable to execute QEMU command 'block-job-complete': The active block job for device 'drive-virtio-disk0' cannot be completed
to
$ virsh blockcommit --wait --verbose VM vda --base vda[1] --active --pivot Block commit: [100 %]error: failed to pivot job for disk vda error: block copy still active: disk 'vda' not ready for pivot yet
[...]
-- /kashyap
On Wed, Jan 04, 2017 at 07:16:03PM +0200, Ala Hino wrote:
I understand that libvirt provides async events that are triggered once the block job reaches ready state and successfully pivots as it is reported by qemu. Whats the event that is triggered and what's the handler that we have to implement?
The exact event is: BLOCK_JOB_READY Also, as Peter Krempa once pointed out, the above is exposed via XML: Libvirt expose the state of the copy job in the XML and forward the READY event from qemu to the users. A running copy job exposes itself in the xml as: <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/systemrescuecd-x86-4.8.0.iso'/> <backingStore/> <mirror type='file' file='/tmp/ble.img' format='raw' job='copy'> <format type='raw'/> <source file='/tmp/ble.img'/> </mirror> [...] </disk> While the ready copy job is exposed as: <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/var/lib/libvirt/images/systemrescuecd-x86-4.8.0.iso'/> <backingStore/> <mirror type='file' file='/tmp/ble.img' format='raw' job='copy' ready='yes'> <format type='raw'/> <source file='/tmp/ble.img'/> </mirror> [...] </disk>
To be more specific, I am looking for the onEVENT callback method that is invoked when the event is fired.
Afraid, I don't know the exact callback you have to implement. Perhaps Peter or Eric Blake can say. But some more related context (lots of reading), as the issue is a bit more complicated. You might also want to look at the below two bugs: https://bugzilla.redhat.com/show_bug.cgi?id=1382165 -- virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs` And (now fixed): https://bugzilla.redhat.com/show_bug.cgi?id=1372613 -- Improve live block device job status reporting via virDomainBlockJobInfo() And may want to go through this entire thread: https://www.redhat.com/archives/libvir-list/2016-October/msg00217.html -- RFC: Exposing "ready" bool (of `query-block-jobs`) or QMP BLOCK_JOB_READY event [...] -- /kashyap
participants (2)
-
Ala Hino -
Kashyap Chamarthy