5:26 a.m.
Hi guys,
When performing live merge, in few cases, we see the following exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 736, in wrapper
return f(*a, **kw)
File "/usr/share/vdsm/virt/vm.py", line 5278, in run
self.tryPivot()
File "/usr/share/vdsm/virt/vm.py", line 5247, in tryPivot
ret = self.vm._dom.blockJobAbort(self.drive.name, flags)
File "/usr/share/vdsm/virt/virdomain.py", line 68, in f
ret = attr(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py",
line 124, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 733, in
blockJobAbort
if ret == -1: raise libvirtError ('virDomainBlockJobAbort()
failed', dom=self)
libvirtError: block copy still active: disk 'vdb' not ready for pivot yet
That exception observed in following BZs:
https://bugzilla.redhat.com/1376580
https://bugzilla.redhat.com/1397122
I am trying to understand what this exception indicates in order to handle
it appropriately when thrown by libvirt.
Thanks,
Ala
10:28 a.m.
On Wed, Dec 28, 2016 at 01:26:12PM +0200, Ala Hino wrote:
Hi guys,
When performing live merge, in few cases, we see the following exception:
[...]
libvirtError: block copy still active: disk 'vdb' not ready
for pivot yet
You can see if the block operation is still in progress or not by doing:
`sudo virsh blockjob vm1 vdb --info`.
That exception observed in following BZs:
https://bugzilla.redhat.com/1376580
https://bugzilla.redhat.com/1397122
[I see, that the above are oVirt bugs.]
I am trying to understand what this exception indicates in order to
handle
it appropriately when thrown by libvirt.
Here's the commit that adds the above error message. In short: Until
the 'query-block-jobs' QMP command emits the flag BLOCK_JOB_READY, the
copy / mirror block operation is truly not complete.
commit eae59247c59aa02147b2b4a50177e8e877fdb218
Author: Peter Krempa <pkrempa(a)redhat.com>
Date: Wed Jul 15 15:11:02 2015 +0200
qemu: Update state of block job to READY only if it actually is ready
Few parts of the code looked at the current progress of and assumed that
a two phase blockjob is in the _READY state as soon as the progress
reached 100% (info.cur == info.end). In current versions of qemu this
assumption is invalid and qemu exposes a new flag 'ready' in the
query-block-jobs output that is set to true if the job is actually
finished.
This patch adds internal data handling for reading the 'ready' flag and
acting appropriately as long as the flag is present.
While this still doesn't fix the virsh client problem with two phase
block jobs and the --pivot option, it at least improves the error
message:
$ virsh blockcommit --wait --verbose vm vda --base vda[1] --active --pivot
Block commit: [100 %]error: failed to pivot job for disk vda
error: internal error: unable to execute QEMU command
'block-job-complete': The active block job for device 'drive-virtio-disk0'
cannot be completed
to
$ virsh blockcommit --wait --verbose VM vda --base vda[1] --active --pivot
Block commit: [100 %]error: failed to pivot job for disk vda
error: block copy still active: disk 'vda' not ready for pivot yet
[...]
--
/kashyap
11:16 a.m.
I understand that libvirt provides async events that are triggered once the
block job reaches ready state and successfully pivots as it is reported by
qemu.
Whats the event that is triggered and what's the handler that we have to
implement?
To be more specific, I am looking for the onEVENT callback method that is
invoked when the event is fired.
On Tue, Jan 3, 2017 at 6:28 PM, Kashyap Chamarthy <kchamart(a)redhat.com>
wrote:
On Wed, Dec 28, 2016 at 01:26:12PM +0200, Ala Hino wrote:
> Hi guys,
>
> When performing live merge, in few cases, we see the following exception:
[...]
> libvirtError: block copy still active: disk 'vdb' not ready for pivot yet
You can see if the block operation is still in progress or not by doing:
`sudo virsh blockjob vm1 vdb --info`.
> That exception observed in following BZs:
>
> https://bugzilla.redhat.com/1376580
> https://bugzilla.redhat.com/1397122
[I see, that the above are oVirt bugs.]
> I am trying to understand what this exception indicates in order to
handle
> it appropriately when thrown by libvirt.
Here's the commit that adds the above error message. In short: Until
the 'query-block-jobs' QMP command emits the flag BLOCK_JOB_READY, the
copy / mirror block operation is truly not complete.
commit eae59247c59aa02147b2b4a50177e8e877fdb218
Author: Peter Krempa <pkrempa(a)redhat.com>
Date: Wed Jul 15 15:11:02 2015 +0200
qemu: Update state of block job to READY only if it actually is
ready
Few parts of the code looked at the current progress of and
assumed that
a two phase blockjob is in the _READY state as soon as the progress
reached 100% (info.cur == info.end). In current versions of qemu
this
assumption is invalid and qemu exposes a new flag 'ready' in the
query-block-jobs output that is set to true if the job is actually
finished.
This patch adds internal data handling for reading the 'ready'
flag and
acting appropriately as long as the flag is present.
While this still doesn't fix the virsh client problem with two
phase
block jobs and the --pivot option, it at least improves the error
message:
$ virsh blockcommit --wait --verbose vm vda --base vda[1]
--active --pivot
Block commit: [100 %]error: failed to pivot job for disk vda
error: internal error: unable to execute QEMU command
'block-job-complete': The active block job for device
'drive-virtio-disk0'
cannot be completed
to
$ virsh blockcommit --wait --verbose VM vda --base vda[1]
--active --pivot
Block commit: [100 %]error: failed to pivot job for disk vda
error: block copy still active: disk 'vda' not ready for pivot yet
[...]
--
/kashyap
4:16 a.m.
On Wed, Jan 04, 2017 at 07:16:03PM +0200, Ala Hino wrote:
I understand that libvirt provides async events that are triggered
once the
block job reaches ready state and successfully pivots as it is reported by
qemu.
Whats the event that is triggered and what's the handler that we have to
implement?
The exact event is: BLOCK_JOB_READY
Also, as Peter Krempa once pointed out, the above is exposed via XML:
Libvirt expose the state of the copy job in the XML and forward the
READY event from qemu to the users.
A running copy job exposes itself in the xml as:
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source
file='/var/lib/libvirt/images/systemrescuecd-x86-4.8.0.iso'/>
<backingStore/>
<mirror type='file' file='/tmp/ble.img' format='raw'
job='copy'>
<format type='raw'/>
<source file='/tmp/ble.img'/>
</mirror>
[...]
</disk>
While the ready copy job is exposed as:
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source
file='/var/lib/libvirt/images/systemrescuecd-x86-4.8.0.iso'/>
<backingStore/>
<mirror type='file' file='/tmp/ble.img' format='raw'
job='copy' ready='yes'>
<format type='raw'/>
<source file='/tmp/ble.img'/>
</mirror>
[...]
</disk>
To be more specific, I am looking for the onEVENT callback method
that is
invoked when the event is fired.
Afraid, I don't know the exact callback you have to implement. Perhaps
Peter or Eric Blake can say.
But some more related context (lots of reading), as the issue is a bit
more complicated.
You might also want to look at the below two bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1382165 --
virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats &
the "ready" field of `query-block-jobs`
And (now fixed):
https://bugzilla.redhat.com/show_bug.cgi?id=1372613 -- Improve live
block device job status reporting via virDomainBlockJobInfo()
And may want to go through this entire thread:
https://www.redhat.com/archives/libvir-list/2016-October/msg00217.html
-- RFC: Exposing "ready" bool (of `query-block-jobs`) or QMP
BLOCK_JOB_READY event
[...]
--
/kashyap
2908
days inactive
2916
days old
3 comments
2 participants
participants (2)
-
Ala Hino -
Kashyap Chamarthy