Re: [libvirt-users] recovering from deleted snapshot
On Jun 30, 2017 6:22 AM, "Peter Krempa" <pkrempa@redhat.com> wrote:
On Fri, Jun 30, 2017 at 12:05:47 +0200, Peter Krempa wrote:
On Thu, Jun 22, 2017 at 11:02:41 -0400, Doug Hughes wrote:
[...]
virsh blockcommit $vm $DISK --active --pivot
Normally this works fine, though something went wrong on the 20th. something happened to make the blockcommit fail, but the -snap file got deleted (note to self - check return code from blockcommit command!)
So now I'm in a state. The domain i still running. but it's running off the -snapshot that is in the xml. I googled around for how to recover a blockcommit from a deleted snapshot, but didn't find anything. (pointers welcome)
In fact, it's way simpler. If libvirt still knows about the overlay image (this is necessary only so that it can say the proper things to qemu) you can re-do the block commit:
$ virsh list Id Name State ---------------------------------------------------- 3 fedora23 running
$ virsh snapshot-create-as --disk-only --no-metadata fedora23 Domain snapshot 1498817916 created $ virsh domblklist fedora23 Target Source ------------------------------------------------ vda /var/lib/libvirt/images/fedora23.1498817916 hda -
$ rm /var/lib/libvirt/images/fedora23.1498817916 $ ls /var/lib/libvirt/images/fedora23.1498817916 ls: cannot access '/var/lib/libvirt/images/fedora23.1498817916': No such file or directory $ virsh blockcommit --active --pivot fedora23 vda
Successfully pivoted $ virsh domblklist fedora23 Target Source ------------------------------------------------ vda /var/lib/libvirt/images/fedora23.qcow2 hda -
Thanks for the reply! The original image is still there. Only the 1st and only top-level snapshot is deleted. the blockcommit fails though: [root@vm1 ~]# virsh blockcommit serv1r2 vda --active --pivot error: block copy still active: disk 'vda' already in active block job
On Fri, Jun 30, 2017 at 09:23:29 -0400, Doug Hughes wrote:
On Jun 30, 2017 6:22 AM, "Peter Krempa" <pkrempa@redhat.com> wrote:
On Fri, Jun 30, 2017 at 12:05:47 +0200, Peter Krempa wrote:
On Thu, Jun 22, 2017 at 11:02:41 -0400, Doug Hughes wrote:
[...]
file or directory
$ virsh blockcommit --active --pivot fedora23 vda
Successfully pivoted $ virsh domblklist fedora23 Target Source ------------------------------------------------ vda /var/lib/libvirt/images/fedora23.qcow2 hda -
Thanks for the reply! The original image is still there. Only the 1st and only top-level snapshot is deleted. the blockcommit fails though: [root@vm1 ~]# virsh blockcommit serv1r2 vda --active --pivot error: block copy still active: disk 'vda' already in active block job
What does 'virsh blockjob serv1r2 vda' report? If it's something like this: $ virsh blockjob fedora23 vda Active Block Commit: [100 %] Then you just need to finalize the block job by: virsh blockjob --pivot serv1r2 vda If that's not the case, please post what virsh blockjob is reporting and the <disk> part of the XML. In case where libvirt falsely thinks that the job did not finish I'll guide you how to recover it, since restarting libvirtd would make libvirt clear the block job busy flag, but libvirt would lose the metadata.
sorry about top post, tablet mail client... brilliant! that did it! Any idea what happened to cause it to partially fail and get into this state? oot@vm1 ~]# virsh blockjob serv1r2 vda Active Block Commit: [100 %] [root@vm1 ~]# virsh blockjob --pivot serv1r2 vda [root@vm1 ~]# [root@vm1 ~]# ls -l /var/lib/libvirt/images/serv1r2.qcow2 -rw-r--r--. 1 qemu qemu 54200696832 Jun 30 10:39 /var/lib/libvirt/images/serv1r2.qcow2 [root@vm1 ~]# date Fri Jun 30 10:39:12 EDT 2017 [root@vm1 ~]# date Fri Jun 30 10:39:51 EDT 2017 [root@vm1 ~]# ls -l /var/lib/libvirt/images/serv1r2.qcow2 -rw-r--r--. 1 qemu qemu 54200696832 Jun 30 10:40 /var/lib/libvirt/images/serv1r2.qcow2 [root@vm1 ~]# Sent from my android device. -----Original Message----- From: Peter Krempa <pkrempa@redhat.com> To: Doug Hughes <doug.hughes@keystonenap.com> Cc: libvirt-users@redhat.com Sent: Fri, 30 Jun 2017 9:38 Subject: Re: [libvirt-users] recovering from deleted snapshot On Fri, Jun 30, 2017 at 09:23:29 -0400, Doug Hughes wrote:
On Jun 30, 2017 6:22 AM, "Peter Krempa" <pkrempa@redhat.com> wrote:
On Fri, Jun 30, 2017 at 12:05:47 +0200, Peter Krempa wrote:
On Thu, Jun 22, 2017 at 11:02:41 -0400, Doug Hughes wrote:
[...]
file or directory
$ virsh blockcommit --active --pivot fedora23 vda
Successfully pivoted $ virsh domblklist fedora23 Target Source ------------------------------------------------ vda /var/lib/libvirt/images/fedora23.qcow2 hda -
Thanks for the reply! The original image is still there. Only the 1st and only top-level snapshot is deleted. the blockcommit fails though: [root@vm1 ~]# virsh blockcommit serv1r2 vda --active --pivot error: block copy still active: disk 'vda' already in active block job
What does 'virsh blockjob serv1r2 vda' report? If it's something like this: $ virsh blockjob fedora23 vda Active Block Commit: [100 %] Then you just need to finalize the block job by: virsh blockjob --pivot serv1r2 vda If that's not the case, please post what virsh blockjob is reporting and the <disk> part of the XML. In case where libvirt falsely thinks that the job did not finish I'll guide you how to recover it, since restarting libvirtd would make libvirt clear the block job busy flag, but libvirt would lose the metadata.
On Fri, Jun 30, 2017 at 10:41:25 -0400, doug.hughes@keystonenap.com wrote:
sorry about top post, tablet mail client...
brilliant! that did it! Any idea what happened to cause it to partially fail and get into this state?
virsh blockcommit --pivot is actually 3 operations in sequence: 1) initiate the commit job 2) wait till it finishes 3) initiate the pivot job (this is synchronous, so the wait till it finises is itegrated in the API call) Apparently in your case the waiting part failed for some reason and thus the job was not finished, lingering in the synchronised state. The unfortunate part is that you don't know which operation failed. Unfortunately it does not seem that we have wa waiting version of 'virsh blockjob' that could be used to do the waiting. You can do the polling manualy though. The third operation can be split out by default, since you can specify --wait instead of --pivot for 'virsh blockcommit' and it will finish after step 2 above. Also one of the cases which would help is that libvirt could actually remove the file after commit, which would avoid the original problem, since the file would still be there.
oot@vm1 ~]# virsh blockjob serv1r2 vda Active Block Commit: [100 %]
[root@vm1 ~]# virsh blockjob --pivot serv1r2 vda
[root@vm1 ~]# [root@vm1 ~]# ls -l /var/lib/libvirt/images/serv1r2.qcow2 -rw-r--r--. 1 qemu qemu 54200696832 Jun 30 10:39 /var/lib/libvirt/images/serv1r2.qcow2 [root@vm1 ~]# date Fri Jun 30 10:39:12 EDT 2017 [root@vm1 ~]# date Fri Jun 30 10:39:51 EDT 2017 [root@vm1 ~]# ls -l /var/lib/libvirt/images/serv1r2.qcow2 -rw-r--r--. 1 qemu qemu 54200696832 Jun 30 10:40 /var/lib/libvirt/images/serv1r2.qcow2 [root@vm1 ~]#
Yes, I find it annoying that I have to manually remove the no-nos bless leftover snapshot file. That'd be a great feature. In the mean time I will look at adding this additional error handling in my script. Thanks much! On Jun 30, 2017 10:55 AM, "Peter Krempa" <pkrempa@redhat.com> wrote:
On Fri, Jun 30, 2017 at 10:41:25 -0400, doug.hughes@keystonenap.com wrote:
sorry about top post, tablet mail client...
brilliant! that did it! Any idea what happened to cause it to partially fail and get into this state?
virsh blockcommit --pivot is actually 3 operations in sequence: 1) initiate the commit job 2) wait till it finishes 3) initiate the pivot job (this is synchronous, so the wait till it finises is itegrated in the API call)
Apparently in your case the waiting part failed for some reason and thus the job was not finished, lingering in the synchronised state.
The unfortunate part is that you don't know which operation failed.
Unfortunately it does not seem that we have wa waiting version of 'virsh blockjob' that could be used to do the waiting. You can do the polling manualy though.
The third operation can be split out by default, since you can specify --wait instead of --pivot for 'virsh blockcommit' and it will finish after step 2 above.
Also one of the cases which would help is that libvirt could actually remove the file after commit, which would avoid the original problem, since the file would still be there.
oot@vm1 ~]# virsh blockjob serv1r2 vda Active Block Commit: [100 %]
[root@vm1 ~]# virsh blockjob --pivot serv1r2 vda
[root@vm1 ~]# [root@vm1 ~]# ls -l /var/lib/libvirt/images/serv1r2.qcow2 -rw-r--r--. 1 qemu qemu 54200696832 Jun 30 10:39 /var/lib/libvirt/images/ serv1r2.qcow2 [root@vm1 ~]# date Fri Jun 30 10:39:12 EDT 2017 [root@vm1 ~]# date Fri Jun 30 10:39:51 EDT 2017 [root@vm1 ~]# ls -l /var/lib/libvirt/images/serv1r2.qcow2 -rw-r--r--. 1 qemu qemu 54200696832 Jun 30 10:40 /var/lib/libvirt/images/ serv1r2.qcow2 [root@vm1 ~]#
participants (3)
-
Doug Hughes -
doug.hughes＠keystonenap.com
-
Peter Krempa