[libvirt-users] Efficient live disk backup with active blockcommit : Failed 'block-commit': Could not reopen file: Permission denied
Hi , Working on the simple POC : Advanced snapshot using libvirt and qemu . Following are the exact steps which are followed . 1. Created as base VM - Ubuntu 15.10 with following libvirt and qemu versions Using library: libvirt 1.2.16 Using API: QEMU 1.2.16 Running hypervisor: QEMU 2.3.0 QEMU emulator version 2.3.0 (Debian 1:2.3+dfsg-5ubuntu9.1), Copyright (c) 2003-2008 Fabrice Bellard 2.Created nested VM (vm-01) using VirtManager on base VM. 3.Used following commands to take the multiple snapshots - virsh # list Id Name State ---------------------------------------------------- 7 vm-01 running virsh # domblklist vm-01 Target Source ------------------------------------------------ vda /home/ubuntu/vm-01.img hda /dev/sr0 snapshot-create-as --domain vm-01 snap1 --diskspec vda,file=/var/lib/libvirt/sn1.qcow2 --disk-only --atomic snapshot-create-as --domain vm-01 snap2 --diskspec vda,file=/var/lib/libvirt/sn2.qcow2 --disk-only --atomic virsh #snapshot-list vm-01 ------------------------------------------------------------ snap1 2015-12-07 11:51:51 -0800 disk-snapshot snap2 2015-12-07 11:52:28 -0800 disk-snapshot virsh # snapshot-list vm-01 --tree snap1 | +- snap2 List the current block device in use, again. It can be noticed, the new overlay 'sn2.qcow2' is the current disk in use: virsh # domblklist vm-01 Target Source ------------------------------------------------ vda /var/lib/libvirt/sn2.qcow2 hda /dev/sr0 Now that the backup is finished, perform active blockcommit by live mergning contents of sn2 into base: ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error. Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated. Thanks, Keyur Bhalerao Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/
I had similar issues on Ubuntu 15.10. I’ve downgraded Qemu to 2.1. Now it works like a charm. Van: libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@redhat.com] Namens Keyur Bhalerao Verzonden: maandag 7 december 2015 21:53 Aan: libvirt-users@redhat.com Onderwerp: [libvirt-users] Efficient live disk backup with active blockcommit : Failed 'block-commit': Could not reopen file: Permission denied Hi , Working on the simple POC : Advanced snapshot using libvirt and qemu . Following are the exact steps which are followed . 1. Created as base VM - Ubuntu 15.10 with following libvirt and qemu versions Using library: libvirt 1.2.16 Using API: QEMU 1.2.16 Running hypervisor: QEMU 2.3.0 QEMU emulator version 2.3.0 (Debian 1:2.3+dfsg-5ubuntu9.1), Copyright (c) 2003-2008 Fabrice Bellard 2.Created nested VM (vm-01) using VirtManager on base VM. 3.Used following commands to take the multiple snapshots - virsh # list Id Name State ---------------------------------------------------- 7 vm-01 running virsh # domblklist vm-01 Target Source ------------------------------------------------ vda /home/ubuntu/vm-01.img hda /dev/sr0 snapshot-create-as --domain vm-01 snap1 --diskspec vda,file=/var/lib/libvirt/sn1.qcow2 --disk-only --atomic snapshot-create-as --domain vm-01 snap2 --diskspec vda,file=/var/lib/libvirt/sn2.qcow2 --disk-only --atomic virsh #snapshot-list vm-01 ------------------------------------------------------------ snap1 2015-12-07 11:51:51 -0800 disk-snapshot snap2 2015-12-07 11:52:28 -0800 disk-snapshot virsh # snapshot-list vm-01 --tree snap1 | +- snap2 List the current block device in use, again. It can be noticed, the new overlay 'sn2.qcow2' is the current disk in use: virsh # domblklist vm-01 Target Source ------------------------------------------------ vda /var/lib/libvirt/sn2.qcow2 hda /dev/sr0 Now that the backup is finished, perform active blockcommit by live mergning contents of sn2 into base: ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error. Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated. Thanks, Keyur Bhalerao Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/
On Mon, Dec 07, 2015 at 12:52:34PM -0800, Keyur Bhalerao wrote: [. . .] ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot
error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied
Fairly obvious suggestions, ensure permissions are set so that QEMU has permissions to the dir that contains images. Or it could be an AppArmor issue. Also, check your logs: /var/log/libvirt/qemu/$VM.log /var/log/libvirt/libvirtd.log (assuming you have debug logging enabled)
I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error.
Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated.
Thanks, Keyur Bhalerao
Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/
-- /kashyap
Thanks Kashyap for replying. QEMU has permissions to the directory which is having images. I will look into the AppArmor issue and find out more about this. Also as per suggestion from the link i have uncommented the user and group from the qemu.conf file - https://github.com/jedi4ever/veewee/issues/996 By checking more on the AppArmor and sVirt issue , i have modified the qemu.conf for security_driver="none". The result for this is no permission error but another error which can be related to QEMU 2.3.0 virsh # blockcommit vm-01 vda --verbose --pivot --active Block Commit: [100 %]error: failed to pivot job for disk vda error: internal error: unable to execute QEMU command 'block-job-complete': The active block job for device 'drive-virtio-disk0' cannot be completed Searched more for this type of error and found - https://bugs.launchpad.net/qemu/+bug/1455475 Looks like bug in QEMU 2.3.0 . Checking now with QEMU 2.1 for the same. Let me know if anything to more look for. Thanks, Keyur Bhalerao On Tue, Dec 8, 2015 at 5:18 AM, Kashyap Chamarthy <kchamart@redhat.com> wrote:
On Mon, Dec 07, 2015 at 12:52:34PM -0800, Keyur Bhalerao wrote:
[. . .]
ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot
error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied
Fairly obvious suggestions, ensure permissions are set so that QEMU has permissions to the dir that contains images.
Or it could be an AppArmor issue.
Also, check your logs:
/var/log/libvirt/qemu/$VM.log /var/log/libvirt/libvirtd.log (assuming you have debug logging enabled)
I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error.
Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated.
Thanks, Keyur Bhalerao
Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/
-- /kashyap
Hi Was trying to get specific qemu-kvm version. Do you know any way to get the 2.1.2 installed. Was trying sudo apt-get install qemu-kvm=1.2.1 but its not working. Tried with sudo apt-get install qemu-kvm tree=1.2.1 But both have failed for Version not found. Thanks Keyur On Tue, Dec 8, 2015 at 11:21 AM, Keyur Bhalerao <keyurbhalerao@gmail.com> wrote:
Thanks Kashyap for replying.
QEMU has permissions to the directory which is having images. I will look into the AppArmor issue and find out more about this. Also as per suggestion from the link i have uncommented the user and group from the qemu.conf file - https://github.com/jedi4ever/veewee/issues/996
By checking more on the AppArmor and sVirt issue , i have modified the qemu.conf for security_driver="none". The result for this is no permission error but another error which can be related to QEMU 2.3.0
virsh # blockcommit vm-01 vda --verbose --pivot --active Block Commit: [100 %]error: failed to pivot job for disk vda error: internal error: unable to execute QEMU command 'block-job-complete': The active block job for device 'drive-virtio-disk0' cannot be completed
Searched more for this type of error and found -
https://bugs.launchpad.net/qemu/+bug/1455475
Looks like bug in QEMU 2.3.0 . Checking now with QEMU 2.1 for the same.
Let me know if anything to more look for.
Thanks, Keyur Bhalerao
On Tue, Dec 8, 2015 at 5:18 AM, Kashyap Chamarthy <kchamart@redhat.com> wrote:
On Mon, Dec 07, 2015 at 12:52:34PM -0800, Keyur Bhalerao wrote:
[. . .]
ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot
error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied
Fairly obvious suggestions, ensure permissions are set so that QEMU has permissions to the dir that contains images.
Or it could be an AppArmor issue.
Also, check your logs:
/var/log/libvirt/qemu/$VM.log /var/log/libvirt/libvirtd.log (assuming you have debug logging enabled)
I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error.
Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated.
Thanks, Keyur Bhalerao
Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/
-- /kashyap
This is not a libvirt/redhat question… With your method you can only revert to versions which are available in the standard repositories. You’ll have to download the packages manually to force a different version. Use with caution. There is a reason why the version isn’t in the standard repositories. Van: libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@redhat.com] Namens Keyur Bhalerao Verzonden: woensdag 9 december 2015 0:05 Aan: Kashyap Chamarthy CC: libvirt-users@redhat.com Onderwerp: Re: [libvirt-users] Efficient live disk backup with active blockcommit : Failed 'block-commit': Could not reopen file: Permission denied Hi Was trying to get specific qemu-kvm version. Do you know any way to get the 2.1.2 installed. Was trying sudo apt-get install qemu-kvm=1.2.1 but its not working. Tried with sudo apt-get install qemu-kvm tree=1.2.1 But both have failed for Version not found. Thanks Keyur On Tue, Dec 8, 2015 at 11:21 AM, Keyur Bhalerao <keyurbhalerao@gmail.com<mailto:keyurbhalerao@gmail.com>> wrote: Thanks Kashyap for replying. QEMU has permissions to the directory which is having images. I will look into the AppArmor issue and find out more about this. Also as per suggestion from the link i have uncommented the user and group from the qemu.conf file - https://github.com/jedi4ever/veewee/issues/996 By checking more on the AppArmor and sVirt issue , i have modified the qemu.conf for security_driver="none". The result for this is no permission error but another error which can be related to QEMU 2.3.0 virsh # blockcommit vm-01 vda --verbose --pivot --active Block Commit: [100 %]error: failed to pivot job for disk vda error: internal error: unable to execute QEMU command 'block-job-complete': The active block job for device 'drive-virtio-disk0' cannot be completed Searched more for this type of error and found - https://bugs.launchpad.net/qemu/+bug/1455475 Looks like bug in QEMU 2.3.0 . Checking now with QEMU 2.1 for the same. Let me know if anything to more look for. Thanks, Keyur Bhalerao On Tue, Dec 8, 2015 at 5:18 AM, Kashyap Chamarthy <kchamart@redhat.com<mailto:kchamart@redhat.com>> wrote: On Mon, Dec 07, 2015 at 12:52:34PM -0800, Keyur Bhalerao wrote: [. . .] ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot
error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied
Fairly obvious suggestions, ensure permissions are set so that QEMU has permissions to the dir that contains images. Or it could be an AppArmor issue. Also, check your logs: /var/log/libvirt/qemu/$VM.log /var/log/libvirt/libvirtd.log (assuming you have debug logging enabled)
I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error.
Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated.
Thanks, Keyur Bhalerao
Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/ -- /kashyap
Hi , Based on the discussion and links referred as mentioned in earlier thread , I could able to perform the block commit and other operations successfully. Here are the changes required to perform block commit 1. Used Ubuntu 15.04 which has following versions of Libvirt and Qemu Compiled against library: libvirt 1.2.12 Using library: libvirt 1.2.12 Using API: QEMU 1.2.12 Running hypervisor: QEMU 2.2.0 -- Need to mention here : The Qemu version 2.3 certainly has some issues regarding the block commit as lowering the version to 2.2 resolved "active block job" issue. 2. Modified the configuration /etc/libvirt/qemu.conf - Had to modify the qemu.conf file which is under /etc/libvirt to resolve the permission denied error. - Added security_driver="none" Supportive reference : https://libvirt.org/drvqemu.html I have two question regarding this.- 1. Is there any way we can avoid modifying /etc/libvirt/qemu.conf for security driver changes and perform block commit . ? 2. Has anyone tried with Qemu 2.4 for blockcommit and related operations ? Does it work ? (I know this might not be relevant to libvirt users for sure it will help if someone trying block commit and related operations) Thanks, Keyur Bhalerao On Tue, Dec 8, 2015 at 11:32 PM, Dominique Ramaekers < dominique.ramaekers@cometal.be> wrote:
This is not a libvirt/redhat question…
With your method you can only revert to versions which are available in the standard repositories. You’ll have to download the packages manually to force a different version. Use with caution. There is a reason why the version isn’t in the standard repositories.
*Van:* libvirt-users-bounces@redhat.com [mailto: libvirt-users-bounces@redhat.com] *Namens *Keyur Bhalerao *Verzonden:* woensdag 9 december 2015 0:05 *Aan:* Kashyap Chamarthy *CC:* libvirt-users@redhat.com *Onderwerp:* Re: [libvirt-users] Efficient live disk backup with active blockcommit : Failed 'block-commit': Could not reopen file: Permission denied
Hi
Was trying to get specific qemu-kvm version. Do you know any way to get the 2.1.2 installed. Was trying sudo apt-get install qemu-kvm=1.2.1 but its not working.
Tried with sudo apt-get install qemu-kvm tree=1.2.1
But both have failed for Version not found.
Thanks
Keyur
On Tue, Dec 8, 2015 at 11:21 AM, Keyur Bhalerao <keyurbhalerao@gmail.com> wrote:
Thanks Kashyap for replying.
QEMU has permissions to the directory which is having images. I will look into the AppArmor issue and find out more about this.
Also as per suggestion from the link i have uncommented the user and group from the qemu.conf file - https://github.com/jedi4ever/veewee/issues/996
By checking more on the AppArmor and sVirt issue , i have modified the qemu.conf for security_driver="none". The result for this is no permission error but another error which can be related to QEMU 2.3.0
virsh # blockcommit vm-01 vda --verbose --pivot --active
Block Commit: [100 %]error: failed to pivot job for disk vda
error: internal error: unable to execute QEMU command 'block-job-complete': The active block job for device 'drive-virtio-disk0' cannot be completed
Searched more for this type of error and found -
https://bugs.launchpad.net/qemu/+bug/1455475
Looks like bug in QEMU 2.3.0 . Checking now with QEMU 2.1 for the same.
Let me know if anything to more look for.
Thanks,
Keyur Bhalerao
On Tue, Dec 8, 2015 at 5:18 AM, Kashyap Chamarthy <kchamart@redhat.com> wrote:
On Mon, Dec 07, 2015 at 12:52:34PM -0800, Keyur Bhalerao wrote:
[. . .]
ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot
error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied
Fairly obvious suggestions, ensure permissions are set so that QEMU has permissions to the dir that contains images.
Or it could be an AppArmor issue.
Also, check your logs:
/var/log/libvirt/qemu/$VM.log /var/log/libvirt/libvirtd.log (assuming you have debug logging enabled)
I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error.
Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated.
Thanks, Keyur Bhalerao
Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/
-- /kashyap
Hint, don’t upgrade you libvirt 1.2.12 to 1.2.16 yet. I use 1.2.16 with Qemu 2.2. and now I get a warning at the end of commiting the external snapshot: warning : virEventPollUpdateTimeout:268 : Ignoring invalid update timer -1 All seems to go well so I won’t downgrade libvirt but it still worries me… I’m going to test Qemu 2.4 in Ubuntu 16.04 in march… Van: Keyur Bhalerao [mailto:keyurbhalerao@gmail.com] Verzonden: dinsdag 5 januari 2016 21:53 Aan: Dominique Ramaekers CC: Kashyap Chamarthy; libvirt-users@redhat.com Onderwerp: Re: [libvirt-users] Efficient live disk backup with active blockcommit : Failed 'block-commit': Could not reopen file: Permission denied Hi , Based on the discussion and links referred as mentioned in earlier thread , I could able to perform the block commit and other operations successfully. Here are the changes required to perform block commit 1. Used Ubuntu 15.04 which has following versions of Libvirt and Qemu Compiled against library: libvirt 1.2.12 Using library: libvirt 1.2.12 Using API: QEMU 1.2.12 Running hypervisor: QEMU 2.2.0 -- Need to mention here : The Qemu version 2.3 certainly has some issues regarding the block commit as lowering the version to 2.2 resolved "active block job" issue. 2. Modified the configuration /etc/libvirt/qemu.conf - Had to modify the qemu.conf file which is under /etc/libvirt to resolve the permission denied error. - Added security_driver="none" Supportive reference : https://libvirt.org/drvqemu.html I have two question regarding this.- 1. Is there any way we can avoid modifying /etc/libvirt/qemu.conf for security driver changes and perform block commit . ? 2. Has anyone tried with Qemu 2.4 for blockcommit and related operations ? Does it work ? (I know this might not be relevant to libvirt users for sure it will help if someone trying block commit and related operations) Thanks, Keyur Bhalerao On Tue, Dec 8, 2015 at 11:32 PM, Dominique Ramaekers <dominique.ramaekers@cometal.be<mailto:dominique.ramaekers@cometal.be>> wrote: This is not a libvirt/redhat question… With your method you can only revert to versions which are available in the standard repositories. You’ll have to download the packages manually to force a different version. Use with caution. There is a reason why the version isn’t in the standard repositories. Van: libvirt-users-bounces@redhat.com<mailto:libvirt-users-bounces@redhat.com> [mailto:libvirt-users-bounces@redhat.com<mailto:libvirt-users-bounces@redhat.com>] Namens Keyur Bhalerao Verzonden: woensdag 9 december 2015 0:05 Aan: Kashyap Chamarthy CC: libvirt-users@redhat.com<mailto:libvirt-users@redhat.com> Onderwerp: Re: [libvirt-users] Efficient live disk backup with active blockcommit : Failed 'block-commit': Could not reopen file: Permission denied Hi Was trying to get specific qemu-kvm version. Do you know any way to get the 2.1.2 installed. Was trying sudo apt-get install qemu-kvm=1.2.1 but its not working. Tried with sudo apt-get install qemu-kvm tree=1.2.1 But both have failed for Version not found. Thanks Keyur On Tue, Dec 8, 2015 at 11:21 AM, Keyur Bhalerao <keyurbhalerao@gmail.com<mailto:keyurbhalerao@gmail.com>> wrote: Thanks Kashyap for replying. QEMU has permissions to the directory which is having images. I will look into the AppArmor issue and find out more about this. Also as per suggestion from the link i have uncommented the user and group from the qemu.conf file - https://github.com/jedi4ever/veewee/issues/996 By checking more on the AppArmor and sVirt issue , i have modified the qemu.conf for security_driver="none". The result for this is no permission error but another error which can be related to QEMU 2.3.0 virsh # blockcommit vm-01 vda --verbose --pivot --active Block Commit: [100 %]error: failed to pivot job for disk vda error: internal error: unable to execute QEMU command 'block-job-complete': The active block job for device 'drive-virtio-disk0' cannot be completed Searched more for this type of error and found - https://bugs.launchpad.net/qemu/+bug/1455475 Looks like bug in QEMU 2.3.0 . Checking now with QEMU 2.1 for the same. Let me know if anything to more look for. Thanks, Keyur Bhalerao On Tue, Dec 8, 2015 at 5:18 AM, Kashyap Chamarthy <kchamart@redhat.com<mailto:kchamart@redhat.com>> wrote: On Mon, Dec 07, 2015 at 12:52:34PM -0800, Keyur Bhalerao wrote: [. . .] ubuntu@keyurubuntu:~$ virsh blockcommit vm-01 vda --active --verbose --pivot
error: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied
Fairly obvious suggestions, ensure permissions are set so that QEMU has permissions to the dir that contains images. Or it could be an AppArmor issue. Also, check your logs: /var/log/libvirt/qemu/$VM.log /var/log/libvirt/libvirtd.log (assuming you have debug logging enabled)
I am not sure why this is getting as permission denied. I have tried couple of times reinstalling and creating new VMs . Every time when i have tried to block-commit getting landed into same error.
Is there any troubleshooting or solution to this issue. Stuck on this issue for last 2 days. Quick help much appreciated.
Thanks, Keyur Bhalerao
Reference used - http://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit - http://kashyapc.com/2012/09/14/externaland-live-snapshots-with-libvirt/ -- /kashyap
participants (3)
-
Dominique Ramaekers -
Kashyap Chamarthy -
Keyur Bhalerao