[libvirt-users] Pivot without copy
Hi, Is it possible to "pivot" to a new image without doing blockcopy or blockpull? I know how to use snapshots and blockpull to create a new image and pivot to using it live, but what I would like to do is to have a VM switch from using imageA.qcow2 to image2.qcow2 while running. I don't see why this wouldn't be possible since some of the existing libvirt tools can do this when they are done. I would love to see an example from a bash terminal as well as how it would be done using the python API. Regards, Mathew
On 05/19/2015 12:52 PM, Mathew Moon wrote:
Hi,
Is it possible to "pivot" to a new image without doing blockcopy or blockpull?
No. Qemu does not support arbitrary reopening of a backing chain yet (even with the 'change-backing-file' QMP command, that is just rewriting contents of the qcow2 metadata, and not actually reopening the chain). The only way to pivot an image on a live guest is via blockcommit or blockcopy. And one of the reasons that this is so is that pivoting only makes sense if you can guarantee that both source and destination have the same guest-visible contents - but without some block job that synchronizes two images just prior to the pivot and then cleanly flushes all pending I/O at the point of the pivot, you can't make that guarantee for any disk image that is being actively modified by guest execution. If you don't mind guest downtime, then you can save the guest state, modify the save file to edit in the updated file name, then restore from the saved state.
I know how to use snapshots and blockpull to create a new image and pivot to using it live, but what I would like to do is to have a VM switch from using imageA.qcow2 to image2.qcow2 while running. I don't see why this wouldn't be possible since some of the existing libvirt tools can do this when they are done. I would love to see an example from a bash terminal as well as how it would be done using the python API.
blockpull is not the most efficient, and forces you to use a qcow2 destination; but it was indeed the first way to accomplish a pivot from one disk to another. But these days, with new enough libvirt and qemu, the ideal pivot uses snapshot-create, blockcopy, and active blockcommit, and lets you pivot to any destination format (does not have to be qcow2). Roughly: $ # create a qcow2 wrapper $ virsh snapshot-create-as $dom --no-metadata --disk-only \ --diskspec vda,file=wrapA $ # copy the backing file to the pivot location, ideally \ taking advantage of faster-than-cp methods $ $your_cool_cp imageA imageB $ # create a destination qcow2 file $ qemu-img create -f qcow2 -obacking_file=imageB,backing_fmt=$fmt wrapB $ # block copy - note below that you may need to make $dom transient $ virsh blockcopy $dom vda wrapB --shallow --reuse-external --pivot \ --verbose $ # commit the wrapper back into the pivoted base $ virsh blockcommit $dom vda --shallow --active --verbose --pivot $ # delete the temporaries $ rm wrapA wrapB If $your_cool_cp is fast, such as taking advantage of a copy-on-write primitive of your storage array, then the overall operation can complete in under a second, because the amount of data stored in the temporary wrapA and copied over to wrapB and finally merged into imageB is so small that it doesn't take long (much faster than blockpull, which copies the entire disk contents regardless of how much changed in the meantime). qemu 2.4 will be adding persistent bitmaps, and the hope is that libvirt can take advantage of that to allow blockcopy to work without requiring a transient domain. But in the meantime, you may have to use 'virsh dumpxml $dom > $file', 'virsh undefine $dom' prior to the blockcopy, then 'edit $file to updated name', 'virsh define $file' afterwards to restore it to persistent. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi Eric, Thanks for the info. I see the value in this, but it isn't quite what I was looking for. Basically what I want to do is to switch between snapshots quickly. For instance, I am currently working on designing a HA SQL implementation with failover. So right now I have 5 VM's running postgresql as a replication group. I am trying a lot of different things and often have to take a snapshot of all 5 VM's, do some work, and then to revert to the previous snapshot I have to: 1. kill the VM 2. edit the VM's config 3. start the VM What I would like is to be able to "pivot" the "current" image to be the previous snapshot (in most cases). I understand that if this were possible then there would be applications crash because something on disk doesn't jive with what is in memory, but I am ok with that, I just want to get my databases back to "square one" quickly after something in the procedure I am working on foo bar's my db and it is replicated to 4 other nodes. Any insights on how this could be accomplished using libvirt without rebooting? On Tue, May 19, 2015 at 2:20 PM, Eric Blake <eblake@redhat.com> wrote:
On 05/19/2015 12:52 PM, Mathew Moon wrote:
Hi,
Is it possible to "pivot" to a new image without doing blockcopy or blockpull?
No. Qemu does not support arbitrary reopening of a backing chain yet (even with the 'change-backing-file' QMP command, that is just rewriting contents of the qcow2 metadata, and not actually reopening the chain). The only way to pivot an image on a live guest is via blockcommit or blockcopy. And one of the reasons that this is so is that pivoting only makes sense if you can guarantee that both source and destination have the same guest-visible contents - but without some block job that synchronizes two images just prior to the pivot and then cleanly flushes all pending I/O at the point of the pivot, you can't make that guarantee for any disk image that is being actively modified by guest execution.
If you don't mind guest downtime, then you can save the guest state, modify the save file to edit in the updated file name, then restore from the saved state.
I know how to use snapshots and blockpull to create a new image and pivot to using it live, but what I would like to do is to have a VM switch from using imageA.qcow2 to image2.qcow2 while running. I don't see why this wouldn't be possible since some of the existing libvirt tools can do this when they are done. I would love to see an example from a bash terminal as well as how it would be done using the python API.
blockpull is not the most efficient, and forces you to use a qcow2 destination; but it was indeed the first way to accomplish a pivot from one disk to another. But these days, with new enough libvirt and qemu, the ideal pivot uses snapshot-create, blockcopy, and active blockcommit, and lets you pivot to any destination format (does not have to be qcow2). Roughly:
$ # create a qcow2 wrapper $ virsh snapshot-create-as $dom --no-metadata --disk-only \ --diskspec vda,file=wrapA $ # copy the backing file to the pivot location, ideally \ taking advantage of faster-than-cp methods $ $your_cool_cp imageA imageB $ # create a destination qcow2 file $ qemu-img create -f qcow2 -obacking_file=imageB,backing_fmt=$fmt wrapB $ # block copy - note below that you may need to make $dom transient $ virsh blockcopy $dom vda wrapB --shallow --reuse-external --pivot \ --verbose $ # commit the wrapper back into the pivoted base $ virsh blockcommit $dom vda --shallow --active --verbose --pivot $ # delete the temporaries $ rm wrapA wrapB
If $your_cool_cp is fast, such as taking advantage of a copy-on-write primitive of your storage array, then the overall operation can complete in under a second, because the amount of data stored in the temporary wrapA and copied over to wrapB and finally merged into imageB is so small that it doesn't take long (much faster than blockpull, which copies the entire disk contents regardless of how much changed in the meantime).
qemu 2.4 will be adding persistent bitmaps, and the hope is that libvirt can take advantage of that to allow blockcopy to work without requiring a transient domain. But in the meantime, you may have to use 'virsh dumpxml $dom > $file', 'virsh undefine $dom' prior to the blockcopy, then 'edit $file to updated name', 'virsh define $file' afterwards to restore it to persistent.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
It seems that my version of libvirt does not support the flags that yours does either. I have the latest version from the Ubuntu repos and am running 14.04. What version are you running and do you know if these features depend on libvirt, or qemu-img to be upgraded? ============================================================================================= root@farnsworth:/var/lib/libvirt/images# virsh -v 1.2.2 root@farnsworth:/var/lib/libvirt/images# qemu-system-x86_64 --version QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.11), Copyright (c) 2003-2008 Fabrice Bellard root@farnsworth:/var/lib/libvirt/images# qemu-img --version qemu-img version 2.0.0, Copyright (c) 2004-2008 Fabrice Bellard usage: qemu-img command [command options] QEMU disk image utility ============================================================================================= On Tue, May 19, 2015 at 3:41 PM, Mathew Moon <mathew.moon@vipaar.com> wrote:
Hi Eric,
Thanks for the info. I see the value in this, but it isn't quite what I was looking for. Basically what I want to do is to switch between snapshots quickly. For instance, I am currently working on designing a HA SQL implementation with failover. So right now I have 5 VM's running postgresql as a replication group. I am trying a lot of different things and often have to take a snapshot of all 5 VM's, do some work, and then to revert to the previous snapshot I have to:
1. kill the VM 2. edit the VM's config 3. start the VM
What I would like is to be able to "pivot" the "current" image to be the previous snapshot (in most cases). I understand that if this were possible then there would be applications crash because something on disk doesn't jive with what is in memory, but I am ok with that, I just want to get my databases back to "square one" quickly after something in the procedure I am working on foo bar's my db and it is replicated to 4 other nodes.
Any insights on how this could be accomplished using libvirt without rebooting?
On Tue, May 19, 2015 at 2:20 PM, Eric Blake <eblake@redhat.com> wrote:
On 05/19/2015 12:52 PM, Mathew Moon wrote:
Hi,
Is it possible to "pivot" to a new image without doing blockcopy or blockpull?
No. Qemu does not support arbitrary reopening of a backing chain yet (even with the 'change-backing-file' QMP command, that is just rewriting contents of the qcow2 metadata, and not actually reopening the chain). The only way to pivot an image on a live guest is via blockcommit or blockcopy. And one of the reasons that this is so is that pivoting only makes sense if you can guarantee that both source and destination have the same guest-visible contents - but without some block job that synchronizes two images just prior to the pivot and then cleanly flushes all pending I/O at the point of the pivot, you can't make that guarantee for any disk image that is being actively modified by guest execution.
If you don't mind guest downtime, then you can save the guest state, modify the save file to edit in the updated file name, then restore from the saved state.
I know how to use snapshots and blockpull to create a new image and pivot to using it live, but what I would like to do is to have a VM switch from using imageA.qcow2 to image2.qcow2 while running. I don't see why this wouldn't be possible since some of the existing libvirt tools can do this when they are done. I would love to see an example from a bash terminal as well as how it would be done using the python API.
blockpull is not the most efficient, and forces you to use a qcow2 destination; but it was indeed the first way to accomplish a pivot from one disk to another. But these days, with new enough libvirt and qemu, the ideal pivot uses snapshot-create, blockcopy, and active blockcommit, and lets you pivot to any destination format (does not have to be qcow2). Roughly:
$ # create a qcow2 wrapper $ virsh snapshot-create-as $dom --no-metadata --disk-only \ --diskspec vda,file=wrapA $ # copy the backing file to the pivot location, ideally \ taking advantage of faster-than-cp methods $ $your_cool_cp imageA imageB $ # create a destination qcow2 file $ qemu-img create -f qcow2 -obacking_file=imageB,backing_fmt=$fmt wrapB $ # block copy - note below that you may need to make $dom transient $ virsh blockcopy $dom vda wrapB --shallow --reuse-external --pivot \ --verbose $ # commit the wrapper back into the pivoted base $ virsh blockcommit $dom vda --shallow --active --verbose --pivot $ # delete the temporaries $ rm wrapA wrapB
If $your_cool_cp is fast, such as taking advantage of a copy-on-write primitive of your storage array, then the overall operation can complete in under a second, because the amount of data stored in the temporary wrapA and copied over to wrapB and finally merged into imageB is so small that it doesn't take long (much faster than blockpull, which copies the entire disk contents regardless of how much changed in the meantime).
qemu 2.4 will be adding persistent bitmaps, and the hope is that libvirt can take advantage of that to allow blockcopy to work without requiring a transient domain. But in the meantime, you may have to use 'virsh dumpxml $dom > $file', 'virsh undefine $dom' prior to the blockcopy, then 'edit $file to updated name', 'virsh define $file' afterwards to restore it to persistent.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 05/19/2015 03:06 PM, Mathew Moon wrote:
It seems that my version of libvirt does not support the flags that yours does either. I have the latest version from the Ubuntu repos and am running 14.04. What version are you running and do you know if these features depend on libvirt, or qemu-img to be upgraded?
============================================================================================= root@farnsworth:/var/lib/libvirt/images# virsh -v 1.2.2
libvirt needs to be at least 1.2.6 for active commit
root@farnsworth:/var/lib/libvirt/images# qemu-system-x86_64 --version QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.11), Copyright (c)
Likewise, qemu needs to be at least 2.2 -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Hi, Sorry about top-posting, didn't know it was a nuisance. As for developing, I wish I could but I have never spent any time coding in c, c++, etc. I am a systems guy, not a developer, so I only work with scripting languages. I would love to have the tech chops to contribute though. If I have a chance I will look through the relevant code and see if there is anything that I can do, but I suspect it will be a bit over my head. I will definitely check out doing snapshot-revert with internal snapshots and hot plugging. With hot plugging I assume you mean to create a storage volume, put it in as a virtual 'cd drive' and mount it that way. Is that correct?
On 05/19/2015 03:29 PM, Mathew Moon wrote:
Hi,
Sorry about top-posting, didn't know it was a nuisance. As for
It's okay - we were all new once.
developing, I wish I could but I have never spent any time coding in c, c++, etc. I am a systems guy, not a developer, so I only work with scripting languages. I would love to have the tech chops to contribute though. If I have a chance I will look through the relevant code and see if there is anything that I can do, but I suspect it will be a bit over my head.
That's okay. Even offering to test patches when they are eventually written is useful.
I will definitely check out doing snapshot-revert with internal snapshots and hot plugging. With hot plugging I assume you mean to create a storage volume, put it in as a virtual 'cd drive' and mount it that way. Is that correct?
Not quite a virtual 'cd drive', but an actual IDE, SCSI, or USB drive, at least as far as the emulation presents it from the guest's point of view. The commands 'virsh attach-disk' and 'virsh detach-disk' are the hot-[un]plug wrappers, which basically behave the same as bare metal system adding or removing a disk from the hardware bus while the guest is running (not all hardware handles it gracefully, but there are definitely storage arrays out there that manage just fine). -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 05/19/2015 02:41 PM, Mathew Moon wrote:
Hi Eric,
[please don't top-post on technical lists]
Thanks for the info. I see the value in this, but it isn't quite what I was looking for. Basically what I want to do is to switch between snapshots quickly. For instance, I am currently working on designing a HA SQL implementation with failover. So right now I have 5 VM's running postgresql as a replication group. I am trying a lot of different things and often have to take a snapshot of all 5 VM's, do some work, and then to revert to the previous snapshot I have to:
1. kill the VM 2. edit the VM's config 3. start the VM
What I would like is to be able to "pivot" the "current" image to be the previous snapshot (in most cases). I understand that if this were possible then there would be applications crash because something on disk doesn't jive with what is in memory, but I am ok with that, I just want to get my databases back to "square one" quickly after something in the procedure I am working on foo bar's my db and it is replicated to 4 other nodes.
That sounds more like snapshot-revert, except that we haven't wired up snapshot revert to work with external snapshots yet. You can experiment with internal snapshots, but those have more guest downtime than external snapshots. Would you like to help write the patches for reverting to external snapshots?
Any insights on how this could be accomplished using libvirt without rebooting?
Snapshot revert will roll back both the disk state and the memory state to an earlier point in guest time. But if you don't care about guest memory, you could possibly hot-unplug the disk that has the state you no longer want, then hot-plug the updated disk that does have what you want. If your guest can handle storage going away and then being plugged in, then that is just as effective as anything that snapshots would do at rolling back to a known disk state. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (2)
-
Eric Blake -
Mathew Moon