[libvirt-users] Can I move the disk image of the guest while it is running?
Hello guys! We want to move some guests' disk images to different locations (within the same machine, locally), but we do not want to turn off the guests. Can we do that while the guest is running, like live migration? I did not find any related documentation or posts about it. Let's say the guest has a disk at /ssd/image.raw We want to move that to /hdd/image.raw Is there any way to do that without interrupting the guest? -- Üdvözlettel / Best regards Horváth Gergely | gergely.horvath@inepex.com IneTrack - Nyomkövetés egyszerűen | Inepex Kft. Ügyfélszolgálat: support@inetrack.hu | +36 30 825 7646 | support.inetrack.hu Web: www.inetrack.hu | nyomkovetes-blog.hu | facebook.com/inetrack Inepex - The White Label GPS fleet-tracking platform | www.inepex.com
On 02/05/2014 08:58 AM, Gergely Horváth wrote:
Hello guys!
We want to move some guests' disk images to different locations (within the same machine, locally), but we do not want to turn off the guests. Can we do that while the guest is running, like live migration?
Yes, live storage migration is possible; although at the moment, qemu is lacking a way to restart the operation if it fails midstream, so libvirt only allows the operation if you are willing to temporarily make your guest transient.
I did not find any related documentation or posts about it.
Let's say the guest has a disk at /ssd/image.raw We want to move that to /hdd/image.raw
Is there any way to do that without interrupting the guest?
Not rigorously tested, but this should do it: virsh dumpxml $dom > file virsh undefine $dom virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot virsh define file -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
Thank you Eric, On 2014-02-05 17:23, Eric Blake wrote:
Yes, live storage migration is possible; although at the moment, qemu is lacking a way to restart the operation if it fails midstream, so libvirt only allows the operation if you are willing to temporarily make your guest transient.
What does this mean? Will I loose anything if - for example - there is not enough space on the target device? Or it will still use the original disk image? AFAIK, a transient guest only means it will disappear after the virtualisation session ends.
virsh dumpxml $dom > file virsh undefine $dom virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot virsh define file
I could not find anything about "pivot" or "pivoting"? What does --pivot do in this case? Thank you. -- Üdvözlettel / Best regards Horváth Gergely | gergely.horvath@inepex.com IneTrack - Nyomkövetés egyszerűen | Inepex Kft. Ügyfélszolgálat: support@inetrack.hu | +36 30 825 7646 | support.inetrack.hu Web: www.inetrack.hu | nyomkovetes-blog.hu | facebook.com/inetrack Inepex - The White Label GPS fleet-tracking platform | www.inepex.com
On 02/05/2014 02:54 PM, Gergely Horváth wrote:
Thank you Eric,
On 2014-02-05 17:23, Eric Blake wrote:
Yes, live storage migration is possible; although at the moment, qemu is lacking a way to restart the operation if it fails midstream, so libvirt only allows the operation if you are willing to temporarily make your guest transient.
What does this mean? Will I loose anything if - for example - there is not enough space on the target device? Or it will still use the original disk image?
If blockcopy fails before mirroring is in sync, such as for insufficient space on the destination, then the source is still valid, with no data loss. If it reaches sync, then both source and destination are copies of each other until you quit the operation (the --wait flag tells virsh to automatically wait for sync, and the --pivot flag says to quit the operation as soon as sync is reached, rather than doing an indefinite mirror). As long as the guest doesn't quit in the meantime, you haven't lost anything. If the guest powers itself off in the middle of the blockcopy, the blockcopy will fail, but the original is still intact at the state it was when the guest shut down.
AFAIK, a transient guest only means it will disappear after the virtualisation session ends.
Correct, which is why you can then re-define the guest with the XML you saved earlier, so that it is once again persistent and can be safely powered off. If blockcopy --pivot succeeded, then the guest uses only the destination, and you have successfully gotten rid of the need to refer to the source.
virsh dumpxml $dom > file virsh undefine $dom virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot virsh define file
Oh, I just realized a caveat - before redefining the file, you'll need to edit it to reflect a successful blockcopy (otherwise, if the guest stops and then is rebooted, the reboot will still use the original source, rather than the destination); a safer set of steps might be: virsh dumpxml $dom > file virsh undefine $dom if virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot; then virsh dumpxml $dom > file # refresh the xml to reflect new disk source fi virsh define file so that file restores you to the original disk location if blockcopy fails, but tracks the new location if blockcopy succeeds. [In my development work, I've been testing on throwaway disks, where I don't care if I clobber things by starting over again - but in a production environment, it pays to be more careful, and possibly to experiment on a safe domain with throwaway disk before doing it on your real domain]
I could not find anything about "pivot" or "pivoting"? What does --pivot do in this case?
--pivot says to break mirroring by using just the destination (in other words, pivot away from the source). The alternative is to use --finish, which says break mirroring by using just the source (in other words, the destination is now a point-in-time snapshot at the time you broke mirroring - useful for backup purposes). And if you are terrified of the possible consequences of a transient domain, there is another possibility for moving your data to a new disk, except that it forces you to convert to qcow2: virsh snapshot-create-as $dom --no-metadata --disk-only \ --diskspec /ssd/image.raw,file=/hdd/image.qcow2 virsh blockpull $dom --wait --verbose and also has the drawback that if the guest quits in the middle of the blockpull, then you have to track both the original file and the snapshot wrapper - but at least blockpull is restartable from where it left off, for the next time the guest is running, whereas blockcopy has to start from scratch. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
----- Original Message -----
From: "Eric Blake" <eblake@redhat.com> To: "Gergely Horváth" <gergely.horvath@inepex.com>, libvirt-users@redhat.com Sent: Wednesday, February 5, 2014 4:47:47 PM Subject: Re: [libvirt-users] Can I move the disk image of the guest while it is running?
On 02/05/2014 02:54 PM, Gergely Horváth wrote:
Thank you Eric,
On 2014-02-05 17:23, Eric Blake wrote:
Yes, live storage migration is possible; although at the moment, qemu is lacking a way to restart the operation if it fails midstream, so libvirt only allows the operation if you are willing to temporarily make your guest transient.
What does this mean? Will I loose anything if - for example - there is not enough space on the target device? Or it will still use the original disk image?
If blockcopy fails before mirroring is in sync, such as for insufficient space on the destination, then the source is still valid, with no data loss. If it reaches sync, then both source and destination are copies of each other until you quit the operation (the --wait flag tells virsh to automatically wait for sync, and the --pivot flag says to quit the operation as soon as sync is reached, rather than doing an indefinite mirror). As long as the guest doesn't quit in the meantime, you haven't lost anything. If the guest powers itself off in the middle of the blockcopy, the blockcopy will fail, but the original is still intact at the state it was when the guest shut down.
AFAIK, a transient guest only means it will disappear after the virtualisation session ends.
Correct, which is why you can then re-define the guest with the XML you saved earlier, so that it is once again persistent and can be safely powered off. If blockcopy --pivot succeeded, then the guest uses only the destination, and you have successfully gotten rid of the need to refer to the source.
virsh dumpxml $dom > file virsh undefine $dom virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot virsh define file
Oh, I just realized a caveat - before redefining the file, you'll need to edit it to reflect a successful blockcopy (otherwise, if the guest stops and then is rebooted, the reboot will still use the original source, rather than the destination); a safer set of steps might be:
virsh dumpxml $dom > file virsh undefine $dom if virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot; then virsh dumpxml $dom > file # refresh the xml to reflect new disk source fi virsh define file
so that file restores you to the original disk location if blockcopy fails, but tracks the new location if blockcopy succeeds.
[In my development work, I've been testing on throwaway disks, where I don't care if I clobber things by starting over again - but in a production environment, it pays to be more careful, and possibly to experiment on a safe domain with throwaway disk before doing it on your real domain]
I could not find anything about "pivot" or "pivoting"? What does --pivot do in this case?
--pivot says to break mirroring by using just the destination (in other words, pivot away from the source). The alternative is to use --finish, which says break mirroring by using just the source (in other words, the destination is now a point-in-time snapshot at the time you broke mirroring - useful for backup purposes).
And if you are terrified of the possible consequences of a transient domain, there is another possibility for moving your data to a new disk, except that it forces you to convert to qcow2:
virsh snapshot-create-as $dom --no-metadata --disk-only \ --diskspec /ssd/image.raw,file=/hdd/image.qcow2 virsh blockpull $dom --wait --verbose
and also has the drawback that if the guest quits in the middle of the blockpull, then you have to track both the original file and the snapshot wrapper - but at least blockpull is restartable from where it left off, for the next time the guest is running, whereas blockcopy has to start from scratch.
Eric, For scripted live backups, which non-transient technique would you prefer: 1. snapshot-create-as followed by blockpull, which results in the original file being left as a standalone copy of the VM from that point in time; it can then be copied off to a backup disk once blockpull completes 2. snapshot-create-as followed by copying the backing file (since it is now read-only) and then using blockcommit to pull the interim changes back into the original/backing file (and removing the snapshot file) For #2 I suppose you could hash the backing file and the copy once it finishes to verify integrity of the backup copy. Thanks, Andrew
On 02/06/2014 08:14 AM, Andrew Martin wrote:
Eric,
For scripted live backups, which non-transient technique would you prefer: 1. snapshot-create-as followed by blockpull, which results in the original file being left as a standalone copy of the VM from that point in time; it can then be copied off to a backup disk once blockpull completes
Slower (because it pulls the entire contents of the original file into the new file), but likely to succeed.
2. snapshot-create-as followed by copying the backing file (since it is now read-only) and then using blockcommit to pull the interim changes back into the original/backing file (and removing the snapshot file)
Ideal, but not supported until (unreleased) qemu 2.0. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
----- Original Message -----
From: "Eric Blake" <eblake@redhat.com> To: "Andrew Martin" <amartin@xes-inc.com> Cc: "Gergely Horváth" <gergely.horvath@inepex.com>, libvirt-users@redhat.com Sent: Thursday, February 6, 2014 9:31:29 AM Subject: Re: [libvirt-users] Can I move the disk image of the guest while it is running?
On 02/06/2014 08:14 AM, Andrew Martin wrote:
Eric,
For scripted live backups, which non-transient technique would you prefer: 1. snapshot-create-as followed by blockpull, which results in the original file being left as a standalone copy of the VM from that point in time; it can then be copied off to a backup disk once blockpull completes
Slower (because it pulls the entire contents of the original file into the new file), but likely to succeed. Is there a faster, non-transient way to do this with the current versions of qemu and libvirt?
2. snapshot-create-as followed by copying the backing file (since it is now read-only) and then using blockcommit to pull the interim changes back into the original/backing file (and removing the snapshot file)
Ideal, but not supported until (unreleased) qemu 2.0.
Ah, does blockcommit currently only work with offline VMs with qemu 1.x?
On 02/06/2014 01:25 PM, Andrew Martin wrote:
----- Original Message -----
From: "Eric Blake" <eblake@redhat.com> To: "Andrew Martin" <amartin@xes-inc.com> Cc: "Gergely Horváth" <gergely.horvath@inepex.com>, libvirt-users@redhat.com Sent: Thursday, February 6, 2014 9:31:29 AM Subject: Re: [libvirt-users] Can I move the disk image of the guest while it is running?
On 02/06/2014 08:14 AM, Andrew Martin wrote:
Eric,
For scripted live backups, which non-transient technique would you prefer: 1. snapshot-create-as followed by blockpull, which results in the original file being left as a standalone copy of the VM from that point in time; it can then be copied off to a backup disk once blockpull completes
Slower (because it pulls the entire contents of the original file into the new file), but likely to succeed. Is there a faster, non-transient way to do this with the current versions of qemu and libvirt?
Sadly, no. The whole area of block chain management is still undergoing active development, and qemu 2.0 definitely adds some nice features to the mix that libvirt will have to be taught to support.
2. snapshot-create-as followed by copying the backing file (since it is now read-only) and then using blockcommit to pull the interim changes back into the original/backing file (and removing the snapshot file)
Ideal, but not supported until (unreleased) qemu 2.0.
Ah, does blockcommit currently only work with offline VMs with qemu 1.x?
Blockcommit does not yet work with offline VMs. But you can get the same behavior as what offline blockcommit would do by using qemu-img manually. One idea for a patch is to see if 'qemu -M none' can drive a block commit operation using a long-running qemu process that does nothing but the block operation; if that works, then we could wire up libvirt to do long-running block operations that survive a libvirtd restart. But the reason libvirt doesn't currently support offline operations is that we have no way to keep a long-running qemu-img operation running smoothly across a libvirtd restart. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
----- Original Message -----
From: "Eric Blake" <eblake@redhat.com> To: "Andrew Martin" <amartin@xes-inc.com> Cc: "Gergely Horváth" <gergely.horvath@inepex.com>, libvirt-users@redhat.com Sent: Thursday, February 6, 2014 2:35:35 PM Subject: Re: [libvirt-users] Can I move the disk image of the guest while it is running?
On 02/06/2014 01:25 PM, Andrew Martin wrote:
----- Original Message -----
From: "Eric Blake" <eblake@redhat.com> To: "Andrew Martin" <amartin@xes-inc.com> Cc: "Gergely Horváth" <gergely.horvath@inepex.com>, libvirt-users@redhat.com Sent: Thursday, February 6, 2014 9:31:29 AM Subject: Re: [libvirt-users] Can I move the disk image of the guest while it is running?
On 02/06/2014 08:14 AM, Andrew Martin wrote:
Eric,
For scripted live backups, which non-transient technique would you prefer: 1. snapshot-create-as followed by blockpull, which results in the original file being left as a standalone copy of the VM from that point in time; it can then be copied off to a backup disk once blockpull completes
Slower (because it pulls the entire contents of the original file into the new file), but likely to succeed. Is there a faster, non-transient way to do this with the current versions of qemu and libvirt?
Sadly, no. The whole area of block chain management is still undergoing active development, and qemu 2.0 definitely adds some nice features to the mix that libvirt will have to be taught to support.
2. snapshot-create-as followed by copying the backing file (since it is now read-only) and then using blockcommit to pull the interim changes back into the original/backing file (and removing the snapshot file)
Ideal, but not supported until (unreleased) qemu 2.0.
Ah, does blockcommit currently only work with offline VMs with qemu 1.x?
Blockcommit does not yet work with offline VMs. But you can get the same behavior as what offline blockcommit would do by using qemu-img manually. One idea for a patch is to see if 'qemu -M none' can drive a block commit operation using a long-running qemu process that does nothing but the block operation; if that works, then we could wire up libvirt to do long-running block operations that survive a libvirtd restart. But the reason libvirt doesn't currently support offline operations is that we have no way to keep a long-running qemu-img operation running smoothly across a libvirtd restart. Thanks for the clarification!
Andrew
On Wed, Feb 05, 2014 at 10:54:15PM +0100, Gergely Horváth wrote:
Thank you Eric,
On 2014-02-05 17:23, Eric Blake wrote:
Yes, live storage migration is possible; although at the moment, qemu is lacking a way to restart the operation if it fails midstream, so libvirt only allows the operation if you are willing to temporarily make your guest transient.
What does this mean? Will I loose anything if - for example - there is not enough space on the target device?
As I understand it, no, you'll not lose anything. Here's some old notes for live backup with 'virsh blockcopy', this was after discussing with Eric http://kashyapc.fedorapeople.org/virt/lc-2012/live-backup-with-blockcopy.txt Currently due to QEMU's limitation, libvirt expects a domain to be trasient. Because, blockcopy jobs last forever until canceled, which implies that they should last across domain restarts if the domain were persistent. But qemu doesn't yet provide a way to restart a copy job on domain restart. So the trick is to temporarily make the domain transient
Or it will still use the original disk image?
AFAIK, a transient guest only means it will disappear after the virtualisation session ends.
The disk image stays intact, just it'll disappear from libvirt's view, hence you take a copy of your XML file below ('virsh dumpxml') does that
virsh dumpxml $dom > file virsh undefine $dom virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot virsh define file
I could not find anything about "pivot" or "pivoting"? What does --pivot do in this case?
When you "pivot", libvirt will use the new copy (i.e. /hdd/image.raw in this case), and this is done while the guest is running. Eric, please correct if I said something wrong. -- /kashyap
On Thu, Feb 06, 2014 at 12:02:11PM +0530, Kashyap Chamarthy wrote:
On Wed, Feb 05, 2014 at 10:54:15PM +0100, Gergely Horváth wrote:
Thank you Eric,
On 2014-02-05 17:23, Eric Blake wrote:
Yes, live storage migration is possible; although at the moment, qemu is lacking a way to restart the operation if it fails midstream, so libvirt only allows the operation if you are willing to temporarily make your guest transient.
What does this mean? Will I loose anything if - for example - there is not enough space on the target device?
As I understand it, no, you'll not lose anything.
Here's some old notes for live backup with 'virsh blockcopy', this was after discussing with Eric
http://kashyapc.fedorapeople.org/virt/lc-2012/live-backup-with-blockcopy.txt
Currently due to QEMU's limitation, libvirt expects a domain to be trasient. Because, blockcopy jobs last forever until canceled, which implies that they should last across domain restarts if the domain were persistent. But qemu doesn't yet provide a way to restart a copy job on domain restart. So the trick is to temporarily make the domain transient
Or it will still use the original disk image?
AFAIK, a transient guest only means it will disappear after the virtualisation session ends.
The disk image stays intact, just it'll disappear from libvirt's view, hence you take a copy of your XML file below ('virsh dumpxml') does that
virsh dumpxml $dom > file virsh undefine $dom virsh blockcopy $dom /ssd/image.raw /hdd/image.raw \ --wait --verbose --pivot virsh define file
I could not find anything about "pivot" or "pivoting"? What does --pivot do in this case?
When you "pivot", libvirt will use the new copy (i.e. /hdd/image.raw in this case), and this is done while the guest is running.
Eric, please correct if I said something wrong.
Ah, Eric already explained it more eloquently (I hit send before seeing his email). -- /kashyap
participants (4)
-
Andrew Martin -
Eric Blake -
Gergely Horváth -
Kashyap Chamarthy