4:26 p.m.
This notes is based on an IRC conversation with Eric Blake, to have
efficient non-shared storage live migration. Thought I'd post my notes
here before I forget. Please review and spot if there are any
inaccuracies.
Procedure
---------
(1) Starting from disk A, create a snapshot A <- A':
$ virsh snapshot-create-as \
--domain f20vm snap1 snap1-desc \
--diskspec hda,file=/export/vmimages/A'.qcow2 \
--disk-only --atomic
(2) Background copy of A to B:
$ virsh blockcopy \
--domain vm1 vda /export/vmimages/B.qcow2 \
--wait --verbose --shallow \
--finish
(3) Create an empty B' with backing file B:
$ qemu-img create -f qcow2 -b B.qcow2 \
-o backing_fmt=qcow2 B'.qcow2
[or]
$ virsh vol-create-as default B'.qcow2 1G \
--format qcow2 \
--backing-vol B.qcow2 --backing-vol-format qcow2
(4) Do a shallow blockcopy of A' to B':
$ virsh blockcopy \
--domain vm1 vda /export/vmimages/B'.qcow2 \
--wait --verbose --shallow \
--finish
(5) Then live shallow commit of B:
$ virsh blockcommit \
--domain f20vm vda \
--wait --verbose --shallow \
--pivot --active --finish
Block Commit: [100 %]
Successfully pivoted
--
/kashyap
1:24 a.m.
On Thu, Sep 25, 2014 at 07:56:09PM +0530, Kashyap Chamarthy wrote:
This notes is based on an IRC conversation with Eric Blake, to have
efficient non-shared storage live migration. Thought I'd post my notes
here before I forget. Please review and spot if there are any
inaccuracies.
Noting a couple of things I missed. . .
Procedure
---------
(1) Starting from disk A, create a snapshot A <- A':
$ virsh snapshot-create-as \
--domain f20vm snap1 snap1-desc \
--diskspec hda,file=/export/vmimages/A'.qcow2 \
--disk-only --atomic
Before performing a live blockcopy, make the domain transient (as
persistent dirty bitmap support is yet to arrive in QEMU).
Take backup of the guest XML:
$ virsh dumpxml f20vm > /var/tmp/f20vm.xml
Undefine the running guest, thus turning it to a transient guest:
$ virsh undefine f20vm
(2) Background copy of A to B:
$ virsh blockcopy \
--domain vm1 vda /export/vmimages/B.qcow2 \
--wait --verbose --shallow \
--finish
(3) Create an empty B' with backing file B:
$ qemu-img create -f qcow2 -b B.qcow2 \
-o backing_fmt=qcow2 B'.qcow2
[or]
$ virsh vol-create-as default B'.qcow2 1G \
--format qcow2 \
--backing-vol B.qcow2 --backing-vol-format qcow2
(4) Do a shallow blockcopy of A' to B':
$ virsh blockcopy \
--domain vm1 vda /export/vmimages/B'.qcow2 \
--wait --verbose --shallow \
--finish
Since the a chain was already created in step (3), I should have used
'--reuse-external' flag at this shallow blockcopy.
(5) Then live shallow commit of B:
$ virsh blockcommit \
--domain f20vm vda \
--wait --verbose --shallow \
--pivot --active --finish
Block Commit: [100 %]
Successfully pivoted
--
/kashyap
11:33 a.m.
On Thu, Sep 25, 2014 at 3:26 PM, Kashyap Chamarthy <kchamart(a)redhat.com> wrote:
This notes is based on an IRC conversation with Eric Blake, to have
efficient non-shared storage live migration. Thought I'd post my notes
here before I forget. Please review and spot if there are any
inaccuracies.
What are you trying to achieve?
Stefan
12:51 p.m.
On Mon, Sep 29, 2014 at 10:33:37AM +0100, Stefan Hajnoczi wrote:
On Thu, Sep 25, 2014 at 3:26 PM, Kashyap Chamarthy
<kchamart(a)redhat.com> wrote:
> This notes is based on an IRC conversation with Eric Blake, to have
> efficient non-shared storage live migration. Thought I'd post my notes
> here before I forget. Please review and spot if there are any
> inaccuracies.
What are you trying to achieve?
Hmm, I clearly failed to articulate what I was trying to do then. Allow
me to rephrase.
So, this was just an attempt out of curiosity derived from an IRC
conversation with Eric Blake on #virt (OFTC) some months ago -- once
libvirt starts using QEMU 2.0's (block) commit of active layer of a disk
image chain into its immediate base or one of the intermediate images,
live migration with non-shared storage should be more efficient (with a
combination of blockcopy+blockcommit).
Now that all the relevant bits are in place for awhile, So, I was just
trying to test that works out.
- Relevant git commit from libvirt:
318cfabdb442f41c9a7016889526c67aad26a503 (blockcommit: document
semantics of committing active layer)
- Relevant git commit from QEMU:
20a63d2cec838c2dde4d246c4d7abe747d9b7a11 (commit: Support commit
active layer)
Hope I explain it a bit more clear.
More context: I intend to a do user/admin style talk at the upcoming
LinuxCon/CloudOpen in Düsseldorf, so was just testing and experimenting
to see what kind of practical steps can virt/cloud software (like
OpenStack) users/adminstrators can take advantage of the newer features
in libvirt/QEMU when dealing with snapshots and large disk image
chains/merge. To keep things less complicated, I'm mainly tinkering with
file-based snapshots.
https://kashyapc.fedorapeople.org/virt/lcce-2014/Abstract-CloudOpen-Eu-20...
--
/kashyap
1:35 a.m.
On 09/25/2014 08:26 AM, Kashyap Chamarthy wrote:
This notes is based on an IRC conversation with Eric Blake, to have
efficient non-shared storage live migration. Thought I'd post my notes
here before I forget. Please review and spot if there are any
inaccuracies.
Procedure
---------
(1) Starting from disk A, create a snapshot A <- A':
$ virsh snapshot-create-as \
--domain f20vm snap1 snap1-desc \
--diskspec hda,file=/export/vmimages/A'.qcow2 \
--disk-only --atomic
If you are using this snapshot only for the side-effect of growing the
chain, you can add --no-metadata here instead of deleting the snapshot
later when it gets invalidated [1]. Of course, if you pass
--no-metadata, the snapshot name (snap1) and description (snap1-desc)
are no longer important.
(2) Background copy of A to B:
$ virsh blockcopy \
--domain vm1 vda /export/vmimages/B.qcow2 \
--wait --verbose --shallow \
--finish
This step is not quite right. You are asking for a shallow copy of the
current file for disk 'vda' (that is, A'.qcow2). But that is NOT the
same as the base A image. For this step, libvirt does not yet have an
easy way to access the contents of a backing chain of a live domain; you
CAN use 'virsh vol-*' commands to do a background copy from storage
pools, but it may be easier to just resort to normal file system tools:
cp /export/vmimages/A.qcow2 /export/vmimages/B.qcow2
or even rely on storage-array-specific commands to set up a trivial
clone with no real time overhead (for example, some iscsi storage arrays
allow efficient copy-on-write cloning of storage volumes by creating a
new name that shares the same original contents of A.qcow2 as its
starting point; and since we are about to delete A.qcow2 later on, we
never need any actual data copying).
(3) Create an empty B' with backing file B:
$ qemu-img create -f qcow2 -b B.qcow2 \
-o backing_fmt=qcow2 B'.qcow2
[or]
$ virsh vol-create-as default B'.qcow2 1G \
--format qcow2 \
--backing-vol B.qcow2 --backing-vol-format qcow2
[side note - we should really teach libvirt to not REQUIRE a size when
creating an empty wrapper around an existing image]
(4) Do a shallow blockcopy of A' to B':
$ virsh blockcopy \
--domain vm1 vda /export/vmimages/B'.qcow2 \
--wait --verbose --shallow \
--finish
For this to work, you need to also use the --reuse-external flag to take
advantage of the backing chain already recorded in B'.qcow2 (without the
flag, the command will complain that B'.qcow2 already exists if it is a
regular file; if it is a block device, it will just silently ignore the
contents of the block device and treat B'.qcow2 as though an absolute
path to A.qcow2 were its backing file).
(5) Then live shallow commit of B:
$ virsh blockcommit \
--domain f20vm vda \
--wait --verbose --shallow \
--pivot --active --finish
Block Commit: [100 %]
Successfully pivoted
With steps 2 and 4 corrected, this indeed shortens the chain back down
to just B.qcow2. And once this happens, you no longer need the path to
A.qcow2 or A'.qcow2; you can also delete B'.qcow2. But back to the
point I made earlier at [1]: if this is all you do, then 'virsh
snapshot-list' will still show 'snap1' as a snapshot that tries to refer
to A'.qcow2; since you just invalidated that with the copy, you'd need
to 'virsh snapshot-delete --metadata vm1 snap1' to get rid of the stale
snapshot (if you don't tweak step 1 to avoid creating that snapshot
metadata in the first place).
The NICE part about this whole sequence is that the backing file does
NOT have to be qcow2, and it is VERY efficient timewise, if you happen
to have an efficient way to do step 2. That is, I can go from a
multi-gigabyte raw file A.img to raw file B.img in less than a second,
assuming the guest isn't doing much I/O in the meantime, when scripting
all these steps together, and without any guest downtime.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
11:07 a.m.
On Tue, Oct 07, 2014 at 05:35:00PM -0600, Eric Blake wrote:
On 09/25/2014 08:26 AM, Kashyap Chamarthy wrote:
> This notes is based on an IRC conversation with Eric Blake, to have
> efficient non-shared storage live migration. Thought I'd post my notes
> here before I forget. Please review and spot if there are any
> inaccuracies.
>
> Procedure
> ---------
>
> (1) Starting from disk A, create a snapshot A <- A':
>
> $ virsh snapshot-create-as \
> --domain f20vm snap1 snap1-desc \
> --diskspec hda,file=/export/vmimages/A'.qcow2 \
> --disk-only --atomic
If you are using this snapshot only for the side-effect of growing the
chain, you can add --no-metadata here instead of deleting the snapshot
later when it gets invalidated [1]. Of course, if you pass
--no-metadata, the snapshot name (snap1) and description (snap1-desc)
are no longer important.
Right, until proper cleaner revert to external snapshot mechanisms are
in place, I should make a habit of passing '--no-metadata' when creating
external snapshots for the above reason (as I usually do end up deleting
the related libvirt metadata as part of cleanup).
>
> (2) Background copy of A to B:
>
> $ virsh blockcopy \
> --domain vm1 vda /export/vmimages/B.qcow2 \
> --wait --verbose --shallow \
> --finish
This step is not quite right. You are asking for a shallow copy of the
current file for disk 'vda' (that is, A'.qcow2). But that is NOT the
same as the base A image.
Oh right, thanks for catching this mistake.
For this step, libvirt does not yet have an easy way to access the
contents of a backing chain of a live domain; you CAN use 'virsh
vol-*' commands to do a background copy from storage pools, but it may
be easier to just resort to normal file system tools:
cp /export/vmimages/A.qcow2 /export/vmimages/B.qcow2
Yeah, simple and less commands to type too.
or even rely on storage-array-specific commands to set up a trivial
clone with no real time overhead (for example, some iscsi storage arrays
allow efficient copy-on-write cloning of storage volumes by creating a
new name that shares the same original contents of A.qcow2 as its
starting point; and since we are about to delete A.qcow2 later on, we
never need any actual data copying).
>
> (3) Create an empty B' with backing file B:
>
> $ qemu-img create -f qcow2 -b B.qcow2 \
> -o backing_fmt=qcow2 B'.qcow2
>
> [or]
>
> $ virsh vol-create-as default B'.qcow2 1G \
> --format qcow2 \
> --backing-vol B.qcow2 --backing-vol-format qcow2
[side note - we should really teach libvirt to not REQUIRE a size when
creating an empty wrapper around an existing image]
Filed: https://bugzilla.redhat.com/show_bug.cgi?id=1150411
>
> (4) Do a shallow blockcopy of A' to B':
>
> $ virsh blockcopy \
> --domain vm1 vda /export/vmimages/B'.qcow2 \
> --wait --verbose --shallow \
> --finish
For this to work, you need to also use the --reuse-external flag
True, I self-corrected in my other response in this thread, but thanks
for noticing.
to take
advantage of the backing chain already recorded in B'.qcow2 (without the
flag, the command will complain that B'.qcow2 already exists if it is a
regular file; if it is a block device, it will just silently ignore the
contents of the block device and treat B'.qcow2 as though an absolute
path to A.qcow2 were its backing file).
>
> (5) Then live shallow commit of B:
>
> $ virsh blockcommit \
> --domain f20vm vda \
> --wait --verbose --shallow \
> --pivot --active --finish
> Block Commit: [100 %]
> Successfully pivoted
With steps 2 and 4 corrected, this indeed shortens the chain back down
to just B.qcow2. And once this happens, you no longer need the path to
A.qcow2 or A'.qcow2; you can also delete B'.qcow2. But back to the
point I made earlier at [1]: if this is all you do, then 'virsh
snapshot-list' will still show 'snap1' as a snapshot that tries to refer
to A'.qcow2; since you just invalidated that with the copy, you'd need
to 'virsh snapshot-delete --metadata vm1 snap1' to get rid of the stale
snapshot (if you don't tweak step 1 to avoid creating that snapshot
metadata in the first place).
Thanks for this reminder, I'll script this as part of my tests to ensure
it's not missed.
The NICE part about this whole sequence is that the backing file
does
NOT have to be qcow2, and it is VERY efficient timewise, if you happen
to have an efficient way to do step 2. That is, I can go from a
multi-gigabyte raw file A.img to raw file B.img in less than a second,
assuming the guest isn't doing much I/O in the meantime, when scripting
all these steps together, and without any guest downtime.
Thanks again, for your meticulous review.
--
/kashyap
3634
days inactive
3647
days old
5 comments
3 participants
participants (3)
-
Eric Blake -
Kashyap Chamarthy -
Stefan Hajnoczi