[libvirt PATCH 0/2] rST-ify and rewrite a couple of block layer-related docs
I've cleaned up these two wiki articles that I wrote a while ago and turned them into rST. https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit https://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-includin... Once this series merges, I'll make the above two wiki pages point to the kbase docs. Kashyap Chamarthy (2): docs: kbase: Add a doc on live full disk backup docs: kbase: Add a doc on merging disk image chains docs/kbase/index.rst | 6 + docs/kbase/live_full_disk_backup.rst | 111 +++++++++++++ docs/kbase/merging_disk_image_chains.rst | 197 +++++++++++++++++++++++ 3 files changed, 314 insertions(+) create mode 100644 docs/kbase/live_full_disk_backup.rst create mode 100644 docs/kbase/merging_disk_image_chains.rst -- 2.30.2
This is a rewrite of: https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit Once this commit merges, the above wiki should point to this kbase document. Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com> --- docs/kbase/index.rst | 3 + docs/kbase/live_full_disk_backup.rst | 111 +++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 docs/kbase/live_full_disk_backup.rst diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst index 532804fe05..5ee3b5fce8 100644 --- a/docs/kbase/index.rst +++ b/docs/kbase/index.rst @@ -12,6 +12,9 @@ Usage Explanation of how disk backing chain specification impacts libvirt's behaviour and basic troubleshooting steps of disk problems. +`Live full disk backup <live_full_disk_backup.html>`__ + A walkthrough of how to take effective live full disk backups. + `Virtio-FS <virtiofs.html>`__ Share a filesystem between the guest and the host diff --git a/docs/kbase/live_full_disk_backup.rst b/docs/kbase/live_full_disk_backup.rst new file mode 100644 index 0000000000..f4ef7f2cc7 --- /dev/null +++ b/docs/kbase/live_full_disk_backup.rst @@ -0,0 +1,111 @@ +Efficient live full disk backup +=============================== + +Overview +-------- + +Live full disk backups are preferred in many scenarios, *despite* their +space requirements. The following outlines an efficient method to do +that using libvirt's APIs. This method involves concepts: the notion of +`backing chains <https://libvirt.org/kbase/backing_chains.html>`_, +`QCOW2 overlays +<https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#disk-image-backing-chain-notation>`_, +and a special operation called "active block-commit", which allows +live-merging an overlay disk image into its backing file. + +At a high-level, this is the approach to take a live ful disk backup: +Assuming a guest with a single disk image, create a temporary live QCOW2 +overlay (commonly called as "external snapshot") to track the live guest +writes. Then backup the original disk image while the guest (live QEMU) +keeps writing to the temporary overlay. Finally, perform the "active +block-commit" opertion to live-merge the temporary overlay disk contents +into the original image — i.e. the backing file — and "pivot" the live +QEMU process to point to it. + +The below assumes you're using *at least* these versions (preferably +much newer): QEMU 2.1 and libvirt-1.2.9. + +Detailed procedure +------------------ + +1. Start with a guest with a single disk image, ``base.raw``, which is + where the live QEMU is pointing at, and recording the guest writes:: + + base.raw (live QEMU) + +2. List the current block device(s) in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + +3. Create the live "external disk snapshot" (or more correctly, "an + overlay"):: + + $ virsh snapshot-create-as --domain vm1 overlay1 \ + --diskspec vda,file=/var/lib/libvirt/images/overlay1.qcow2 \ + --disk-only + + The disk image chain looks as follows:: + + base.raw <-- overlay1.qcow2 (live QEMU) + +.. note:: + Above, if you have QEMU guest agent installed in your virtual + machine, use the ``--quiesce`` option with ``virsh + snapshot-create-as [...]`` to ensure you have a consistent disk + state. + + Optionally, you can also supply the ``--no-metadata`` option to + ``virsh snapshot-create-as`` to tell libvirt not track the snapshot + metadata. Otherwise, when you decide to merge snapshot overlays, + you have to explicitly clean the libvirt metadata using ``virsh + snapshot-delete vm1 --metadata [name|--current]``. + +4. Now, take a backup the orignal image, ``base.raw``, to a different + location using ``cp`` or ``rsync``:: + + $ cp /var/lib/libvirt/images/base.raw + /export/backups/copy1_base.raw + + # Or: + + $ rsync -avhW --progress /var/lib/libvirt/images/base.raw \ + /export/backups/copy1_base.raw + +5. Enumerate the current block device(s) in use, again. Notice that the + current disk image in use is the above-created overlay, + ``overlay1.qcow2``:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda vda,file=/var/lib/libvirt/images/overlay1.qcow2 + +6. Once the backup of the original image completes, now perform the + "active block-commit" to live-merge the contents of + ``overlay1.qcow2`` into ``base.raw`` *and* pivot the live QEMU back + to the original:: + + $ virsh blockcommit vm1 vda --active --verbose --pivot + +7. After the above operation completes, again list the current block + device(s) in use. And notice that the live QEMU is now writing to + the original base image:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + + +The final updated disk image "chain" will be a single consolidated +disk:: + + [base.raw] (live QEMU) + + +Now you can safely **discard the overlay image**, ``overlay1.qcow2`` — +it is no longer valid; and its contents are now fully merged into the +base image. -- 2.30.2
On 5/4/21 6:16 PM, Kashyap Chamarthy wrote:
This is a rewrite of:
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
Once this commit merges, the above wiki should point to this kbase document.
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com> --- docs/kbase/index.rst | 3 + docs/kbase/live_full_disk_backup.rst | 111 +++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 docs/kbase/live_full_disk_backup.rst
Alright, first let me say that's it's difficult to chose which out of libxml2 or rst2html gives less useful message on error. More on that below. You will need to register this file in a list so that meson builds HTML: diff --git i/docs/kbase/meson.build w/docs/kbase/meson.build index 7b4e7abbd3..51d4bc7b90 100644 --- i/docs/kbase/meson.build +++ w/docs/kbase/meson.build @@ -6,6 +6,7 @@ docs_kbase_files = [ 'index', 'kvm-realtime', 'launch_security_sev', + 'live_full_disk_backup', 'locking-lockd', 'locking', 'locking-sanlock',
diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst index 532804fe05..5ee3b5fce8 100644 --- a/docs/kbase/index.rst +++ b/docs/kbase/index.rst @@ -12,6 +12,9 @@ Usage Explanation of how disk backing chain specification impacts libvirt's behaviour and basic troubleshooting steps of disk problems.
+`Live full disk backup <live_full_disk_backup.html>`__ + A walkthrough of how to take effective live full disk backups. + `Virtio-FS <virtiofs.html>`__ Share a filesystem between the guest and the host
diff --git a/docs/kbase/live_full_disk_backup.rst b/docs/kbase/live_full_disk_backup.rst new file mode 100644 index 0000000000..f4ef7f2cc7 --- /dev/null +++ b/docs/kbase/live_full_disk_backup.rst @@ -0,0 +1,111 @@ +Efficient live full disk backup +===============================
AAand this is the place that rst2html identified as problematic. While the true problem is below [1]. Anyway, how about making this h1 type of header, and [2]. Also, you can generate table of contents here simply by putting: .. contents::
+ +Overview +--------
2: this type h2? I mean putting "==.." at the beginning of the file, and s/-/=/
+ +Live full disk backups are preferred in many scenarios, *despite* their +space requirements. The following outlines an efficient method to do +that using libvirt's APIs. This method involves concepts: the notion of +`backing chains <https://libvirt.org/kbase/backing_chains.html>`_, +`QCOW2 overlays +<https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#disk-image-backing-chain-notation>`_, +and a special operation called "active block-commit", which allows +live-merging an overlay disk image into its backing file. + +At a high-level, this is the approach to take a live ful disk backup: +Assuming a guest with a single disk image, create a temporary live QCOW2 +overlay (commonly called as "external snapshot") to track the live guest +writes. Then backup the original disk image while the guest (live QEMU) +keeps writing to the temporary overlay. Finally, perform the "active +block-commit" opertion to live-merge the temporary overlay disk contents +into the original image — i.e. the backing file — and "pivot" the live +QEMU process to point to it. + +The below assumes you're using *at least* these versions (preferably +much newer): QEMU 2.1 and libvirt-1.2.9. + +Detailed procedure +------------------ + +1. Start with a guest with a single disk image, ``base.raw``, which is
How about replacing "1. 2. 3. ..." with "#."? That way paragraphs will get numbered automatically and we won't need to renumber them when introducing a new one or shifting them around (unlikely, I know).
+ where the live QEMU is pointing at, and recording the guest writes:: + + base.raw (live QEMU) + +2. List the current block device(s) in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + +3. Create the live "external disk snapshot" (or more correctly, "an + overlay"):: + + $ virsh snapshot-create-as --domain vm1 overlay1 \ + --diskspec vda,file=/var/lib/libvirt/images/overlay1.qcow2 \ + --disk-only + + The disk image chain looks as follows:: + + base.raw <-- overlay1.qcow2 (live QEMU) + +.. note::
1: this ^^ this is the problem, because it splits this numbered list into two. I guess after I found out what the problem was I understand the error message a bit: ../docs/kbase/live_full_disk_backup.rst:3: (INFO/1) Enumerated list start value not ordinal-1: "4" (ordinal 4) Anyway, putting two spaces at the beginning of the line fixes the problem: " .. note::"
+ Above, if you have QEMU guest agent installed in your virtual + machine, use the ``--quiesce`` option with ``virsh + snapshot-create-as [...]`` to ensure you have a consistent disk + state. + + Optionally, you can also supply the ``--no-metadata`` option to + ``virsh snapshot-create-as`` to tell libvirt not track the snapshot + metadata. Otherwise, when you decide to merge snapshot overlays, + you have to explicitly clean the libvirt metadata using ``virsh + snapshot-delete vm1 --metadata [name|--current]``. + +4. Now, take a backup the orignal image, ``base.raw``, to a different + location using ``cp`` or ``rsync``:: + + $ cp /var/lib/libvirt/images/base.raw + /export/backups/copy1_base.raw + + # Or: + + $ rsync -avhW --progress /var/lib/libvirt/images/base.raw \ + /export/backups/copy1_base.raw + +5. Enumerate the current block device(s) in use, again. Notice that the + current disk image in use is the above-created overlay, + ``overlay1.qcow2``:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda vda,file=/var/lib/libvirt/images/overlay1.qcow2 + +6. Once the backup of the original image completes, now perform the + "active block-commit" to live-merge the contents of + ``overlay1.qcow2`` into ``base.raw`` *and* pivot the live QEMU back + to the original:: + + $ virsh blockcommit vm1 vda --active --verbose --pivot + +7. After the above operation completes, again list the current block + device(s) in use. And notice that the live QEMU is now writing to + the original base image:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + + +The final updated disk image "chain" will be a single consolidated +disk:: + + [base.raw] (live QEMU) + + +Now you can safely **discard the overlay image**, ``overlay1.qcow2`` — +it is no longer valid; and its contents are now fully merged into the +base image.
Michal
On Fri, May 07, 2021 at 12:29:26PM +0200, Michal Prívozník wrote:
On 5/4/21 6:16 PM, Kashyap Chamarthy wrote:
[...]
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com> --- docs/kbase/index.rst | 3 + docs/kbase/live_full_disk_backup.rst | 111 +++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 docs/kbase/live_full_disk_backup.rst
Alright, first let me say that's it's difficult to chose which out of libxml2 or rst2html gives less useful message on error. More on that below.
:-) I saw the awful error locally after you mentioned "--strict" on IRC.
You will need to register this file in a list so that meson builds HTML:
Ah, will do.
diff --git i/docs/kbase/meson.build w/docs/kbase/meson.build index 7b4e7abbd3..51d4bc7b90 100644 --- i/docs/kbase/meson.build +++ w/docs/kbase/meson.build @@ -6,6 +6,7 @@ docs_kbase_files = [ 'index', 'kvm-realtime', 'launch_security_sev', + 'live_full_disk_backup', 'locking-lockd', 'locking', 'locking-sanlock',
[...]
index 0000000000..f4ef7f2cc7 --- /dev/null +++ b/docs/kbase/live_full_disk_backup.rst @@ -0,0 +1,111 @@ +Efficient live full disk backup +===============================
AAand this is the place that rst2html identified as problematic. While the true problem is below [1]. Anyway, how about making this h1 type of header, and [2].
Ah, strange. I should've followed existing pattern. I just wrote the doc based on the pattern I just go used to: h1 content ========== h2 content ---------- h3 content ~~~~~~~~~~ I'll respin the docs to match the existing convention; it's also consistent.
Also, you can generate table of contents here simply by putting:
.. contents::
Yes; will fix in next iteration.
+ +Overview +--------
2: this type h2? I mean putting "==.." at the beginning of the file, and s/-/=/
Yeah, it's supposed to be. Maybe it's only in the Sphinx-based world. As noted above, I'll re-do it match existing convention. [...]
+Detailed procedure +------------------ + +1. Start with a guest with a single disk image, ``base.raw``, which is
How about replacing "1. 2. 3. ..." with "#."? That way paragraphs will get numbered automatically and we won't need to renumber them when introducing a new one or shifting them around (unlikely, I know).
I wondered about it too. I'll change it to the generic "#.". [...]
+ The disk image chain looks as follows:: + + base.raw <-- overlay1.qcow2 (live QEMU) + +.. note::
1: this ^^ this is the problem, because it splits this numbered list into two. I guess after I found out what the problem was I understand the error message a bit:
../docs/kbase/live_full_disk_backup.rst:3: (INFO/1) Enumerated list start value not ordinal-1: "4" (ordinal 4)
That's an unreadable error from `rst2html5`. Thanks for digging! I didn't hit it as I did not run `rst2html5` with `--strict`.
Anyway, putting two spaces at the beginning of the line fixes the problem: " .. note::"
Yep; will fix in next version. Thanks for the review. [...] -- /kashyap
On Tue, May 04, 2021 at 18:16:12 +0200, Kashyap Chamarthy wrote:
This is a rewrite of:
https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit
Once this commit merges, the above wiki should point to this kbase document.
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com> --- docs/kbase/index.rst | 3 + docs/kbase/live_full_disk_backup.rst | 111 +++++++++++++++++++++++++++ 2 files changed, 114 insertions(+) create mode 100644 docs/kbase/live_full_disk_backup.rst
[...]
diff --git a/docs/kbase/live_full_disk_backup.rst b/docs/kbase/live_full_disk_backup.rst new file mode 100644 index 0000000000..f4ef7f2cc7 --- /dev/null +++ b/docs/kbase/live_full_disk_backup.rst @@ -0,0 +1,111 @@ +Efficient live full disk backup +===============================
It feels nowadays a bit weird to add a document which outlines everything you can achieve with the virDomainBackupBegin API without any of the extra steps. I'd suggest you note that this predates the backup APIs or link to them somehow.
+ +Overview +-------- + +Live full disk backups are preferred in many scenarios, *despite* their +space requirements. The following outlines an efficient method to do +that using libvirt's APIs. This method involves concepts: the notion of +`backing chains <https://libvirt.org/kbase/backing_chains.html>`_, +`QCOW2 overlays +<https://qemu.readthedocs.io/en/latest/interop/live-block-operations.html#disk-image-backing-chain-notation>`_, +and a special operation called "active block-commit", which allows +live-merging an overlay disk image into its backing file.
On Fri, May 07, 2021 at 01:22:13PM +0200, Peter Krempa wrote:
On Tue, May 04, 2021 at 18:16:12 +0200, Kashyap Chamarthy wrote:
[...]
diff --git a/docs/kbase/live_full_disk_backup.rst b/docs/kbase/live_full_disk_backup.rst new file mode 100644 index 0000000000..f4ef7f2cc7 --- /dev/null +++ b/docs/kbase/live_full_disk_backup.rst @@ -0,0 +1,111 @@ +Efficient live full disk backup +===============================
It feels nowadays a bit weird to add a document which outlines everything you can achieve with the virDomainBackupBegin API without any of the extra steps.
Right, I wrote them when virDomainBackupBegin() and friends were not there yet :-). I see that the newer API is mentioned in the "internals" doc here, which outlines how libvirt implements incremental backups and some "pseudo algorithms" on external snapshots: https://libvirt.org/kbase/incrementalbackupinternals.html
I'd suggest you note that this predates the backup APIs or link to them somehow.
Yep, I'll mention and link to it. But also I think a separate, use-case drive doc for virDomainBackupBegin() and friends can also be very handy. I'll make that a TODO after I clean-up these two first. [...] -- /kashyap
This is a rewrite of: https://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-includin... Once this commit merges, the above wiki should point to this kbase document. Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com> --- docs/kbase/index.rst | 3 + docs/kbase/merging_disk_image_chains.rst | 197 +++++++++++++++++++++++ 2 files changed, 200 insertions(+) create mode 100644 docs/kbase/merging_disk_image_chains.rst diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst index 5ee3b5fce8..8e8d959e92 100644 --- a/docs/kbase/index.rst +++ b/docs/kbase/index.rst @@ -15,6 +15,9 @@ Usage `Live full disk backup <live_full_disk_backup.html>`__ A walkthrough of how to take effective live full disk backups. +`Merging disk image chains <merging_disk_image_chains.html>`__ + Ways to reduce or consolidate disk image chains. + `Virtio-FS <virtiofs.html>`__ Share a filesystem between the guest and the host diff --git a/docs/kbase/merging_disk_image_chains.rst b/docs/kbase/merging_disk_image_chains.rst new file mode 100644 index 0000000000..07732c4e0d --- /dev/null +++ b/docs/kbase/merging_disk_image_chains.rst @@ -0,0 +1,197 @@ +Merging disk image image chains +=============================== + +Context +------- + +Sometimes a `disk image chain +<https://libvirt.org/kbase/backing_chains.html>`_ can get long and +cumbersome. For the purpose of illustration, consider this smaller disk +image chain:: + + base.raw <-- a.qcow2 <-- b.qcow2 <-- c.qcow2 (live QEMU) + +You may want to reduce the backing chain length, or consolidate *all* +the disk images in the chain into a single image. But you want to +accomplish this *without* incurring guest down time. Here's how to go +about it. + +The same principles used in the `live full disk backup +<https://libvirt.org/kbase/live_full_disk_backup.html>` document are +used here too. + +Reducing the disk image chain length +------------------------------------ + +Starting the earlier image chain:: + + base.raw <-- a.qcow2 <-- b.qcow2 <-- c.qcow2 (live QEMU) + +Reduce the length of the chain by two images, with the resulting chain +being:: + + base.raw <-- c.qcow2 (live QEMU) + +Where the ``base.raw`` contains the contents of ``a.qcow2`` and +``b.qcow2``. + +1. Start by listing the current active disk image in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + +2. Create the image chain by creating three QCOW2 overlays (or "external + snapshots") on top of each other, while adding some differentiating + content in each image:: + + $ virsh snapshot-create-as --domain vm1 snap1 \ + --diskspec vda,file=/var/lib/libvirt/images/a.qcow2 \ + --disk-only --no-metadata + + # <Add a file in the guest> + + $ virsh snapshot-create-as --domain vm1 snap2 \ + --diskspec vda,file=/var/lib/libvirt/images/b.qcow2 \ + --disk-only --no-metadata + + # <Add another file in the guest> + + $ virsh snapshot-create-as --domain vm1 snap3 \ + --diskspec vda,file=/var/lib/libvirt/images/c.qcow2 \ + --disk-only --no-metadata + +3. Enumerate the backing file chain (here the ``force-share`` option + simply allows ``qemu-img`` to safely query the disk image status + while it is active) :: + + $ qemu-img info --force-share --backing-chain /var/lib/libvirt/images/cur.qcow2 + [...] + +4. Again, list the current active disk image in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/c.qcow2 + +5. Perform the "block-commit" by specify the "base" and "top" images, + i.e. merge the contents of ``b.qcow2`` *and* ``a.qcow2`` into + ``base.raw``, *and* :: + + $ virsh blockcommit vm1 vda \ + --base=/var/lib/libvirt/images/base.raw + --top=/var/lib/libvirt/images/b.qcow2 + +A note on accessing 'base' and 'top' images +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Specifying file paths, as above, make sense when your disks are in the +local filesystem. However, when using more complicated setups such as +network block device (NBD) disks, there are no file paths. Hhere is +where accessing the disk by its index number comes into picture. + +As an example, the below is the ``<disk>`` element of the guest XML for +with the original disk image chain of four images:: + + ... + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2'/> + <source file='/var/lib/libvirt/images/c.qcow2' index='4'/> + <backingStore type='file' index='3'> + <format type='qcow2'/> + <source file='/var/lib/libvirt/images/b.qcow2'/> + <backingStore type='file' index='2'> + <format type='qcow2'/> + <source file='/var/lib/libvirt/images/a.qcow2'/> + <backingStore type='file' index='1'> + <format type='raw'/> + <source file='/var/lib/libvirt/images/base.raw'/> + <backingStore/> + </backingStore> + </backingStore> + </backingStore> + <target dev='vda' bus='virtio'/> + <alias name='virtio-disk0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> + </disk> + ... + +And we can merge the images ``a.qcow2`` and ``b.qcow2`` into +``base.qcow2`` using the index numbers 1 (for ``base.qcow2``) and :: + + $> virsh blockcommit vm1 vda --base='vda[1]' --top='vda[3]' + +Note that the shell quoting is necessary here, since unquoted ``[1]`` +would do an unexpected shell "globbing" (i.e. file name expansion) if +you have a file '1' in the current directory + +Accessing the disk images via their index numbers is more useful when +you're using blockCommit() API programmatically. + + +Consolidating the entire chain into a single image +-------------------------------------------------- + +Again, starting the original image chain:: + + base.raw <-- a.qcow2 <-- b.qcow2 <-- c.qcow2 (live QEMU) + +Reduce the length of the chain by two images, with the resulting chain +being:: + + base.raw (live QEMU) + +Where the ``base.raw`` contains the contents of ``a.qcow2``, ``b.qcow2`` +and ``c.qcow2``; *and* the live QEMU is piovoted to point to the +``base.raw``. + + +1. Use the same procedure discussed earlier to create the disk image + chain. + + +2. Now perform the "active block-commit" operation:: + + $ virsh blockcommit vm1 vda --verbose --pivot --active + Block Commit: [100 %] + Successfully pivoted + + Notes: + + - ``--active``: It performs a two-stage operation: first, the contents + from top images (``a.qcow2``, ``b.qcow2``, and ``c.qcow2``) are + committed into the base image; and in the second stage, the the + "block-commit" operation remains awake to synchronize any further + changes from top images into base. Here the user can take two + actions: cancel the job, or pivot the job, i.e. adjust the base + image as the current active image. + + - ``--pivot``: Once data is committed from sn1, sn2 and current into + base, it pivots the live QEMU to use base as the active image. + + - ``--verbose``: It shows the progress of block operation. + + +3. Again, check the current active block device in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + + +4. Enumerate the backing file chain:: + + $ qemu-img info --backing-chain /var/lib/libvirt/images/base.raw + [...] + + And the final resulting disk image "chain" will be a single, + consolidated disk image:: + + [base] (live QEMU) + +It is worth bearing in mind that once the above pivot completes, *all* +three overlay files — ``a.qcow2``, ``b.qcow2``, and ``c.qcow2`` — are no +longer valid, and can be safely discarded. -- 2.30.2
On 5/4/21 6:16 PM, Kashyap Chamarthy wrote:
This is a rewrite of:
https://wiki.libvirt.org/page/Live-merge-an-entire-disk-image-chain-includin...
Once this commit merges, the above wiki should point to this kbase document.
Signed-off-by: Kashyap Chamarthy <kchamart@redhat.com> --- docs/kbase/index.rst | 3 + docs/kbase/merging_disk_image_chains.rst | 197 +++++++++++++++++++++++ 2 files changed, 200 insertions(+) create mode 100644 docs/kbase/merging_disk_image_chains.rst
Basically same comments here. meson.build, TOC, promote titles to higher rank, replace 1. 2. 3. .. with #. and ...
diff --git a/docs/kbase/index.rst b/docs/kbase/index.rst index 5ee3b5fce8..8e8d959e92 100644 --- a/docs/kbase/index.rst +++ b/docs/kbase/index.rst @@ -15,6 +15,9 @@ Usage `Live full disk backup <live_full_disk_backup.html>`__ A walkthrough of how to take effective live full disk backups.
+`Merging disk image chains <merging_disk_image_chains.html>`__ + Ways to reduce or consolidate disk image chains. + `Virtio-FS <virtiofs.html>`__ Share a filesystem between the guest and the host
diff --git a/docs/kbase/merging_disk_image_chains.rst b/docs/kbase/merging_disk_image_chains.rst new file mode 100644 index 0000000000..07732c4e0d --- /dev/null +++ b/docs/kbase/merging_disk_image_chains.rst @@ -0,0 +1,197 @@ +Merging disk image image chains +=============================== + +Context +------- + +Sometimes a `disk image chain +<https://libvirt.org/kbase/backing_chains.html>`_ can get long and +cumbersome. For the purpose of illustration, consider this smaller disk +image chain:: + + base.raw <-- a.qcow2 <-- b.qcow2 <-- c.qcow2 (live QEMU) + +You may want to reduce the backing chain length, or consolidate *all* +the disk images in the chain into a single image. But you want to +accomplish this *without* incurring guest down time. Here's how to go +about it. + +The same principles used in the `live full disk backup +<https://libvirt.org/kbase/live_full_disk_backup.html>` document are +used here too. + +Reducing the disk image chain length +------------------------------------ + +Starting the earlier image chain:: + + base.raw <-- a.qcow2 <-- b.qcow2 <-- c.qcow2 (live QEMU) + +Reduce the length of the chain by two images, with the resulting chain +being:: + + base.raw <-- c.qcow2 (live QEMU) + +Where the ``base.raw`` contains the contents of ``a.qcow2`` and +``b.qcow2``. + +1. Start by listing the current active disk image in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + +2. Create the image chain by creating three QCOW2 overlays (or "external + snapshots") on top of each other, while adding some differentiating + content in each image:: + + $ virsh snapshot-create-as --domain vm1 snap1 \ + --diskspec vda,file=/var/lib/libvirt/images/a.qcow2 \ + --disk-only --no-metadata + + # <Add a file in the guest> + + $ virsh snapshot-create-as --domain vm1 snap2 \ + --diskspec vda,file=/var/lib/libvirt/images/b.qcow2 \ + --disk-only --no-metadata + + # <Add another file in the guest> + + $ virsh snapshot-create-as --domain vm1 snap3 \ + --diskspec vda,file=/var/lib/libvirt/images/c.qcow2 \ + --disk-only --no-metadata + +3. Enumerate the backing file chain (here the ``force-share`` option + simply allows ``qemu-img`` to safely query the disk image status + while it is active) :: + + $ qemu-img info --force-share --backing-chain /var/lib/libvirt/images/cur.qcow2 + [...] + +4. Again, list the current active disk image in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/c.qcow2 + +5. Perform the "block-commit" by specify the "base" and "top" images, + i.e. merge the contents of ``b.qcow2`` *and* ``a.qcow2`` into + ``base.raw``, *and* :: + + $ virsh blockcommit vm1 vda \ + --base=/var/lib/libvirt/images/base.raw + --top=/var/lib/libvirt/images/b.qcow2 + +A note on accessing 'base' and 'top' images +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Specifying file paths, as above, make sense when your disks are in the +local filesystem. However, when using more complicated setups such as +network block device (NBD) disks, there are no file paths. Hhere is +where accessing the disk by its index number comes into picture. + +As an example, the below is the ``<disk>`` element of the guest XML for +with the original disk image chain of four images:: + + ... + <disk type='file' device='disk'> + <driver name='qemu' type='qcow2'/> + <source file='/var/lib/libvirt/images/c.qcow2' index='4'/> + <backingStore type='file' index='3'> + <format type='qcow2'/> + <source file='/var/lib/libvirt/images/b.qcow2'/> + <backingStore type='file' index='2'> + <format type='qcow2'/> + <source file='/var/lib/libvirt/images/a.qcow2'/> + <backingStore type='file' index='1'> + <format type='raw'/> + <source file='/var/lib/libvirt/images/base.raw'/> + <backingStore/> + </backingStore> + </backingStore> + </backingStore> + <target dev='vda' bus='virtio'/> + <alias name='virtio-disk0'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> + </disk> + ... + +And we can merge the images ``a.qcow2`` and ``b.qcow2`` into +``base.qcow2`` using the index numbers 1 (for ``base.qcow2``) and :: + + $> virsh blockcommit vm1 vda --base='vda[1]' --top='vda[3]' + +Note that the shell quoting is necessary here, since unquoted ``[1]`` +would do an unexpected shell "globbing" (i.e. file name expansion) if +you have a file '1' in the current directory + +Accessing the disk images via their index numbers is more useful when +you're using blockCommit() API programmatically. + + +Consolidating the entire chain into a single image +-------------------------------------------------- + +Again, starting the original image chain:: + + base.raw <-- a.qcow2 <-- b.qcow2 <-- c.qcow2 (live QEMU) + +Reduce the length of the chain by two images, with the resulting chain +being:: + + base.raw (live QEMU) + +Where the ``base.raw`` contains the contents of ``a.qcow2``, ``b.qcow2`` +and ``c.qcow2``; *and* the live QEMU is piovoted to point to the +``base.raw``. + + +1. Use the same procedure discussed earlier to create the disk image + chain. + + +2. Now perform the "active block-commit" operation:: + + $ virsh blockcommit vm1 vda --verbose --pivot --active + Block Commit: [100 %] + Successfully pivoted + + Notes:
.. starting from here until
+ + - ``--active``: It performs a two-stage operation: first, the contents + from top images (``a.qcow2``, ``b.qcow2``, and ``c.qcow2``) are + committed into the base image; and in the second stage, the the + "block-commit" operation remains awake to synchronize any further + changes from top images into base. Here the user can take two + actions: cancel the job, or pivot the job, i.e. adjust the base + image as the current active image. + + - ``--pivot``: Once data is committed from sn1, sn2 and current into + base, it pivots the live QEMU to use base as the active image. + + - ``--verbose``: It shows the progress of block operation.
.. here prepend lines with one space more so that the paragraph is aligned. Once again, this misalignment makes rst2html unhappy.
+ + +3. Again, check the current active block device in use:: + + $ virsh domblklist vm1 + Target Source + ------------------------------------------------ + vda /var/lib/libvirt/images/base.raw + + +4. Enumerate the backing file chain:: + + $ qemu-img info --backing-chain /var/lib/libvirt/images/base.raw + [...] + + And the final resulting disk image "chain" will be a single, + consolidated disk image:: + + [base] (live QEMU) + +It is worth bearing in mind that once the above pivot completes, *all* +three overlay files — ``a.qcow2``, ``b.qcow2``, and ``c.qcow2`` — are no +longer valid, and can be safely discarded.
Michal
On Tue, May 04, 2021 at 06:16:11PM +0200, Kashyap Chamarthy wrote: [...]
Kashyap Chamarthy (2): docs: kbase: Add a doc on live full disk backup docs: kbase: Add a doc on merging disk image chains
Here's the HTML rendering (converted with `rst2html`): https://kashyapc.fedorapeople.org/libvirt/kbase/live_full_disk_backup.html https://kashyapc.fedorapeople.org/libvirt/kbase/merging_disk_image_chains.ht...
docs/kbase/index.rst | 6 + docs/kbase/live_full_disk_backup.rst | 111 +++++++++++++ docs/kbase/merging_disk_image_chains.rst | 197 +++++++++++++++++++++++ 3 files changed, 314 insertions(+) create mode 100644 docs/kbase/live_full_disk_backup.rst create mode 100644 docs/kbase/merging_disk_image_chains.rst
-- 2.30.2
-- /kashyap
participants (3)
-
Kashyap Chamarthy -
Michal Prívozník -
Peter Krempa