On 6/24/20 9:07 AM, Peter Krempa wrote: Reviewed-by: Eric Blake <eblake(a)redhat.com&gt; Maybe it's also worth a paragraph mentioning how diagramming the chains we typically use <- for 'Backed by', as in: Base <- Top -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org

Emphaisze what needs to happen and also that creating a snapshot doesn't create the appropriate bitmaps. Also mention that granularity is kept. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/incrementalbackupinternals.rst | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/kbase/incrementalbackupinternals.rst b/docs/kbase/incrementalbackupinternals.rst index eada0d2935..dbef1e96f3 100644 --- a/docs/kbase/incrementalbackupinternals.rst +++ b/docs/kbase/incrementalbackupinternals.rst @@ -109,17 +109,18 @@ as ``base image``. The topmost overlay is the image which is being written to by the VM and is also described as the ``active`` layer or image. -Handling of bitmaps -------------------- - -Creating an external snapshot involves adding a new layer to the backing chain -on top of the previous chain. In this step there are no new bitmaps created by -default, which would mean that backups become impossible after this step. - -To prevent this from happening we need to re-create the active bitmaps in the -new top/active layer of the backing chain which allows us to continue tracking -the changes with same granularity as before and also allows libvirt to stitch -together all the corresponding bitmaps to do a backup across snapshots. +Handling of bitmaps during snapshots +------------------------------------ + +Creating an external snapshot involves adding a overlay on top of the previously +active image. Libvirt requires that all ``block-dirty-bitmaps`` which correspond +to the checkpoint must be created in the new overlay before any write from the +guest reaches the overlay to continue tracking which blocks are dirtied. + +Since there are no new bitmaps created by ``qemu`` or ``qemu-img`` by default +when creating an overlay, we need to re-create the appropriate (see below.) +bitmaps in the new overlay based on the previously active bitmaps in the active +image. The new bitmaps are created with the same granularity. After taking a snapshot of the ``vda`` disk from the example above placed into ``vda-2.qcow2`` the following topology will be created: -- 2.26.2

Add a section that outlines usage of tools to handle bitmaps and introduce terms corresponding to the output of qemu-img to be used in further sections. With this we can simplify the section about checking bitmap health as we don't have to explain the qemu-img output but can refer to the newly defined terms. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/incrementalbackupinternals.rst | 58 ++++++++++++++++------- 1 file changed, 40 insertions(+), 18 deletions(-) diff --git a/docs/kbase/incrementalbackupinternals.rst b/docs/kbase/incrementalbackupinternals.rst index dbef1e96f3..adc415e282 100644 --- a/docs/kbase/incrementalbackupinternals.rst +++ b/docs/kbase/incrementalbackupinternals.rst @@ -139,29 +139,17 @@ After taking a snapshot of the ``vda`` disk from the example above placed into | | | | +----------------+ +----------------+ -Checking bitmap health ----------------------- +Manipulating bitmaps in shell +----------------------------- -QEMU optimizes disk writes by only updating the bitmaps in certain cases. This -also can cause problems in cases when e.g. QEMU crashes. +**NOTE:** Any of the examples expect that the full image chain isn't used by any +running VM at the time. -For a chain of corresponding bitmaps in a backing chain to be considered valid -and eligible for use with ``virDomainBackupBegin`` it must conform to the -following rules: - -1) Top image must contain the bitmap -2) If any of the backing images in the chain contain the bitmap too, all - contiguous images must have the bitmap (no gaps) -3) all of the above bitmaps must be marked as active - (``auto`` flag in ``qemu-img`` output, ``recording`` in qemu) -4) none of the above bitmaps can be inconsistent - (``in-use`` flag in ``qemu-img`` provided that it's not used on image which - is currently in use by a qemu instance, or ``inconsistent`` in qemu) +``qemu-img info`` command reports information about dirty bitmaps in an image: :: - # check that image has bitmaps - $ qemu-img info vda-1.qcow2 + $ qemu-img info -f qcow2 vda-1.qcow2 image: vda-1.qcow2 file format: qcow2 virtual size: 100 MiB (104857600 bytes) @@ -186,6 +174,40 @@ following rules: refcount bits: 16 corrupt: false +The ``flags`` have following meanings: + +``auto`` - **recording** + + The bitmap is automatically activated when the image is opened for wrinting + and thus it's actively recording writes. + +``in-use`` - **inconsistent** + + The bitmap was not properly saved when the qemu process was shut down last + time thus didn't conistently record all the changed sectors. + +It's reccomended to use ``--output=json`` parameter to work with a machine +readable output rather than trying to process the human readable output by +scripts. For processing JSON in shell the ``jq`` tool can be used. + +Checking bitmap health +---------------------- + +QEMU optimizes disk writes by only updating the bitmaps in certain cases. This +also can cause problems in cases when e.g. QEMU crashes. + +For a chain of corresponding bitmaps in a backing chain images to be considered +valid and eligible for use for an incremental backup with +``virDomainBackupBegin`` the bitmaps intended to be used must conform to the +following rules: + +1) active/topmost image must contain the bitmap +2) if bitmap with the same name is contained in one of the backing images it + must be a contiguougs subchain starting from the topmost image which contains + the bitmaps (no gaps) +3) all of the above bitmaps must be marked as **recording** +4) all of the above bitmaps must not be **inconsistent** + (See also the ``qemuBlockBitmapChainIsValid`` helper method in ``src/qemu/qemu_block.c``) -- 2.26.2

Define what users should look for when wanting to manipulate bitmaps themselves. Later on a patch will turn the bash algorithms into pseudocode for simplicity. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/incrementalbackupinternals.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/kbase/incrementalbackupinternals.rst b/docs/kbase/incrementalbackupinternals.rst index adc415e282..c8a0a66baa 100644 --- a/docs/kbase/incrementalbackupinternals.rst +++ b/docs/kbase/incrementalbackupinternals.rst @@ -190,6 +190,20 @@ It's reccomended to use ``--output=json`` parameter to work with a machine readable output rather than trying to process the human readable output by scripts. For processing JSON in shell the ``jq`` tool can be used. +``qemu-img bitmap`` command allows modification of block-dirty-bitmaps of an +image. It supports the following operations relevant to this document (see man +page for full list of operations): + +``--add NAME`` + Creates a new bitmap named ``NAME``. Optionally ``-g`` can be used to + specify granularity. + +``--remove NAME`` + Deletes bitmap ``NAME``. + +``--merge SRCBITMAP -b SRCFILE -F SRCFILEFMT DSTBITMAP`` + Merges bitmap ``SRCBITMAP`` from ``SRCFILE`` into ``DSTBITMAP``. + Checking bitmap health ---------------------- -- 2.26.2

Simplify the docs and reduce maintenance burden by just describing the algorithm by a pseudo-language. Users are encouraged to use libvirt anyways and projects such as oVirt which do some management of storage themselves are unlikely to use bash anyways. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/incrementalbackupinternals.rst | 44 ++++++----------------- 1 file changed, 11 insertions(+), 33 deletions(-) diff --git a/docs/kbase/incrementalbackupinternals.rst b/docs/kbase/incrementalbackupinternals.rst index c8a0a66baa..9a96ef6df3 100644 --- a/docs/kbase/incrementalbackupinternals.rst +++ b/docs/kbase/incrementalbackupinternals.rst @@ -228,42 +228,20 @@ following rules: Creating external snapshots manually -------------------------------------- -To create the same topology outside of libvirt (e.g when doing snapshots offline) -a new ``qemu-img`` which supports the ``bitmap`` subcommand is recommended. The -following algorithm then ensures that the new image after snapshot will work -with backups (note that ``jq`` is a JSON processor): +To create the same topology outside of libvirt (e.g when doing snapshots +offline) the following pseudo-algorithm ensures that the new image after +snapshot will work with backups. ``OVERLAY`` corresponds to the new overlay +image, ``ACTIVE`` corresponds to the topmost image of the active chain prior to +the snapshot. :: - #!/bin/bash + create image OVERLAY on top of ACTIVE - # arguments - SNAP_IMG="vda-2.qcow2" - BACKING_IMG="vda-1.qcow2" + for each BITMAP in ACTIVE: + let GRANULARITY = granularity of BITMAP in ACTIVE - # constants - snapshots and bitmaps work only with qcow2 - SNAP_FMT="qcow2" - BACKING_IMG_FMT="qcow2" + if BITMAP isn't RECORDING or is INCONSISTENT: + continue - # create snapshot overlay - qemu-img create -f "$SNAP_FMT" -F "$BACKING_IMG_FMT" -b "$BACKING_IMG" "$SNAP_IMG" - - BACKING_IMG_INFO=$(qemu-img info --output=json -f "$BACKING_IMG_FMT" "$BACKING_IMG") - BACKING_BITMAPS=$(jq '."format-specific".data.bitmaps' <<< "$BACKING_IMG_INFO") - - if [ "x$BACKING_BITMAPS" = "xnull" ]; then - exit 0 - fi - - for BACKING_BITMAP_ in $(jq -c '.[]' <<< "$BACKING_BITMAPS"); do - BITMAP_FLAGS=$(jq -c -r '.flags[]' <<< "$BACKING_BITMAP_") - BITMAP_NAME=$(jq -r '.name' <<< "$BACKING_BITMAP_") - - if grep 'in-use' <<< "$BITMAP_FLAGS" || - grep -v 'auto' <<< "$BITMAP_FLAGS"; then - continue - fi - - qemu-img bitmap -f "$SNAP_FMT" "$SNAP_IMG" --add "$BITMAP_NAME" - - done + create RECORDING bitmap named BITMAP in OVERLAY with GRANULARITY -- 2.26.2

oVirt does merge images out of libvirt in some cases. Add docs outlining how it's done from a high level. Signed-off-by: Peter Krempa <pkrempa(a)redhat.com&gt; --- docs/kbase/incrementalbackupinternals.rst | 37 +++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/docs/kbase/incrementalbackupinternals.rst b/docs/kbase/incrementalbackupinternals.rst index 9a96ef6df3..e50bf52e89 100644 --- a/docs/kbase/incrementalbackupinternals.rst +++ b/docs/kbase/incrementalbackupinternals.rst @@ -245,3 +245,40 @@ the snapshot. continue create RECORDING bitmap named BITMAP in OVERLAY with GRANULARITY + +Commiting external snapshots manually +------------------------------------- + +``block commit`` refers to an operation where data from a subchain of the +backing chain is merged down into the backing image of the subchain removing all +images in the subchain . + +``COMMIT_TOP`` refers to the top of the subchain to merge into ``COMMIT_BASE`` +(which stays in the new chain). + +It's strongly advised to use ``virDomainBlockCommit`` API in libvirt directly if +possible. Inactive VMs can be started with ``VIR_DOMAIN_START_PAUSED`` flag +(``virsh start --paused``) to prevent OS from running. + +Otherwise the following pseudo-algorithm can be used: + +Note: A ``valid`` bitmap chain is a set of images containing bitmaps which +conform to the rules about valid bitmaps mentioned above. + +:: + + commit data from COMMIT_TOP to COMMIT_BASE + + let BITMAPS = valid bitmap chains in COMMIT_TOP + + for each BITMAP in BITMAPS + let GRANULARITY = granularity of BITMAP in ACTIVE + + if BITMAP is not present in COMMIT_BASE: + create RECORDING bitmap named BITMAP in COMMIT_BASE with GRANULARITY + + for each IMAGE between COMMIT_TOP(inclusive) and COMMIT_BASE(exclusive): + if BITMAP is not present in IMAGE: + break + + merge BITMAP in IMAGE into BITMAP in COMMIT_BASE -- 2.26.2

On Wed, Jun 24, 2020 at 09:35:22 -0500, Eric Blake wrote: oVirt doesn't use block pull for now and even libvirt doesn't implement bitmap handling for the pull job for now. I'll add information here once I do the implementation.