[PATCH 00/32] qemu: backup: Rewrite/simplify internals and document semantics
Simplify how we approach bitmap handling to increase robustness and understanding of the code. Additionally document intricacies of how we approach handling of bitmaps so that in cases when users decide to do snapshots outside of libvirt they can re-create the bitmaps so that they will still work with backups (requires new qemu-img). I'll also follow up documenting implicationsof blockjobs on bitmaps later. To fetch this including a commit which allows testing: git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-rewrite-and-document This doesn't require any special qemu patches. Note that this shares many patches from my previous posting which tried to make use of 'block-dirty-bitmap-populate', but keeps explicit bitmap tracking for now. We'll be able to revisit use of the blockjob if the qemu work will continue somehow. Peter Krempa (32): qemu: backup: Split up code traversing checkpoint list looking for bitmaps qemu: backup: Fix backup of disk skipped in an intermediate checkpoint conf: backup: Store incremental backup checkpoint name per-disk qemu: backup: Move fetching of checkpoint list for incremental backup qemublocktest: Add 'empty' test case for bitmaps qemublocktest: Add 'empty' case for incremental backup test qemublocktest: Add 'empty' case for checkpoint deletion qemublocktest: Add 'empty' case for blockcopy bitmap handling test qemublocktest: Add 'empty' case for checkpoint bitmap handling qemublocktest: Disable testcases for all bitmap handling qemublocktest: Delete 'synthetic' bitmap test cases qemublocktest: Extract printing of nodename list qemu: checkpoint: Don't chain bitmaps for checkpoints qemublocktest: Replace 'basic' bitmap detection test case data qemublocktest: Replace 'snapshots' bitmap detection test case data qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots' cases qemublocktest: Add new 'synthetic' bitmap detection and validation test case qemu: checkpoint: Don't merge checkpoints during deletion qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to TEST_CHECKPOINT_DELETE qemublocktest: Re-introduce testing of checkpoint deletion qemu: block: Add universal helper for merging dirty bitmaps for all scenarios qemu: backup: Rewrite backup bitmap handling to the new bitmap semantics qemublocktest: Add 'basic' tests for backup bitmap handling qemublocktest: Add 'snapshots' tests for backup bitmap handling qemu: Rewrite bitmap handling for block commit qemublocktest: Add 'basic' tests for commit bitmap handling qemublocktest: Add 'snapshots' tests for block commit bitmap handling qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job private data qemu: Rewrite bitmap handling for block copy qemublocktest: Add test cases for handling bitmaps during block-copy kbase: Add document outlining internals of incremental backup in qemu docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 +++ src/conf/backup_conf.c | 8 + src/conf/backup_conf.h | 1 + src/qemu/qemu_backup.c | 199 +-- src/qemu/qemu_backup.h | 12 +- src/qemu/qemu_block.c | 500 +++--- src/qemu/qemu_block.h | 21 +- src/qemu/qemu_blockjob.c | 109 +- src/qemu/qemu_blockjob.h | 3 - src/qemu/qemu_checkpoint.c | 174 +-- src/qemu/qemu_checkpoint.h | 1 - src/qemu/qemu_domain.c | 22 - src/qemu/qemu_driver.c | 70 +- tests/qemublocktest.c | 254 ++- .../backupmerge/basic-deep-out.json | 35 +- .../backupmerge/basic-flat-out.json | 23 +- .../backupmerge/basic-intermediate-out.json | 23 +- .../backupmerge/empty-out.json | 3 + .../backupmerge/snapshot-deep-out.json | 38 - .../backupmerge/snapshot-flat-out.json | 6 - .../snapshot-intermediate-out.json | 14 - .../backupmerge/snapshots-deep-out.json | 41 + .../backupmerge/snapshots-flat-out.json | 25 + .../snapshots-intermediate-out.json | 29 + tests/qemublocktestdata/bitmap/basic.json | 229 +-- tests/qemublocktestdata/bitmap/basic.out | 8 +- tests/qemublocktestdata/bitmap/empty.json | 70 + tests/qemublocktestdata/bitmap/empty.out | 1 + .../bitmap/snapshots-synthetic-broken.json | 837 ---------- .../bitmap/snapshots-synthetic-broken.out | 14 - .../snapshots-synthetic-checkpoint.json | 827 ---------- .../bitmap/snapshots-synthetic-checkpoint.out | 13 - tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++---------- tests/qemublocktestdata/bitmap/snapshots.out | 13 +- tests/qemublocktestdata/bitmap/synthetic.json | 606 ++++++-- tests/qemublocktestdata/bitmap/synthetic.out | 19 +- .../bitmapblockcommit/basic-1-2 | 54 +- .../bitmapblockcommit/basic-1-3 | 54 +- .../bitmapblockcommit/basic-2-3 | 1 - .../qemublocktestdata/bitmapblockcommit/empty | 1 + .../bitmapblockcommit/snapshots-1-2 | 76 +- .../bitmapblockcommit/snapshots-1-3 | 77 +- .../bitmapblockcommit/snapshots-1-4 | 99 +- .../bitmapblockcommit/snapshots-1-5 | 103 +- .../bitmapblockcommit/snapshots-2-3 | 48 +- .../bitmapblockcommit/snapshots-2-4 | 58 +- .../bitmapblockcommit/snapshots-2-5 | 62 +- .../bitmapblockcommit/snapshots-3-4 | 34 +- .../bitmapblockcommit/snapshots-3-5 | 42 +- .../bitmapblockcommit/snapshots-4-5 | 17 - .../snapshots-synthetic-broken-1-2 | 57 - .../snapshots-synthetic-broken-1-3 | 112 -- .../snapshots-synthetic-broken-1-4 | 119 -- .../snapshots-synthetic-broken-1-5 | 119 -- .../snapshots-synthetic-broken-2-3 | 89 -- .../snapshots-synthetic-broken-2-4 | 96 -- .../snapshots-synthetic-broken-2-5 | 96 -- .../snapshots-synthetic-broken-3-4 | 27 - .../snapshots-synthetic-broken-3-5 | 27 - .../snapshots-synthetic-broken-4-5 | 20 - .../bitmapblockcopy/basic-deep-out.json | 53 +- .../bitmapblockcopy/basic-shallow-out.json | 53 +- .../bitmapblockcopy/empty-deep-out.json | 0 .../bitmapblockcopy/empty-shallow-out.json | 0 .../bitmapblockcopy/snapshots-deep-out.json | 95 +- .../snapshots-shallow-out.json | 98 +- .../checkpointdelete/basic-current-out.json | 20 - .../basic-intermediate1-out.json | 13 - .../basic-intermediate2-out.json | 13 - .../basic-intermediate3-out.json | 13 - .../checkpointdelete/empty-out.json | 1 + .../snapshots-current-out.json | 20 - .../snapshots-intermediate1-out.json | 20 +- .../snapshots-intermediate2-out.json | 42 +- .../snapshots-intermediate3-out.json | 43 - .../snapshots-noparent-out.json | 15 + ...hots-synthetic-checkpoint-current-out.json | 29 - ...ynthetic-checkpoint-intermediate1-out.json | 31 - ...ynthetic-checkpoint-intermediate2-out.json | 34 - ...ynthetic-checkpoint-intermediate3-out.json | 61 - ...ots-synthetic-checkpoint-noparent-out.json | 27 - .../synthetic-current-out.json | 9 + .../synthetic-intermediate1-out.json | 11 + .../synthetic-intermediate2-out.json | 11 + .../synthetic-intermediate3-out.json | 19 + .../synthetic-noparent-out.json | 11 + .../blockjob-blockdev-in.xml | 4 - 88 files changed, 2927 insertions(+), 5150 deletions(-) create mode 100644 docs/kbase/incrementalbackupinternals.rst create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json create mode 100644 tests/qemublocktestdata/bitmap/empty.json create mode 100644 tests/qemublocktestdata/bitmap/empty.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json -- 2.26.2
The algorithm is getting quite complex. Split out the lookup of range of backing chain storage sources and bitmaps contained in them which correspond to one checkpoint. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 111 ++++++++++++++++++++++------------------- 1 file changed, 61 insertions(+), 50 deletions(-) diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c index b317b841cd..69df6c14c6 100644 --- a/src/qemu/qemu_backup.c +++ b/src/qemu/qemu_backup.c @@ -173,72 +173,83 @@ qemuBackupDiskDataCleanup(virDomainObjPtr vm, } +static int +qemuBackupGetBitmapMergeRange(virStorageSourcePtr from, + const char *bitmapname, + virJSONValuePtr *actions, + virStorageSourcePtr *to, + const char *diskdst, + virHashTablePtr blockNamedNodeData) +{ + g_autoptr(virJSONValue) ret = virJSONValueNewArray(); + virStorageSourcePtr tmpsrc = NULL; + virStorageSourcePtr n; + bool foundbitmap = false; + + for (n = from; virStorageSourceIsBacking(n); n = n->backingStore) { + qemuBlockNamedNodeDataBitmapPtr bitmap = NULL; + + if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + n, + bitmapname))) + break; + + foundbitmap = true; + + if (bitmap->inconsistent) { + virReportError(VIR_ERR_INVALID_ARG, + _("bitmap '%s' for image '%s%u' is inconsistent"), + bitmap->name, diskdst, n->id); + return -1; + } + + if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(ret, + n->nodeformat, + bitmapname) < 0) + return -1; + + tmpsrc = n; + } + + if (!foundbitmap) { + virReportError(VIR_ERR_INVALID_ARG, + _("failed to find bitmap '%s' in image '%s%u'"), + bitmapname, diskdst, from->id); + return -1; + } + + *actions = g_steal_pointer(&ret); + *to = tmpsrc; + + return 0; +} + + virJSONValuePtr qemuBackupDiskPrepareOneBitmapsChain(virDomainMomentDefPtr *incremental, virStorageSourcePtr backingChain, virHashTablePtr blockNamedNodeData, const char *diskdst) { - qemuBlockNamedNodeDataBitmapPtr bitmap; g_autoptr(virJSONValue) ret = NULL; size_t incridx = 0; + virStorageSourcePtr n = backingChain; ret = virJSONValueNewArray(); - if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - backingChain, - incremental[0]->name))) { - virReportError(VIR_ERR_INVALID_ARG, - _("failed to find bitmap '%s' in image '%s%u'"), - incremental[0]->name, diskdst, backingChain->id); - return NULL; - } + for (incridx = 0; incremental[incridx]; incridx++) { + g_autoptr(virJSONValue) tmp = virJSONValueNewArray(); + virStorageSourcePtr tmpsrc = NULL; - while (bitmap) { - if (bitmap->inconsistent) { - virReportError(VIR_ERR_INVALID_ARG, - _("bitmap '%s' for image '%s%u' is inconsistent"), - bitmap->name, diskdst, backingChain->id); + if (qemuBackupGetBitmapMergeRange(n, incremental[incridx]->name, + &tmp, &tmpsrc, diskdst, + blockNamedNodeData) < 0) return NULL; - } - if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(ret, - backingChain->nodeformat, - bitmap->name) < 0) + if (virJSONValueArrayConcat(ret, tmp) < 0) return NULL; - if (backingChain->backingStore && - (bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - backingChain->backingStore, - incremental[incridx]->name))) { - backingChain = backingChain->backingStore; - continue; - } - - if (incremental[incridx + 1]) { - if ((bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - backingChain, - incremental[incridx + 1]->name))) { - incridx++; - continue; - } - - if (backingChain->backingStore && - (bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - backingChain->backingStore, - incremental[incridx + 1]->name))) { - incridx++; - backingChain = backingChain->backingStore; - continue; - } - - virReportError(VIR_ERR_INVALID_ARG, - _("failed to find bitmap '%s' in image '%s%u'"), - incremental[incridx]->name, diskdst, backingChain->id); - return NULL; - } else { - break; - } + n = tmpsrc; } return g_steal_pointer(&ret); -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
The algorithm is getting quite complex. Split out the lookup of range of backing chain storage sources and bitmaps contained in them which correspond to one checkpoint.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 111 ++++++++++++++++++++++------------------- 1 file changed, 61 insertions(+), 50 deletions(-)
diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c index b317b841cd..69df6c14c6 100644 --- a/src/qemu/qemu_backup.c +++ b/src/qemu/qemu_backup.c @@ -173,72 +173,83 @@ qemuBackupDiskDataCleanup(virDomainObjPtr vm, }
+static int +qemuBackupGetBitmapMergeRange(virStorageSourcePtr from, + const char *bitmapname, + virJSONValuePtr *actions, + virStorageSourcePtr *to, + const char *diskdst, + virHashTablePtr blockNamedNodeData) +{ + g_autoptr(virJSONValue) ret = virJSONValueNewArray();
Naming a pointer 'ret' when you will be returning an int seems awkward.
+ virStorageSourcePtr tmpsrc = NULL; + virStorageSourcePtr n; + bool foundbitmap = false; + + for (n = from; virStorageSourceIsBacking(n); n = n->backingStore) { + qemuBlockNamedNodeDataBitmapPtr bitmap = NULL; + + if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + n, + bitmapname))) + break; + + foundbitmap = true; + + if (bitmap->inconsistent) { + virReportError(VIR_ERR_INVALID_ARG, + _("bitmap '%s' for image '%s%u' is inconsistent"), + bitmap->name, diskdst, n->id); + return -1; + } + + if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(ret, + n->nodeformat, + bitmapname) < 0) + return -1; + + tmpsrc = n; + } + + if (!foundbitmap) { + virReportError(VIR_ERR_INVALID_ARG, + _("failed to find bitmap '%s' in image '%s%u'"), + bitmapname, diskdst, from->id); + return -1; + } + + *actions = g_steal_pointer(&ret);
Maybe s/ret/act/. Otherwise, this looks like a sane refactoring. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
If a disk is not captured by one of the intermediate checkpoints the code would fail, but we can easily calculate the bitmaps to merge correctly by skipping over checkpoints which don't describe the disk. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 24 ++++++++++++++++++++++++ tests/qemublocktest.c | 6 ++++++ 2 files changed, 30 insertions(+) diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c index 69df6c14c6..686ed57e82 100644 --- a/src/qemu/qemu_backup.c +++ b/src/qemu/qemu_backup.c @@ -240,6 +240,30 @@ qemuBackupDiskPrepareOneBitmapsChain(virDomainMomentDefPtr *incremental, for (incridx = 0; incremental[incridx]; incridx++) { g_autoptr(virJSONValue) tmp = virJSONValueNewArray(); virStorageSourcePtr tmpsrc = NULL; + virDomainCheckpointDefPtr chkdef = (virDomainCheckpointDefPtr) incremental[incridx]; + bool checkpoint_has_disk = false; + size_t i; + + for (i = 0; i < chkdef->ndisks; i++) { + if (STRNEQ_NULLABLE(diskdst, chkdef->disks[i].name)) + continue; + + if (chkdef->disks[i].type == VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP) + checkpoint_has_disk = true; + + break; + } + + if (!checkpoint_has_disk) { + if (!incremental[incridx + 1]) { + virReportError(VIR_ERR_INVALID_ARG, + _("disk '%s' not found in checkpoint '%s'"), + diskdst, incremental[incridx]->name); + return NULL; + } + + continue; + } if (qemuBackupGetBitmapMergeRange(n, incremental[incridx]->name, &tmp, &tmpsrc, diskdst, diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 0cdedb9ad4..f00d2ff129 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -727,6 +727,12 @@ testQemuBackupGetIncrementalMoment(const char *name) if (!(checkpoint = virDomainCheckpointDefNew())) abort(); + checkpoint->disks = g_new0(virDomainCheckpointDiskDef, 1); + checkpoint->ndisks = 1; + + checkpoint->disks[0].name = g_strdup("testdisk"); + checkpoint->disks[0].type = VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP; + checkpoint->parent.name = g_strdup(name); return (virDomainMomentDefPtr) checkpoint; -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
If a disk is not captured by one of the intermediate checkpoints the code would fail, but we can easily calculate the bitmaps to merge correctly by skipping over checkpoints which don't describe the disk.
If I'm understanding the setup, you can hit this by having two disks, do a checkpoint on both A+B, then another checkpoint on just B, then another checkpoint on both disks again; requesting a differential backup from the first checkpoint now has to realize that no bitmap was created for A during the second checkpoint, but that's not fatal Visually: time-> ----C1-------------C2-------------C3-------now A :bitmap1 :bitmap3 B :bitmap1 :bitmap2 :bitmap3 the differential from C1-now merges bitmap1+3 for disk A, and bitmap1+2+3 for disk B.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 24 ++++++++++++++++++++++++ tests/qemublocktest.c | 6 ++++++ 2 files changed, 30 insertions(+)
diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c index 69df6c14c6..686ed57e82 100644 --- a/src/qemu/qemu_backup.c +++ b/src/qemu/qemu_backup.c @@ -240,6 +240,30 @@ qemuBackupDiskPrepareOneBitmapsChain(virDomainMomentDefPtr *incremental, for (incridx = 0; incremental[incridx]; incridx++) { g_autoptr(virJSONValue) tmp = virJSONValueNewArray(); virStorageSourcePtr tmpsrc = NULL; + virDomainCheckpointDefPtr chkdef = (virDomainCheckpointDefPtr) incremental[incridx]; + bool checkpoint_has_disk = false; + size_t i; + + for (i = 0; i < chkdef->ndisks; i++) { + if (STRNEQ_NULLABLE(diskdst, chkdef->disks[i].name)) + continue; + + if (chkdef->disks[i].type == VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP) + checkpoint_has_disk = true; + + break; + } + + if (!checkpoint_has_disk) { + if (!incremental[incridx + 1]) { + virReportError(VIR_ERR_INVALID_ARG, + _("disk '%s' not found in checkpoint '%s'"), + diskdst, incremental[incridx]->name); + return NULL; + } + + continue; + }
if (qemuBackupGetBitmapMergeRange(n, incremental[incridx]->name, &tmp, &tmpsrc, diskdst, diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 0cdedb9ad4..f00d2ff129 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -727,6 +727,12 @@ testQemuBackupGetIncrementalMoment(const char *name) if (!(checkpoint = virDomainCheckpointDefNew())) abort();
+ checkpoint->disks = g_new0(virDomainCheckpointDiskDef, 1); + checkpoint->ndisks = 1; + + checkpoint->disks[0].name = g_strdup("testdisk"); + checkpoint->disks[0].type = VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP; +
I'm not quite sure how this test change matches the code change above, but the code change looked reasonable.
checkpoint->parent.name = g_strdup(name);
return (virDomainMomentDefPtr) checkpoint;
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Wed, Jun 17, 2020 at 13:21:04 -0500, Eric Blake wrote:
On 6/15/20 12:09 PM, Peter Krempa wrote:
If a disk is not captured by one of the intermediate checkpoints the code would fail, but we can easily calculate the bitmaps to merge correctly by skipping over checkpoints which don't describe the disk.
[...]
diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 0cdedb9ad4..f00d2ff129 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -727,6 +727,12 @@ testQemuBackupGetIncrementalMoment(const char *name) if (!(checkpoint = virDomainCheckpointDefNew())) abort();
+ checkpoint->disks = g_new0(virDomainCheckpointDiskDef, 1); + checkpoint->ndisks = 1; + + checkpoint->disks[0].name = g_strdup("testdisk"); + checkpoint->disks[0].type = VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP; +
I'm not quite sure how this test change matches the code change above, but the code change looked reasonable.
This doesn't really test the change but allows previous tests to work, by actually creating the metadata for a disk named 'testdisk' in the fake checkpoint object which was not needed for backup before.
In preparation to allow heterogenous backups store the 'incremental' field per-disk and fill it by default from the per-backup field. Having this will be important once we'll want to allow incremental backup working while hotplugging a new disk. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/conf/backup_conf.c | 8 ++++++++ src/conf/backup_conf.h | 1 + 2 files changed, 9 insertions(+) diff --git a/src/conf/backup_conf.c b/src/conf/backup_conf.c index c0c6f25d10..92106d8aaa 100644 --- a/src/conf/backup_conf.c +++ b/src/conf/backup_conf.c @@ -72,6 +72,7 @@ virDomainBackupDefFree(virDomainBackupDefPtr def) virDomainBackupDiskDefPtr disk = def->disks + i; g_free(disk->name); + g_free(disk->incremental); g_free(disk->exportname); g_free(disk->exportbitmap); virObjectUnref(disk->store); @@ -505,5 +506,12 @@ virDomainBackupAlignDisks(virDomainBackupDefPtr def, } } + for (i = 0; i < def->ndisks; i++) { + virDomainBackupDiskDefPtr backupdisk = &def->disks[i]; + + if (def->incremental && !backupdisk->incremental) + backupdisk->incremental = g_strdup(def->incremental); + } + return 0; } diff --git a/src/conf/backup_conf.h b/src/conf/backup_conf.h index b5685317c5..172eb1cf1c 100644 --- a/src/conf/backup_conf.h +++ b/src/conf/backup_conf.h @@ -51,6 +51,7 @@ typedef virDomainBackupDiskDef *virDomainBackupDiskDefPtr; struct _virDomainBackupDiskDef { char *name; /* name matching the <target dev='...' of the domain */ virTristateBool backup; /* whether backup is requested */ + char *incremental; /* name of the starting point checkpoint of an incremental backup */ char *exportname; /* name of the NBD export for pull mode backup */ char *exportbitmap; /* name of the bitmap exposed in NBD for pull mode backup */ -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
In preparation to allow heterogenous backups store the 'incremental' field per-disk and fill it by default from the per-backup field.
Having this will be important once we'll want to allow incremental backup working while hotplugging a new disk.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/conf/backup_conf.c | 8 ++++++++ src/conf/backup_conf.h | 1 + 2 files changed, 9 insertions(+)
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Fetch the checkpoint list for every disk specifically based on the new per-disk 'incremental' field. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 108 ++++++++++++++++++++--------------------- 1 file changed, 52 insertions(+), 56 deletions(-) diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c index 686ed57e82..495da86ff0 100644 --- a/src/qemu/qemu_backup.c +++ b/src/qemu/qemu_backup.c @@ -173,6 +173,50 @@ qemuBackupDiskDataCleanup(virDomainObjPtr vm, } +/** + * qemuBackupBeginCollectIncrementalCheckpoints: + * @vm: domain object + * @incrFrom: name of checkpoint representing starting point of incremental backup + * + * Returns a NULL terminated list of pointers to checkpoint definitions in + * chronological order starting from the 'current' checkpoint until reaching + * @incrFrom. + */ +static virDomainMomentDefPtr * +qemuBackupBeginCollectIncrementalCheckpoints(virDomainObjPtr vm, + const char *incrFrom) +{ + virDomainMomentObjPtr n = virDomainCheckpointGetCurrent(vm->checkpoints); + g_autofree virDomainMomentDefPtr *incr = NULL; + size_t nincr = 0; + + while (n) { + virDomainMomentDefPtr def = n->def; + + if (VIR_APPEND_ELEMENT_COPY(incr, nincr, def) < 0) + return NULL; + + if (STREQ(def->name, incrFrom)) { + def = NULL; + if (VIR_APPEND_ELEMENT_COPY(incr, nincr, def) < 0) + return NULL; + + return g_steal_pointer(&incr); + } + + if (!n->def->parent_name) + break; + + n = virDomainCheckpointFindByName(vm->checkpoints, n->def->parent_name); + } + + virReportError(VIR_ERR_OPERATION_INVALID, + _("could not locate checkpoint '%s' for incremental backup"), + incrFrom); + return NULL; +} + + static int qemuBackupGetBitmapMergeRange(virStorageSourcePtr from, const char *bitmapname, @@ -334,11 +378,11 @@ qemuBackupDiskPrepareDataOne(virDomainObjPtr vm, struct qemuBackupDiskData *dd, virJSONValuePtr actions, bool pull, - virDomainMomentDefPtr *incremental, virHashTablePtr blockNamedNodeData, virQEMUDriverConfigPtr cfg) { qemuDomainObjPrivatePtr priv = vm->privateData; + g_autofree virDomainMomentDefPtr *incremental = NULL; /* set data structure */ dd->backupdisk = backupdisk; @@ -363,7 +407,7 @@ qemuBackupDiskPrepareDataOne(virDomainObjPtr vm, * pull mode: * both: original disk */ - if (pull || (incremental && dd->store->format >= VIR_STORAGE_FILE_BACKING)) { + if (pull || (dd->backupdisk->incremental && dd->store->format >= VIR_STORAGE_FILE_BACKING)) { dd->backingStore = dd->domdisk->src; } else { dd->backingStore = dd->terminator = virStorageSourceNew(); @@ -375,7 +419,10 @@ qemuBackupDiskPrepareDataOne(virDomainObjPtr vm, if (qemuDomainPrepareStorageSourceBlockdev(NULL, dd->store, priv, cfg) < 0) return -1; - if (incremental) { + if (dd->backupdisk->incremental) { + if (!(incremental = qemuBackupBeginCollectIncrementalCheckpoints(vm, dd->backupdisk->incremental))) + return -1; + if (dd->backupdisk->exportbitmap) dd->incrementalBitmap = g_strdup(dd->backupdisk->exportbitmap); else @@ -444,7 +491,6 @@ qemuBackupDiskPrepareDataOnePull(virJSONValuePtr actions, static ssize_t qemuBackupDiskPrepareData(virDomainObjPtr vm, virDomainBackupDefPtr def, - virDomainMomentDefPtr *incremental, virHashTablePtr blockNamedNodeData, virJSONValuePtr actions, virQEMUDriverConfigPtr cfg, @@ -467,8 +513,7 @@ qemuBackupDiskPrepareData(virDomainObjPtr vm, ndisks++; if (qemuBackupDiskPrepareDataOne(vm, backupdisk, dd, actions, pull, - incremental, blockNamedNodeData, - cfg) < 0) + blockNamedNodeData, cfg) < 0) goto error; if (pull) { @@ -625,50 +670,6 @@ qemuBackupBeginPullExportDisks(virDomainObjPtr vm, } -/** - * qemuBackupBeginCollectIncrementalCheckpoints: - * @vm: domain object - * @incrFrom: name of checkpoint representing starting point of incremental backup - * - * Returns a NULL terminated list of pointers to checkpoint definitions in - * chronological order starting from the 'current' checkpoint until reaching - * @incrFrom. - */ -static virDomainMomentDefPtr * -qemuBackupBeginCollectIncrementalCheckpoints(virDomainObjPtr vm, - const char *incrFrom) -{ - virDomainMomentObjPtr n = virDomainCheckpointGetCurrent(vm->checkpoints); - g_autofree virDomainMomentDefPtr *incr = NULL; - size_t nincr = 0; - - while (n) { - virDomainMomentDefPtr def = n->def; - - if (VIR_APPEND_ELEMENT_COPY(incr, nincr, def) < 0) - return NULL; - - if (STREQ(def->name, incrFrom)) { - def = NULL; - if (VIR_APPEND_ELEMENT_COPY(incr, nincr, def) < 0) - return NULL; - - return g_steal_pointer(&incr); - } - - if (!n->def->parent_name) - break; - - n = virDomainCheckpointFindByName(vm->checkpoints, n->def->parent_name); - } - - virReportError(VIR_ERR_OPERATION_INVALID, - _("could not locate checkpoint '%s' for incremental backup"), - incrFrom); - return NULL; -} - - void qemuBackupJobTerminate(virDomainObjPtr vm, qemuDomainJobStatus jobstatus) @@ -802,7 +803,6 @@ qemuBackupBegin(virDomainObjPtr vm, bool pull = false; virDomainMomentObjPtr chk = NULL; g_autoptr(virDomainCheckpointDef) chkdef = NULL; - g_autofree virDomainMomentDefPtr *incremental = NULL; g_autoptr(virJSONValue) actions = NULL; struct qemuBackupDiskData *dd = NULL; ssize_t ndd = 0; @@ -870,10 +870,6 @@ qemuBackupBegin(virDomainObjPtr vm, if (virDomainBackupAlignDisks(def, vm->def, suffix) < 0) goto endjob; - if (def->incremental && - !(incremental = qemuBackupBeginCollectIncrementalCheckpoints(vm, def->incremental))) - goto endjob; - actions = virJSONValueNewArray(); /* The 'chk' checkpoint must be rolled back if the transaction command @@ -887,7 +883,7 @@ qemuBackupBegin(virDomainObjPtr vm, if (!(blockNamedNodeData = qemuBlockGetNamedNodeData(vm, QEMU_ASYNC_JOB_BACKUP))) goto endjob; - if ((ndd = qemuBackupDiskPrepareData(vm, def, incremental, blockNamedNodeData, + if ((ndd = qemuBackupDiskPrepareData(vm, def, blockNamedNodeData, actions, cfg, &dd)) <= 0) { if (ndd == 0) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
Fetch the checkpoint list for every disk specifically based on the new per-disk 'incremental' field.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 108 ++++++++++++++++++++--------------------- 1 file changed, 52 insertions(+), 56 deletions(-)
@@ -363,7 +407,7 @@ qemuBackupDiskPrepareDataOne(virDomainObjPtr vm, * pull mode: * both: original disk */ - if (pull || (incremental && dd->store->format >= VIR_STORAGE_FILE_BACKING)) { + if (pull || (dd->backupdisk->incremental && dd->store->format >= VIR_STORAGE_FILE_BACKING)) {
Long line, if you want to break at the &&.
dd->backingStore = dd->domdisk->src; } else { dd->backingStore = dd->terminator = virStorageSourceNew(); @@ -375,7 +419,10 @@ qemuBackupDiskPrepareDataOne(virDomainObjPtr vm, if (qemuDomainPrepareStorageSourceBlockdev(NULL, dd->store, priv, cfg) < 0) return -1;
- if (incremental) { + if (dd->backupdisk->incremental) { + if (!(incremental = qemuBackupBeginCollectIncrementalCheckpoints(vm, dd->backupdisk->incremental)))
Another long line, but here it is harder to suggest a short alternative. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Add test data for an image without bitmaps. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 ++ tests/qemublocktestdata/bitmap/empty.json | 70 +++++++++++++++++++++++ tests/qemublocktestdata/bitmap/empty.out | 1 + 3 files changed, 75 insertions(+) create mode 100644 tests/qemublocktestdata/bitmap/empty.json create mode 100644 tests/qemublocktestdata/bitmap/empty.out diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index f00d2ff129..8d613d7cac 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1292,6 +1292,8 @@ mymain(void) ret = -1; \ } while (0) + TEST_BITMAP_DETECT("empty"); + TEST_BITMAP_DETECT("basic"); TEST_BITMAP_DETECT("synthetic"); TEST_BITMAP_DETECT("snapshots"); @@ -1360,6 +1362,8 @@ mymain(void) ret = -1; \ } while (0) + TEST_BITMAP_VALIDATE("empty", "a", false); + TEST_BITMAP_VALIDATE("basic", "a", true); TEST_BITMAP_VALIDATE("basic", "b", true); TEST_BITMAP_VALIDATE("basic", "c", true); diff --git a/tests/qemublocktestdata/bitmap/empty.json b/tests/qemublocktestdata/bitmap/empty.json new file mode 100644 index 0000000000..ec43b25f0d --- /dev/null +++ b/tests/qemublocktestdata/bitmap/empty.json @@ -0,0 +1,70 @@ +[ + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 10485760, + "filename": "/tmp/pull4.qcow2", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 200704, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "lazy-refcounts": false, + "refcount-bits": 16, + "corrupt": false + } + }, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/pull4.qcow2", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 197120, + "filename": "/tmp/pull4.qcow2", + "format": "file", + "actual-size": 200704, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/pull4.qcow2", + "encryption_key_missing": false + } +] diff --git a/tests/qemublocktestdata/bitmap/empty.out b/tests/qemublocktestdata/bitmap/empty.out new file mode 100644 index 0000000000..3787cbd354 --- /dev/null +++ b/tests/qemublocktestdata/bitmap/empty.out @@ -0,0 +1 @@ +libvirt-1-format: -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
Add test data for an image without bitmaps.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 ++ tests/qemublocktestdata/bitmap/empty.json | 70 +++++++++++++++++++++++ tests/qemublocktestdata/bitmap/empty.out | 1 + 3 files changed, 75 insertions(+) create mode 100644 tests/qemublocktestdata/bitmap/empty.json create mode 100644 tests/qemublocktestdata/bitmap/empty.out
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Use the new test data when calculating incremental backup operations. As incremental backup fails with no bitmap the test code is modified to allow testing this case too. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 19 ++++++++++--------- .../backupmerge/empty-out.json | 1 + 2 files changed, 11 insertions(+), 9 deletions(-) create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 8d613d7cac..9e54c254e8 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -793,17 +793,16 @@ testQemuBackupIncrementalBitmapCalculate(const void *opaque) incremental = testQemuBackupGetIncremental(data->incremental); - if (!(mergebitmaps = qemuBackupDiskPrepareOneBitmapsChain(incremental, - data->chain, - nodedata, - "testdisk"))) { - VIR_TEST_VERBOSE("failed to calculate merged bitmaps"); - return -1; + if ((mergebitmaps = qemuBackupDiskPrepareOneBitmapsChain(incremental, + data->chain, + nodedata, + "testdisk"))) { + if (!(actual = virJSONValueToString(mergebitmaps, true))) + return -1; + } else { + actual = g_strdup("NULL\n"); } - if (!(actual = virJSONValueToString(mergebitmaps, true))) - return -1; - return virTestCompareToFile(actual, expectpath); } @@ -1312,6 +1311,8 @@ mymain(void) ret = -1; \ } while (0) + TEST_BACKUP_BITMAP_CALCULATE("empty", bitmapSourceChain, "a", "empty"); + TEST_BACKUP_BITMAP_CALCULATE("basic-flat", bitmapSourceChain, "current", "basic"); TEST_BACKUP_BITMAP_CALCULATE("basic-intermediate", bitmapSourceChain, "d", "basic"); TEST_BACKUP_BITMAP_CALCULATE("basic-deep", bitmapSourceChain, "a", "basic"); diff --git a/tests/qemublocktestdata/backupmerge/empty-out.json b/tests/qemublocktestdata/backupmerge/empty-out.json new file mode 100644 index 0000000000..7951defec1 --- /dev/null +++ b/tests/qemublocktestdata/backupmerge/empty-out.json @@ -0,0 +1 @@ +NULL -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
Use the new test data when calculating incremental backup operations. As incremental backup fails with no bitmap the test code is modified to allow testing this case too.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 19 ++++++++++--------- .../backupmerge/empty-out.json | 1 + 2 files changed, 11 insertions(+), 9 deletions(-) create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Use the new test data for checkpoint deletion testing. This test also requires modification of the internals to allow checking for test failure. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 13 +++++++------ .../checkpointdelete/empty-out.json | 1 + 2 files changed, 8 insertions(+), 6 deletions(-) create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 9e54c254e8..1821c227d5 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -851,14 +851,13 @@ testQemuCheckpointDeleteMerge(const void *opaque) data->parentbitmap, actions, "testdisk", - &reopenimages) < 0) { - VIR_TEST_VERBOSE("failed to generate checkpoint delete transaction\n"); - return -1; + &reopenimages) >= 0) { + if (virJSONValueToBuffer(actions, &buf, true) < 0) + return -1; + } else { + virBufferAddLit(&buf, "NULL\n"); } - if (virJSONValueToBuffer(actions, &buf, true) < 0) - return -1; - if (reopenimages) { virBufferAddLit(&buf, "reopen nodes:\n"); @@ -1333,6 +1332,8 @@ mymain(void) ret = -1; \ } while (0) + TEST_CHECKPOINT_DELETE_MERGE("empty", "a", NULL, "empty"); + TEST_CHECKPOINT_DELETE_MERGE("basic-noparent", "a", NULL, "basic"); TEST_CHECKPOINT_DELETE_MERGE("basic-intermediate1", "b", "a", "basic"); TEST_CHECKPOINT_DELETE_MERGE("basic-intermediate2", "c", "b", "basic"); diff --git a/tests/qemublocktestdata/checkpointdelete/empty-out.json b/tests/qemublocktestdata/checkpointdelete/empty-out.json new file mode 100644 index 0000000000..7951defec1 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/empty-out.json @@ -0,0 +1 @@ +NULL -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
Use the new test data for checkpoint deletion testing. This test also requires modification of the internals to allow checking for test failure.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 13 +++++++------ .../checkpointdelete/empty-out.json | 1 + 2 files changed, 8 insertions(+), 6 deletions(-) create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 3 +++ tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json | 0 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json | 0 3 files changed, 3 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 1821c227d5..2ddfab72eb 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1408,6 +1408,9 @@ mymain(void) ret = -1; \ } while (0) + TEST_BITMAP_BLOCKCOPY("empty-shallow", true, "empty"); + TEST_BITMAP_BLOCKCOPY("empty-deep", false, "empty"); + TEST_BITMAP_BLOCKCOPY("basic-shallow", true, "basic"); TEST_BITMAP_BLOCKCOPY("basic-deep", false, "basic"); diff --git a/tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json b/tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json b/tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json new file mode 100644 index 0000000000..e69de29bb2 -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 3 +++ tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json | 0 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json | 0 3 files changed, 3 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 2 ++ tests/qemublocktestdata/bitmapblockcommit/empty | 2 ++ 2 files changed, 4 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 2ddfab72eb..d66e169fd8 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1430,6 +1430,8 @@ mymain(void) ret = -1; \ } while (0) + TEST_BITMAP_BLOCKCOMMIT("empty", 1, 2, "empty"); + TEST_BITMAP_BLOCKCOMMIT("basic-1-2", 1, 2, "basic"); TEST_BITMAP_BLOCKCOMMIT("basic-1-3", 1, 3, "basic"); TEST_BITMAP_BLOCKCOMMIT("basic-2-3", 2, 3, "basic"); diff --git a/tests/qemublocktestdata/bitmapblockcommit/empty b/tests/qemublocktestdata/bitmapblockcommit/empty new file mode 100644 index 0000000000..bfc58f994e --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/empty @@ -0,0 +1,2 @@ +pre job bitmap disable: +merge bitmpas: -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 2 ++ tests/qemublocktestdata/bitmapblockcommit/empty | 2 ++ 2 files changed, 4 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Upcoming patches are going to rewrite and semantically modify how bitmaps are handled during blockjobs. This is possible as incremental backup is not yet fully enabled. As the changes are going to be incompatible with any current test data remove all test cases for bitmap handling during checkpoint deletion, incremental backups, block commit, block copy, and bitmap validation operations. The tests will be gradually added back later after the code and test-data is refactored. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 94 ------------- .../backupmerge/basic-deep-out.json | 22 --- .../backupmerge/basic-flat-out.json | 6 - .../backupmerge/basic-intermediate-out.json | 10 -- .../backupmerge/snapshot-deep-out.json | 38 ----- .../backupmerge/snapshot-flat-out.json | 6 - .../snapshot-intermediate-out.json | 14 -- .../bitmapblockcommit/basic-1-2 | 119 ---------------- .../bitmapblockcommit/basic-1-3 | 119 ---------------- .../bitmapblockcommit/basic-2-3 | 2 - .../bitmapblockcommit/snapshots-1-2 | 49 ------- .../bitmapblockcommit/snapshots-1-3 | 76 ---------- .../bitmapblockcommit/snapshots-1-4 | 126 ----------------- .../bitmapblockcommit/snapshots-1-5 | 130 ----------------- .../bitmapblockcommit/snapshots-2-3 | 49 ------- .../bitmapblockcommit/snapshots-2-4 | 99 ------------- .../bitmapblockcommit/snapshots-2-5 | 103 -------------- .../bitmapblockcommit/snapshots-3-4 | 72 ---------- .../bitmapblockcommit/snapshots-3-5 | 76 ---------- .../bitmapblockcommit/snapshots-4-5 | 33 ----- .../snapshots-synthetic-broken-1-2 | 57 -------- .../snapshots-synthetic-broken-1-3 | 112 --------------- .../snapshots-synthetic-broken-1-4 | 119 ---------------- .../snapshots-synthetic-broken-1-5 | 119 ---------------- .../snapshots-synthetic-broken-2-3 | 89 ------------ .../snapshots-synthetic-broken-2-4 | 96 ------------- .../snapshots-synthetic-broken-2-5 | 96 ------------- .../snapshots-synthetic-broken-3-4 | 27 ---- .../snapshots-synthetic-broken-3-5 | 27 ---- .../snapshots-synthetic-broken-4-5 | 20 --- .../bitmapblockcopy/basic-deep-out.json | 117 --------------- .../bitmapblockcopy/basic-shallow-out.json | 117 --------------- .../bitmapblockcopy/snapshots-deep-out.json | 133 ------------------ .../snapshots-shallow-out.json | 48 ------- .../checkpointdelete/basic-current-out.json | 29 ---- .../basic-intermediate1-out.json | 22 --- .../basic-intermediate2-out.json | 22 --- .../basic-intermediate3-out.json | 22 --- .../checkpointdelete/basic-noparent-out.json | 9 -- .../snapshots-current-out.json | 29 ---- .../snapshots-intermediate1-out.json | 24 ---- .../snapshots-intermediate2-out.json | 62 -------- .../snapshots-intermediate3-out.json | 61 -------- .../snapshots-noparent-out.json | 27 ---- ...hots-synthetic-checkpoint-current-out.json | 29 ---- ...ynthetic-checkpoint-intermediate1-out.json | 31 ---- ...ynthetic-checkpoint-intermediate2-out.json | 34 ----- ...ynthetic-checkpoint-intermediate3-out.json | 61 -------- ...ots-synthetic-checkpoint-noparent-out.json | 27 ---- 49 files changed, 2909 deletions(-) delete mode 100644 tests/qemublocktestdata/backupmerge/basic-deep-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/basic-flat-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/basic-intermediate-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json delete mode 100644 tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json delete mode 100644 tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json delete mode 100644 tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/basic-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index d66e169fd8..d2f105486e 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1312,14 +1312,6 @@ mymain(void) TEST_BACKUP_BITMAP_CALCULATE("empty", bitmapSourceChain, "a", "empty"); - TEST_BACKUP_BITMAP_CALCULATE("basic-flat", bitmapSourceChain, "current", "basic"); - TEST_BACKUP_BITMAP_CALCULATE("basic-intermediate", bitmapSourceChain, "d", "basic"); - TEST_BACKUP_BITMAP_CALCULATE("basic-deep", bitmapSourceChain, "a", "basic"); - - TEST_BACKUP_BITMAP_CALCULATE("snapshot-flat", bitmapSourceChain, "current", "snapshots"); - TEST_BACKUP_BITMAP_CALCULATE("snapshot-intermediate", bitmapSourceChain, "d", "snapshots"); - TEST_BACKUP_BITMAP_CALCULATE("snapshot-deep", bitmapSourceChain, "a", "snapshots"); - #define TEST_CHECKPOINT_DELETE_MERGE(testname, delbmp, parbmp, named) \ do { \ checkpointdeletedata.name = testname; \ @@ -1334,24 +1326,6 @@ mymain(void) TEST_CHECKPOINT_DELETE_MERGE("empty", "a", NULL, "empty"); - TEST_CHECKPOINT_DELETE_MERGE("basic-noparent", "a", NULL, "basic"); - TEST_CHECKPOINT_DELETE_MERGE("basic-intermediate1", "b", "a", "basic"); - TEST_CHECKPOINT_DELETE_MERGE("basic-intermediate2", "c", "b", "basic"); - TEST_CHECKPOINT_DELETE_MERGE("basic-intermediate3", "d", "c", "basic"); - TEST_CHECKPOINT_DELETE_MERGE("basic-current", "current", "d", "basic"); - - TEST_CHECKPOINT_DELETE_MERGE("snapshots-noparent", "a", NULL, "snapshots"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-intermediate1", "b", "a", "snapshots"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-intermediate2", "c", "b", "snapshots"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-intermediate3", "d", "c", "snapshots"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-current", "current", "d", "snapshots"); - - TEST_CHECKPOINT_DELETE_MERGE("snapshots-synthetic-checkpoint-noparent", "a", NULL, "snapshots-synthetic-checkpoint"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-synthetic-checkpoint-intermediate1", "b", "a", "snapshots-synthetic-checkpoint"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-synthetic-checkpoint-intermediate2", "c", "b", "snapshots-synthetic-checkpoint"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-synthetic-checkpoint-intermediate3", "d", "c", "snapshots-synthetic-checkpoint"); - TEST_CHECKPOINT_DELETE_MERGE("snapshots-synthetic-checkpoint-current", "current", "d", "snapshots-synthetic-checkpoint"); - #define TEST_BITMAP_VALIDATE(testname, bitmap, rc) \ do { \ blockbitmapvalidatedata.name = testname; \ @@ -1366,36 +1340,6 @@ mymain(void) TEST_BITMAP_VALIDATE("empty", "a", false); - TEST_BITMAP_VALIDATE("basic", "a", true); - TEST_BITMAP_VALIDATE("basic", "b", true); - TEST_BITMAP_VALIDATE("basic", "c", true); - TEST_BITMAP_VALIDATE("basic", "d", true); - TEST_BITMAP_VALIDATE("basic", "current", true); - - TEST_BITMAP_VALIDATE("snapshots", "a", true); - TEST_BITMAP_VALIDATE("snapshots", "b", true); - TEST_BITMAP_VALIDATE("snapshots", "c", true); - TEST_BITMAP_VALIDATE("snapshots", "d", true); - TEST_BITMAP_VALIDATE("snapshots", "current", true); - - TEST_BITMAP_VALIDATE("synthetic", "a", false); - TEST_BITMAP_VALIDATE("synthetic", "b", true); - TEST_BITMAP_VALIDATE("synthetic", "c", true); - TEST_BITMAP_VALIDATE("synthetic", "d", true); - TEST_BITMAP_VALIDATE("synthetic", "current", true); - - TEST_BITMAP_VALIDATE("snapshots-synthetic-checkpoint", "a", true); - TEST_BITMAP_VALIDATE("snapshots-synthetic-checkpoint", "b", true); - TEST_BITMAP_VALIDATE("snapshots-synthetic-checkpoint", "c", true); - TEST_BITMAP_VALIDATE("snapshots-synthetic-checkpoint", "d", true); - TEST_BITMAP_VALIDATE("snapshots-synthetic-checkpoint", "current", true); - - TEST_BITMAP_VALIDATE("snapshots-synthetic-broken", "a", false); - TEST_BITMAP_VALIDATE("snapshots-synthetic-broken", "b", true); - TEST_BITMAP_VALIDATE("snapshots-synthetic-broken", "c", true); - TEST_BITMAP_VALIDATE("snapshots-synthetic-broken", "d", false); - TEST_BITMAP_VALIDATE("snapshots-synthetic-broken", "current", true); - #define TEST_BITMAP_BLOCKCOPY(testname, shllw, ndf) \ do { \ blockbitmapblockcopydata.name = testname; \ @@ -1411,12 +1355,6 @@ mymain(void) TEST_BITMAP_BLOCKCOPY("empty-shallow", true, "empty"); TEST_BITMAP_BLOCKCOPY("empty-deep", false, "empty"); - TEST_BITMAP_BLOCKCOPY("basic-shallow", true, "basic"); - TEST_BITMAP_BLOCKCOPY("basic-deep", false, "basic"); - - TEST_BITMAP_BLOCKCOPY("snapshots-shallow", true, "snapshots"); - TEST_BITMAP_BLOCKCOPY("snapshots-deep", false, "snapshots"); - #define TEST_BITMAP_BLOCKCOMMIT(testname, topimg, baseimg, ndf) \ do {\ @@ -1432,38 +1370,6 @@ mymain(void) TEST_BITMAP_BLOCKCOMMIT("empty", 1, 2, "empty"); - TEST_BITMAP_BLOCKCOMMIT("basic-1-2", 1, 2, "basic"); - TEST_BITMAP_BLOCKCOMMIT("basic-1-3", 1, 3, "basic"); - TEST_BITMAP_BLOCKCOMMIT("basic-2-3", 2, 3, "basic"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-1-2", 1, 2, "snapshots"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-1-3", 1, 3, "snapshots"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-1-4", 1, 4, "snapshots"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-1-5", 1, 5, "snapshots"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-2-3", 2, 3, "snapshots"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-2-4", 2, 4, "snapshots"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-2-5", 2, 5, "snapshots"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-3-4", 3, 4, "snapshots"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-3-5", 3, 5, "snapshots"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-4-5", 4, 5, "snapshots"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-1-2", 1, 2, "snapshots-synthetic-broken"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-1-3", 1, 3, "snapshots-synthetic-broken"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-1-4", 1, 4, "snapshots-synthetic-broken"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-1-5", 1, 5, "snapshots-synthetic-broken"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-2-3", 2, 3, "snapshots-synthetic-broken"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-2-4", 2, 4, "snapshots-synthetic-broken"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-2-5", 2, 5, "snapshots-synthetic-broken"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-3-4", 3, 4, "snapshots-synthetic-broken"); - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-3-5", 3, 5, "snapshots-synthetic-broken"); - - TEST_BITMAP_BLOCKCOMMIT("snapshots-synthetic-broken-4-5", 4, 5, "snapshots-synthetic-broken"); - cleanup: qemuTestDriverFree(&driver); VIR_FREE(capslatest_x86_64); diff --git a/tests/qemublocktestdata/backupmerge/basic-deep-out.json b/tests/qemublocktestdata/backupmerge/basic-deep-out.json deleted file mode 100644 index 28c3d16259..0000000000 --- a/tests/qemublocktestdata/backupmerge/basic-deep-out.json +++ /dev/null @@ -1,22 +0,0 @@ -[ - { - "node": "libvirt-1-format", - "name": "current" - }, - { - "node": "libvirt-1-format", - "name": "d" - }, - { - "node": "libvirt-1-format", - "name": "c" - }, - { - "node": "libvirt-1-format", - "name": "b" - }, - { - "node": "libvirt-1-format", - "name": "a" - } -] diff --git a/tests/qemublocktestdata/backupmerge/basic-flat-out.json b/tests/qemublocktestdata/backupmerge/basic-flat-out.json deleted file mode 100644 index b89252e284..0000000000 --- a/tests/qemublocktestdata/backupmerge/basic-flat-out.json +++ /dev/null @@ -1,6 +0,0 @@ -[ - { - "node": "libvirt-1-format", - "name": "current" - } -] diff --git a/tests/qemublocktestdata/backupmerge/basic-intermediate-out.json b/tests/qemublocktestdata/backupmerge/basic-intermediate-out.json deleted file mode 100644 index 0dffcafd5f..0000000000 --- a/tests/qemublocktestdata/backupmerge/basic-intermediate-out.json +++ /dev/null @@ -1,10 +0,0 @@ -[ - { - "node": "libvirt-1-format", - "name": "current" - }, - { - "node": "libvirt-1-format", - "name": "d" - } -] diff --git a/tests/qemublocktestdata/backupmerge/snapshot-deep-out.json b/tests/qemublocktestdata/backupmerge/snapshot-deep-out.json deleted file mode 100644 index 526fc8d55b..0000000000 --- a/tests/qemublocktestdata/backupmerge/snapshot-deep-out.json +++ /dev/null @@ -1,38 +0,0 @@ -[ - { - "node": "libvirt-1-format", - "name": "current" - }, - { - "node": "libvirt-1-format", - "name": "d" - }, - { - "node": "libvirt-2-format", - "name": "d" - }, - { - "node": "libvirt-2-format", - "name": "c" - }, - { - "node": "libvirt-3-format", - "name": "c" - }, - { - "node": "libvirt-3-format", - "name": "b" - }, - { - "node": "libvirt-3-format", - "name": "a" - }, - { - "node": "libvirt-4-format", - "name": "a" - }, - { - "node": "libvirt-5-format", - "name": "a" - } -] diff --git a/tests/qemublocktestdata/backupmerge/snapshot-flat-out.json b/tests/qemublocktestdata/backupmerge/snapshot-flat-out.json deleted file mode 100644 index b89252e284..0000000000 --- a/tests/qemublocktestdata/backupmerge/snapshot-flat-out.json +++ /dev/null @@ -1,6 +0,0 @@ -[ - { - "node": "libvirt-1-format", - "name": "current" - } -] diff --git a/tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json b/tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json deleted file mode 100644 index 537d776ec6..0000000000 --- a/tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json +++ /dev/null @@ -1,14 +0,0 @@ -[ - { - "node": "libvirt-1-format", - "name": "current" - }, - { - "node": "libvirt-1-format", - "name": "d" - }, - { - "node": "libvirt-2-format", - "name": "d" - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/basic-1-2 b/tests/qemublocktestdata/bitmapblockcommit/basic-1-2 deleted file mode 100644 index 8eeb4c3a11..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/basic-1-2 +++ /dev/null @@ -1,119 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "a", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/basic-1-3 b/tests/qemublocktestdata/bitmapblockcommit/basic-1-3 deleted file mode 100644 index 71b48e31a5..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/basic-1-3 +++ /dev/null @@ -1,119 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "a", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/basic-2-3 b/tests/qemublocktestdata/bitmapblockcommit/basic-2-3 deleted file mode 100644 index bfc58f994e..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/basic-2-3 +++ /dev/null @@ -1,2 +0,0 @@ -pre job bitmap disable: -merge bitmpas: diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 deleted file mode 100644 index 0015b9ceb3..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 +++ /dev/null @@ -1,49 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-2-format", - "name": "d" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 deleted file mode 100644 index 5691b408aa..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 +++ /dev/null @@ -1,76 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-3-format", - "name": "c" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - }, - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 deleted file mode 100644 index 454001531a..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 +++ /dev/null @@ -1,126 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-4-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - }, - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - }, - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 deleted file mode 100644 index 2fd43d7917..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 +++ /dev/null @@ -1,130 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "a" - }, - { - "node": "libvirt-4-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - }, - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - }, - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 deleted file mode 100644 index d719a90bd7..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 +++ /dev/null @@ -1,49 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-3-format", - "name": "c" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "d", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 deleted file mode 100644 index 9e37962344..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 +++ /dev/null @@ -1,99 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-4-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - }, - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "d", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 deleted file mode 100644 index d6b20a5d05..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 +++ /dev/null @@ -1,103 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "a" - }, - { - "node": "libvirt-4-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - }, - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "d", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 deleted file mode 100644 index b96e8910d7..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 +++ /dev/null @@ -1,72 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-4-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "c", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 deleted file mode 100644 index 9570c34c40..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 +++ /dev/null @@ -1,76 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "a" - }, - { - "node": "libvirt-4-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "c", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 deleted file mode 100644 index 7e1020d96e..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 +++ /dev/null @@ -1,33 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-4-format", - "name": "a" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 deleted file mode 100644 index 463120d442..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 +++ /dev/null @@ -1,57 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-2-format", - "name": "oa" - } - }, - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-2-format", - "name": "ob" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-2-format", - "name": "oa" - } - }, - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-2-format", - "name": "ob" - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 deleted file mode 100644 index fec6f95dd1..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 +++ /dev/null @@ -1,112 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-3-format", - "name": "b" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-3-format", - "name": "b" - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "oa", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "oa", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "oa" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "ob", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "ob", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "ob" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 deleted file mode 100644 index 697230f67b..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 +++ /dev/null @@ -1,119 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "b", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "oa", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "oa", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "oa" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "ob", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "ob", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "ob" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 deleted file mode 100644 index 6bf1f0da64..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 +++ /dev/null @@ -1,119 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "b", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "oa", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "oa", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "oa" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "ob", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "ob", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "ob" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 deleted file mode 100644 index f202bb94b1..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 +++ /dev/null @@ -1,89 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-3-format", - "name": "b" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-3-format", - "name": "b" - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "oa", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "oa", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "oa" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-3-format", - "name": "ob", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "ob", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "ob" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 deleted file mode 100644 index 864cc9041b..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 +++ /dev/null @@ -1,96 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "b", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "oa", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "oa", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "oa" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "ob", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "ob", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "ob" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 deleted file mode 100644 index 4c5d8dbe80..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 +++ /dev/null @@ -1,96 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "b", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "oa", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "oa", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "oa" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "ob", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "ob", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "ob" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 deleted file mode 100644 index 367a930a74..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 +++ /dev/null @@ -1,27 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-4-format", - "name": "b", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-4-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 deleted file mode 100644 index 0062ec140c..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 +++ /dev/null @@ -1,27 +0,0 @@ -pre job bitmap disable: -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-5-format", - "name": "b", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-5-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 deleted file mode 100644 index b1f10a8a24..0000000000 --- a/tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 +++ /dev/null @@ -1,20 +0,0 @@ -pre job bitmap disable: -[ - { - "type": "block-dirty-bitmap-disable", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] -merge bitmpas: -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json b/tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json deleted file mode 100644 index 4ed2b97e95..0000000000 --- a/tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json +++ /dev/null @@ -1,117 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "a", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json b/tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json deleted file mode 100644 index 4ed2b97e95..0000000000 --- a/tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json +++ /dev/null @@ -1,117 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "a", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json b/tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json deleted file mode 100644 index 5456553d78..0000000000 --- a/tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json +++ /dev/null @@ -1,133 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "a", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "a" - }, - { - "node": "libvirt-4-format", - "name": "a" - }, - { - "node": "libvirt-5-format", - "name": "a" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - }, - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - }, - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json b/tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json deleted file mode 100644 index ddd47f7ee1..0000000000 --- a/tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json +++ /dev/null @@ -1,48 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "current", - "persistent": true, - "disabled": false, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "current", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "mirror-format-node", - "name": "d", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "mirror-format-node", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-current-out.json b/tests/qemublocktestdata/checkpointdelete/basic-current-out.json deleted file mode 100644 index 1b607567e8..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/basic-current-out.json +++ /dev/null @@ -1,29 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-1-format", - "name": "d" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "current" - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json b/tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json deleted file mode 100644 index eccb7ed15f..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json +++ /dev/null @@ -1,22 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "b" - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json b/tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json deleted file mode 100644 index de40e4b5b0..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json +++ /dev/null @@ -1,22 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "c" - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json b/tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json deleted file mode 100644 index b5d85f43f0..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json +++ /dev/null @@ -1,22 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "d" - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json b/tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json deleted file mode 100644 index e87382fdb4..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json +++ /dev/null @@ -1,9 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "a" - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json deleted file mode 100644 index 1b607567e8..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json +++ /dev/null @@ -1,29 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-1-format", - "name": "d" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "current" - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json deleted file mode 100644 index c9bda3a17a..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json +++ /dev/null @@ -1,24 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-3-format", - "name": "b" - } - } -] -reopen nodes: -libvirt-3-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json deleted file mode 100644 index 8a0e3f2cff..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json +++ /dev/null @@ -1,62 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-2-format", - "name": "c" - } - }, - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-3-format", - "name": "b" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-3-format", - "name": "c" - } - } -] -reopen nodes: -libvirt-3-format -libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json deleted file mode 100644 index 211bc40baf..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json +++ /dev/null @@ -1,61 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-1-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "d" - } - }, - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-2-format", - "name": "c" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-2-format", - "name": "d" - } - } -] -reopen nodes: -libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json deleted file mode 100644 index f750f44da2..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json +++ /dev/null @@ -1,27 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-3-format", - "name": "a" - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-4-format", - "name": "a" - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] -reopen nodes: -libvirt-5-format -libvirt-4-format -libvirt-3-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json deleted file mode 100644 index 1b607567e8..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json +++ /dev/null @@ -1,29 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-1-format", - "name": "d" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "d", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "current" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "current" - } - } -] diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json deleted file mode 100644 index d7e6d18637..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json +++ /dev/null @@ -1,31 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-3-format", - "name": "a" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-3-format", - "target": "a", - "bitmaps": [ - { - "node": "libvirt-3-format", - "name": "b" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-3-format", - "name": "b" - } - } -] -reopen nodes: -libvirt-3-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json deleted file mode 100644 index cfbff010c2..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json +++ /dev/null @@ -1,34 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-2-format", - "name": "b", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "b", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "c" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-2-format", - "name": "c" - } - } -] -reopen nodes: -libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json deleted file mode 100644 index 211bc40baf..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json +++ /dev/null @@ -1,61 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-add", - "data": { - "node": "libvirt-1-format", - "name": "c", - "persistent": true, - "disabled": true, - "granularity": 65536 - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-1-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-1-format", - "name": "d" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-1-format", - "name": "d" - } - }, - { - "type": "block-dirty-bitmap-enable", - "data": { - "node": "libvirt-2-format", - "name": "c" - } - }, - { - "type": "block-dirty-bitmap-merge", - "data": { - "node": "libvirt-2-format", - "target": "c", - "bitmaps": [ - { - "node": "libvirt-2-format", - "name": "d" - } - ] - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-2-format", - "name": "d" - } - } -] -reopen nodes: -libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json deleted file mode 100644 index f750f44da2..0000000000 --- a/tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json +++ /dev/null @@ -1,27 +0,0 @@ -[ - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-3-format", - "name": "a" - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-4-format", - "name": "a" - } - }, - { - "type": "block-dirty-bitmap-remove", - "data": { - "node": "libvirt-5-format", - "name": "a" - } - } -] -reopen nodes: -libvirt-5-format -libvirt-4-format -libvirt-3-format -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
Upcoming patches are going to rewrite and semantically modify how bitmaps are handled during blockjobs. This is possible as incremental backup is not yet fully enabled.
As the changes are going to be incompatible with any current test data remove all test cases for bitmap handling during checkpoint deletion, incremental backups, block commit, block copy, and bitmap validation operations.
The tests will be gradually added back later after the code and test-data is refactored.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> ---
49 files changed, 2909 deletions(-)
And this is the point where we have to commit on which interface we will stick to once we allow incremental backups by default rather than just by opting in. With your idea of keeping multiple bitmaps active, we may need to tweak qemu to be more efficient at how it stores bitmap information (for example, instead of having each guest write modify the in-memory representation of all active bitmaps, which only get flushed to disk when closing the guest, we could instead have qemu store guest writes into a single in-memory bitmap, then merge that into all active bitmaps when flushing) - but that's for qemu to worry about. In the meantime, if your n ew proposed bitmap handling is easier to manage in libvirt, even if it costs more work in current qemu, I think we can live with that. I'm reluctant to ack this patch until I've seen the rest of the series adding support back in, but unless something major turns up in the rest of the series, the approach of a clean slate followed by a new implementation is reasonable enough. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
They will be replaced by a different set which will test scenarios relevant for the new semantics. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 3 - .../bitmap/snapshots-synthetic-broken.json | 837 ------------------ .../bitmap/snapshots-synthetic-broken.out | 14 - .../snapshots-synthetic-checkpoint.json | 827 ----------------- .../bitmap/snapshots-synthetic-checkpoint.out | 13 - tests/qemublocktestdata/bitmap/synthetic.json | 118 --- tests/qemublocktestdata/bitmap/synthetic.out | 6 - 7 files changed, 1818 deletions(-) delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out delete mode 100644 tests/qemublocktestdata/bitmap/synthetic.json delete mode 100644 tests/qemublocktestdata/bitmap/synthetic.out diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index d2f105486e..28cbaeba38 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1293,10 +1293,7 @@ mymain(void) TEST_BITMAP_DETECT("empty"); TEST_BITMAP_DETECT("basic"); - TEST_BITMAP_DETECT("synthetic"); TEST_BITMAP_DETECT("snapshots"); - TEST_BITMAP_DETECT("snapshots-synthetic-checkpoint"); - TEST_BITMAP_DETECT("snapshots-synthetic-broken"); #define TEST_BACKUP_BITMAP_CALCULATE(testname, source, incrbackup, named) \ do { \ diff --git a/tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json b/tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json deleted file mode 100644 index 8cf14d4baa..0000000000 --- a/tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json +++ /dev/null @@ -1,837 +0,0 @@ -[ - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911540", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "d", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "c", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911527", - "backing-filename": "/tmp/pull4.1575911527", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911550", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "in-use", - "auto" - ], - "name": "current", - "granularity": 65536 - }, - { - "flags": [ - "in-use" - ], - "name": "d", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911540", - "backing-filename": "/tmp/pull4.1575911540", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-format", - "backing_file_depth": 4, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911540", - "dirty-bitmaps": [ - { - "name": "d", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "current", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911550", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 393728, - "filename": "/tmp/pull4.1575911550", - "format": "file", - "actual-size": 212992, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911550", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911540", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "d", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "c", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911527", - "backing-filename": "/tmp/pull4.1575911527", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-2-format", - "backing_file_depth": 3, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911527", - "dirty-bitmaps": [ - { - "name": "c", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "oa", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - }, - { - "name": "ob", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - }, - { - "name": "d", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "inconsistent": true, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911540", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 393728, - "filename": "/tmp/pull4.1575911540", - "format": "file", - "actual-size": 212992, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-2-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911540", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-3-format", - "backing_file_depth": 2, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911522", - "dirty-bitmaps": [ - { - "name": "a", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "b", - "recording": true, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911527", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 459264, - "filename": "/tmp/pull4.1575911527", - "format": "file", - "actual-size": 217088, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-3-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911527", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-4-format", - "backing_file_depth": 1, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.qcow2", - "dirty-bitmaps": [ - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911522", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 328192, - "filename": "/tmp/pull4.1575911522", - "format": "file", - "actual-size": 208896, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-4-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911522", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-5-format", - "backing_file_depth": 0, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "dirty-bitmaps": [ - { - "name": "a", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 328192, - "filename": "/tmp/pull4.qcow2", - "format": "file", - "actual-size": 208896, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-5-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - } -] diff --git a/tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out b/tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out deleted file mode 100644 index ad24a580f1..0000000000 --- a/tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out +++ /dev/null @@ -1,14 +0,0 @@ -libvirt-1-format: - d: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -libvirt-2-format: - c: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - oa: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - ob: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - d: record:1 busy:0 persist:1 inconsist:1 gran:65536 dirty:0 -libvirt-3-format: - a: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -libvirt-4-format: -libvirt-5-format: - a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 diff --git a/tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json b/tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json deleted file mode 100644 index 25cc150d67..0000000000 --- a/tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json +++ /dev/null @@ -1,827 +0,0 @@ -[ - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911540", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "d", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "c", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911527", - "backing-filename": "/tmp/pull4.1575911527", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911550", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "in-use", - "auto" - ], - "name": "current", - "granularity": 65536 - }, - { - "flags": [ - "in-use" - ], - "name": "d", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911540", - "backing-filename": "/tmp/pull4.1575911540", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-format", - "backing_file_depth": 4, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911540", - "dirty-bitmaps": [ - { - "name": "d", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "current", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911550", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 393728, - "filename": "/tmp/pull4.1575911550", - "format": "file", - "actual-size": 212992, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911550", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911540", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "d", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "c", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911527", - "backing-filename": "/tmp/pull4.1575911527", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-2-format", - "backing_file_depth": 3, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911527", - "dirty-bitmaps": [ - { - "name": "c", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "d", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911540", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 393728, - "filename": "/tmp/pull4.1575911540", - "format": "file", - "actual-size": 212992, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-2-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911540", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-3-format", - "backing_file_depth": 2, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911522", - "dirty-bitmaps": [ - { - "name": "a", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "b", - "recording": true, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911527", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 459264, - "filename": "/tmp/pull4.1575911527", - "format": "file", - "actual-size": 217088, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-3-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911527", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-4-format", - "backing_file_depth": 1, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.qcow2", - "dirty-bitmaps": [ - { - "name": "a", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911522", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 328192, - "filename": "/tmp/pull4.1575911522", - "format": "file", - "actual-size": 208896, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-4-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911522", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-5-format", - "backing_file_depth": 0, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "dirty-bitmaps": [ - { - "name": "a", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 328192, - "filename": "/tmp/pull4.qcow2", - "format": "file", - "actual-size": 208896, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-5-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - } -] diff --git a/tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out b/tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out deleted file mode 100644 index 0270657001..0000000000 --- a/tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out +++ /dev/null @@ -1,13 +0,0 @@ -libvirt-1-format: - d: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -libvirt-2-format: - c: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -libvirt-3-format: - a: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -libvirt-4-format: - a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -libvirt-5-format: - a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 diff --git a/tests/qemublocktestdata/bitmap/synthetic.json b/tests/qemublocktestdata/bitmap/synthetic.json deleted file mode 100644 index 56882bd615..0000000000 --- a/tests/qemublocktestdata/bitmap/synthetic.json +++ /dev/null @@ -1,118 +0,0 @@ -[ - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 200704, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-format", - "backing_file_depth": 0, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "dirty-bitmaps": [ - { - "name": "current", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - }, - { - "name": "d", - "recording": false, - "persistent": false, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "c", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 1234, - "count": 0 - }, - { - "name": "b", - "recording": false, - "persistent": true, - "busy": true, - "status": "disabled", - "granularity": 65536, - "count": 21314 - }, - { - "name": "a", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "inconsistent": true, - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 197120, - "filename": "/tmp/pull4.qcow2", - "format": "file", - "actual-size": 200704, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - } -] diff --git a/tests/qemublocktestdata/bitmap/synthetic.out b/tests/qemublocktestdata/bitmap/synthetic.out deleted file mode 100644 index e8041c5b77..0000000000 --- a/tests/qemublocktestdata/bitmap/synthetic.out +++ /dev/null @@ -1,6 +0,0 @@ -libvirt-1-format: - current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - d: record:0 busy:0 persist:0 inconsist:0 gran:65536 dirty:0 - c: record:0 busy:0 persist:1 inconsist:0 gran:1234 dirty:0 - b: record:0 busy:1 persist:1 inconsist:0 gran:65536 dirty:21314 - a: record:0 busy:0 persist:1 inconsist:1 gran:65536 dirty:0 -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
They will be replaced by a different set which will test scenarios relevant for the new semantics.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 3 - .../bitmap/snapshots-synthetic-broken.json | 837 ------------------ .../bitmap/snapshots-synthetic-broken.out | 14 - .../snapshots-synthetic-checkpoint.json | 827 ----------------- .../bitmap/snapshots-synthetic-checkpoint.out | 13 - tests/qemublocktestdata/bitmap/synthetic.json | 118 --- tests/qemublocktestdata/bitmap/synthetic.out | 6 - 7 files changed, 1818 deletions(-)
Same comment as for the previous patch - if we get to the end of the series without major issues, then this temporary deletion to make the rewrite easier is fine. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
There will be multiple places where we'll need to print nodenames from a GSList of virStorageSource for testing purposes. Extract the code into a function. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-) diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 28cbaeba38..0ed9b99bc4 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -647,6 +647,23 @@ testQemuDetectBitmaps(const void *opaque) } +static void +testQemuBitmapListPrint(const char *title, + GSList *next, + virBufferPtr buf) +{ + if (!next) + return; + + virBufferAsprintf(buf, "%s\n", title); + + for (; next; next = next->next) { + virStorageSourcePtr src = next->data; + virBufferAsprintf(buf, "%s\n", src->nodeformat); + } +} + + static virStorageSourcePtr testQemuBackupIncrementalBitmapCalculateGetFakeImage(size_t idx) { @@ -829,7 +846,6 @@ testQemuCheckpointDeleteMerge(const void *opaque) g_autoptr(virHashTable) nodedata = NULL; g_autoptr(GSList) reopenimages = NULL; g_auto(virBuffer) buf = VIR_BUFFER_INITIALIZER; - GSList *tmp; expectpath = g_strdup_printf("%s/%s%s-out.json", abs_srcdir, checkpointDeletePrefix, data->name); @@ -858,14 +874,7 @@ testQemuCheckpointDeleteMerge(const void *opaque) virBufferAddLit(&buf, "NULL\n"); } - if (reopenimages) { - virBufferAddLit(&buf, "reopen nodes:\n"); - - for (tmp = reopenimages; tmp; tmp = tmp->next) { - virStorageSourcePtr src = tmp->data; - virBufferAsprintf(&buf, "%s\n", src->nodeformat); - } - } + testQemuBitmapListPrint("reopen nodes:", reopenimages, &buf); actual = virBufferContentAndReset(&buf); -- 2.26.2
On 6/15/20 12:09 PM, Peter Krempa wrote:
There will be multiple places where we'll need to print nodenames from a GSList of virStorageSource for testing purposes. Extract the code into a function.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 27 ++++++++++++++++++--------- 1 file changed, 18 insertions(+), 9 deletions(-)
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Chaining bitmaps for checkpoints (disabling the active one and creating a new) severely overcomplicated all operations in regards to bitmaps. Specifically it requires us re-matching the on-disk state to the internal metadata and in case of merging during block jobs it makes it almost impossible to cover all corner cases. Since the checkpoints and incremental backups were not yet enabled, let's change the design to keep one bitmap per checkpoint. In case of layered snapshots this will be filled in by using dirty-bitmap-populate. Finally the main reason for this unnecessary complexity was the fear that qemu's performance could degrade. In the end I think that addressing the performance issue will be better done in qemu (e.g by keeping an internal bitmap updated with changes and merging it periodically back to the real bitmaps. QEMU writes out changes to disk at shutdown so consistency is not a problem). Removing the relationships between bitmaps frees us from complex handling and also makes all the surrounding code more robust as one broken bitmap doesn't necessarily invalidate whole chains of backups. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_checkpoint.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-) diff --git a/src/qemu/qemu_checkpoint.c b/src/qemu/qemu_checkpoint.c index b71b4a7d14..9b4f3ad396 100644 --- a/src/qemu/qemu_checkpoint.c +++ b/src/qemu/qemu_checkpoint.c @@ -456,7 +456,6 @@ qemuCheckpointPrepare(virQEMUDriverPtr driver, static int qemuCheckpointAddActions(virDomainObjPtr vm, virJSONValuePtr actions, - virDomainMomentObjPtr old_current, virDomainCheckpointDefPtr def) { size_t i; @@ -464,7 +463,6 @@ qemuCheckpointAddActions(virDomainObjPtr vm, for (i = 0; i < def->ndisks; i++) { virDomainCheckpointDiskDef *chkdisk = &def->disks[i]; virDomainDiskDefPtr domdisk = virDomainDiskByTarget(vm->def, chkdisk->name); - virDomainCheckpointDiskDef *parentchkdisk = NULL; /* checkpoint definition validator mandates that the corresponding * domdisk should exist */ @@ -475,23 +473,6 @@ qemuCheckpointAddActions(virDomainObjPtr vm, if (qemuMonitorTransactionBitmapAdd(actions, domdisk->src->nodeformat, chkdisk->bitmap, true, false, 0) < 0) return -1; - - /* We only want one active bitmap for a disk along the - * checkpoint chain, then later differential backups will - * merge the bitmaps (only one active) between the bounding - * checkpoint and the leaf checkpoint. If the same disks are - * involved in each checkpoint, this search terminates in one - * iteration; but it is also possible to have to search - * further than the immediate parent to find another - * checkpoint with a bitmap on the same disk. */ - if ((parentchkdisk = qemuCheckpointFindActiveDiskInParent(vm, old_current, - chkdisk->name))) { - - if (qemuMonitorTransactionBitmapDisable(actions, - domdisk->src->nodeformat, - parentchkdisk->bitmap) < 0) - return -1; - } } return 0; } @@ -540,7 +521,7 @@ qemuCheckpointCreateCommon(virQEMUDriverPtr driver, tmpactions = virJSONValueNewArray(); - if (qemuCheckpointAddActions(vm, tmpactions, parent, *def) < 0) + if (qemuCheckpointAddActions(vm, tmpactions, *def) < 0) return -1; if (!(*chk = virDomainCheckpointAssignDef(vm->checkpoints, *def))) -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Chaining bitmaps for checkpoints (disabling the active one and creating a new) severely overcomplicated all operations in regards to bitmaps.
Specifically it requires us re-matching the on-disk state to the internal metadata and in case of merging during block jobs it makes it almost impossible to cover all corner cases.
Since the checkpoints and incremental backups were not yet enabled, let's change the design to keep one bitmap per checkpoint. In case of layered snapshots this will be filled in by using dirty-bitmap-populate.
Finally the main reason for this unnecessary complexity was the fear that qemu's performance could degrade. In the end I think that addressing the performance issue will be better done in qemu (e.g by keeping an internal bitmap updated with changes and merging it periodically back to the real bitmaps. QEMU writes out changes to disk at shutdown so consistency is not a problem).
Removing the relationships between bitmaps frees us from complex handling and also makes all the surrounding code more robust as one broken bitmap doesn't necessarily invalidate whole chains of backups.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_checkpoint.c | 21 +-------------------- 1 file changed, 1 insertion(+), 20 deletions(-)
Definitely looks simpler. In graphics, pre-patch we had: time --C1-----C2-----C3-----now :b1....: :b2....: :b3.... where the differential from C1 is b1+b2+b3, the differential from C2 is b2+b3, the incremental from C3 is b3. post-patch, we have: time --C1-----C2-----C3-----now :b1................... :b2............ :b3.... where the differential from C1 is b1, the differential from C2 is b2, the incremental from C3 is b3. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Use test data which conforms to the new semantics which changed in the previous patch. The test data was created by the same set of commands as originally in commit 9aac9d5bdab039a50de2d8c627b3a1f1578ed471 Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktestdata/bitmap/basic.json | 229 +++++++++++----------- tests/qemublocktestdata/bitmap/basic.out | 8 +- 2 files changed, 119 insertions(+), 118 deletions(-) diff --git a/tests/qemublocktestdata/bitmap/basic.json b/tests/qemublocktestdata/bitmap/basic.json index 9d418b1a37..718106bf99 100644 --- a/tests/qemublocktestdata/bitmap/basic.json +++ b/tests/qemublocktestdata/bitmap/basic.json @@ -1,117 +1,118 @@ [ - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 200704, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-format", - "backing_file_depth": 0, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "dirty-bitmaps": [ - { - "name": "current", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - }, - { - "name": "d", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "c", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "b", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - }, - { - "name": "a", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 - } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 10485760, + "filename": "/tmp/bitmaps.qcow2", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 200704, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "refcount-bits": 16, + "corrupt": false + } + }, + "dirty-flag": false }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 197120, - "filename": "/tmp/pull4.qcow2", - "format": "file", - "actual-size": 200704, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - } + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "dirty-bitmaps": [ + { + "name": "current", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "d", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "c", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "b", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "a", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.qcow2", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 197120, + "filename": "/tmp/bitmaps.qcow2", + "format": "file", + "actual-size": 200704, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.qcow2", + "encryption_key_missing": false + } ] diff --git a/tests/qemublocktestdata/bitmap/basic.out b/tests/qemublocktestdata/bitmap/basic.out index 539f7d9bad..5c4c35b3f0 100644 --- a/tests/qemublocktestdata/bitmap/basic.out +++ b/tests/qemublocktestdata/bitmap/basic.out @@ -1,6 +1,6 @@ libvirt-1-format: current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - d: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - c: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - b: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - a: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + c: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Use test data which conforms to the new semantics which changed in the previous patch.
The test data was created by the same set of commands as originally in commit 9aac9d5bdab039a50de2d8c627b3a1f1578ed471
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktestdata/bitmap/basic.json | 229 +++++++++++----------- tests/qemublocktestdata/bitmap/basic.out | 8 +- 2 files changed, 119 insertions(+), 118 deletions(-)
- "dirty-bitmaps": [ - { - "name": "current", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 - }, - { - "name": "d", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0
The old code showed "recording":false for past bitmaps;
+ "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib",
The new output is built with a newer qemu that understands different compression types (doesn't change correctness of the patch), and...
+ "dirty-bitmaps": [ + { + "name": "current", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "d", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0
...shows that even the past bitmaps are still active. If qemu has performance problems with multiple active bitmaps, I agree that addressing that in qemu is reasonable.
+++ b/tests/qemublocktestdata/bitmap/basic.out @@ -1,6 +1,6 @@ libvirt-1-format: current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - d: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - c: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - b: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - a: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + c: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0
And this reflects that libvirt is parsing the qemu output according to libvirt's new paradigm. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Use test data which conforms to the new semantics which changed in the previous patch. The test data was created by the same set of commands as originally in commit 0b27b655b1bac480186ce80457113cd5dc34e6a1 Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++---------- tests/qemublocktestdata/bitmap/snapshots.out | 13 +- 2 files changed, 588 insertions(+), 807 deletions(-) diff --git a/tests/qemublocktestdata/bitmap/snapshots.json b/tests/qemublocktestdata/bitmap/snapshots.json index 87e77ad408..2a20aea812 100644 --- a/tests/qemublocktestdata/bitmap/snapshots.json +++ b/tests/qemublocktestdata/bitmap/snapshots.json @@ -1,836 +1,612 @@ [ - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911540", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "d", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "c", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911527", - "backing-filename": "/tmp/pull4.1575911527", - "dirty-flag": false + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 104857600, + "filename": "/tmp/bitmaps.1591792950", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 225280, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ + { + "flags": [ + "in-use", + "auto" + ], + "name": "current", + "granularity": 65536 + }, + { + "flags": [ + "in-use", + "auto" + ], + "name": "c", + "granularity": 65536 }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911550", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "in-use", - "auto" - ], - "name": "current", - "granularity": 65536 - }, - { - "flags": [ - "in-use" - ], - "name": "d", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } + { + "flags": [ + "in-use", + "auto" + ], + "name": "b", + "granularity": 65536 }, - "full-backing-filename": "/tmp/pull4.1575911540", - "backing-filename": "/tmp/pull4.1575911540", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-format", - "backing_file_depth": 4, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911540", - "dirty-bitmaps": [ { - "name": "d", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 + "flags": [ + "in-use", + "auto" + ], + "name": "a", + "granularity": 65536 }, { - "name": "current", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 + "flags": [ + "in-use", + "auto" + ], + "name": "d", + "granularity": 65536 } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911550", - "encryption_key_missing": false + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.1591792938", + "backing-filename": "/tmp/bitmaps.1591792938", + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.1591792938", + "dirty-bitmaps": [ + { + "name": "d", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "a", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "b", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "c", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "current", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1591792950", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 590336, + "filename": "/tmp/bitmaps.1591792950", + "format": "file", + "actual-size": 225280, + "dirty-flag": false }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 393728, - "filename": "/tmp/pull4.1575911550", - "format": "file", - "actual-size": 212992, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-1-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911550", - "encryption_key_missing": false + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false + "file": "/tmp/bitmaps.1591792950", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 104857600, + "filename": "/tmp/bitmaps.1591792938", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 221184, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ + { + "flags": [ + "auto" + ], + "name": "d", + "granularity": 65536 }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911540", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 212992, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "d", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "c", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } + { + "flags": [ + "auto" + ], + "name": "a", + "granularity": 65536 }, - "full-backing-filename": "/tmp/pull4.1575911527", - "backing-filename": "/tmp/pull4.1575911527", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-2-format", - "backing_file_depth": 3, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911527", - "dirty-bitmaps": [ { - "name": "c", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 + "flags": [ + "auto" + ], + "name": "b", + "granularity": 65536 }, { - "name": "d", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 + "flags": [ + "auto" + ], + "name": "c", + "granularity": 65536 } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911540", - "encryption_key_missing": false + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.1591792930", + "backing-filename": "/tmp/bitmaps.1591792930", + "dirty-flag": false }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 393728, - "filename": "/tmp/pull4.1575911540", - "format": "file", - "actual-size": 212992, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-2-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911540", - "encryption_key_missing": false + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-2-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.1591792930", + "dirty-bitmaps": [ + { + "name": "c", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "b", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "a", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "d", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911527", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 217088, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "c", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "b", - "granularity": 65536 - }, - { - "flags": [ - - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.1575911522", - "backing-filename": "/tmp/pull4.1575911522", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-3-format", - "backing_file_depth": 2, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.1575911522", - "dirty-bitmaps": [ + "file": "/tmp/bitmaps.1591792938", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 524800, + "filename": "/tmp/bitmaps.1591792938", + "format": "file", + "actual-size": 221184, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-2-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1591792938", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 104857600, + "filename": "/tmp/bitmaps.1591792930", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 217088, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ { - "name": "a", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 + "flags": [ + "auto" + ], + "name": "c", + "granularity": 65536 }, { - "name": "b", - "recording": false, - "persistent": true, - "busy": false, - "status": "disabled", - "granularity": 65536, - "count": 0 + "flags": [ + "auto" + ], + "name": "b", + "granularity": 65536 }, { - "name": "c", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 + "flags": [ + "auto" + ], + "name": "a", + "granularity": 65536 } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911527", - "encryption_key_missing": false + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.1591792925", + "backing-filename": "/tmp/bitmaps.1591792925", + "dirty-flag": false }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 459264, - "filename": "/tmp/pull4.1575911527", - "format": "file", - "actual-size": 217088, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-3-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911527", - "encryption_key_missing": false + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-3-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.1591792925", + "dirty-bitmaps": [ + { + "name": "a", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "b", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "c", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "backing-image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "backing-filename-format": "qcow2", - "virtual-size": 10485760, - "filename": "/tmp/pull4.1575911522", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "full-backing-filename": "/tmp/pull4.qcow2", - "backing-filename": "/tmp/pull4.qcow2", - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-4-format", - "backing_file_depth": 1, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "backing_file": "/tmp/pull4.qcow2", - "dirty-bitmaps": [ + "file": "/tmp/bitmaps.1591792930", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 459264, + "filename": "/tmp/bitmaps.1591792930", + "format": "file", + "actual-size": 217088, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-3-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1591792930", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 104857600, + "filename": "/tmp/bitmaps.1591792925", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 208896, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ { - "name": "a", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 + "flags": [ + "auto" + ], + "name": "a", + "granularity": 65536 } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911522", - "encryption_key_missing": false + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.qcow2", + "backing-filename": "/tmp/bitmaps.qcow2", + "dirty-flag": false }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 328192, - "filename": "/tmp/pull4.1575911522", - "format": "file", - "actual-size": 208896, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-4-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.1575911522", - "encryption_key_missing": false + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-4-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.qcow2", + "dirty-bitmaps": [ + { + "name": "a", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 10485760, - "filename": "/tmp/pull4.qcow2", - "cluster-size": 65536, - "format": "qcow2", - "actual-size": 208896, - "format-specific": { - "type": "qcow2", - "data": { - "compat": "1.1", - "lazy-refcounts": false, - "bitmaps": [ - { - "flags": [ - "auto" - ], - "name": "a", - "granularity": 65536 - } - ], - "refcount-bits": 16, - "corrupt": false - } - }, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": true, - "node-name": "libvirt-5-format", - "backing_file_depth": 0, - "drv": "qcow2", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "dirty-bitmaps": [ + "file": "/tmp/bitmaps.1591792925", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 328192, + "filename": "/tmp/bitmaps.1591792925", + "format": "file", + "actual-size": 208896, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-4-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1591792925", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 104857600, + "filename": "/tmp/bitmaps.qcow2", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 208896, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ { - "name": "a", - "recording": true, - "persistent": true, - "busy": false, - "status": "active", - "granularity": 65536, - "count": 0 + "flags": [ + "auto" + ], + "name": "a", + "granularity": 65536 } - ], - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-5-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "dirty-bitmaps": [ + { + "name": "a", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.qcow2", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 328192, + "filename": "/tmp/bitmaps.qcow2", + "format": "file", + "actual-size": 208896, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-5-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true }, - { - "iops_rd": 0, - "detect_zeroes": "off", - "image": { - "virtual-size": 328192, - "filename": "/tmp/pull4.qcow2", - "format": "file", - "actual-size": 208896, - "dirty-flag": false - }, - "iops_wr": 0, - "ro": false, - "node-name": "libvirt-5-storage", - "backing_file_depth": 0, - "drv": "file", - "iops": 0, - "bps_wr": 0, - "write_threshold": 0, - "encrypted": false, - "bps": 0, - "bps_rd": 0, - "cache": { - "no-flush": false, - "direct": false, - "writeback": true - }, - "file": "/tmp/pull4.qcow2", - "encryption_key_missing": false - } + "file": "/tmp/bitmaps.qcow2", + "encryption_key_missing": false + } ] diff --git a/tests/qemublocktestdata/bitmap/snapshots.out b/tests/qemublocktestdata/bitmap/snapshots.out index 5dafe946cf..24ca27e4d8 100644 --- a/tests/qemublocktestdata/bitmap/snapshots.out +++ b/tests/qemublocktestdata/bitmap/snapshots.out @@ -1,12 +1,17 @@ libvirt-1-format: - d: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + c: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 libvirt-2-format: - c: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + c: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 libvirt-3-format: - a: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 - b: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 c: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 libvirt-4-format: a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Use test data which conforms to the new semantics which changed in the previous patch.
Well, a recent patch (the previous patch only touched tests, rather than changing semantics)
The test data was created by the same set of commands as originally in commit 0b27b655b1bac480186ce80457113cd5dc34e6a1
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++---------- tests/qemublocktestdata/bitmap/snapshots.out | 13 +- 2 files changed, 588 insertions(+), 807 deletions(-)
+++ b/tests/qemublocktestdata/bitmap/snapshots.out @@ -1,12 +1,17 @@ libvirt-1-format: - d: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + c: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0
So, where the old paradigm stitched bitmaps across snapshots (no need to add a bitmap to a new snapshot layer if it was already inactive in the old layer), the new paradigm says that every reachable checkpoint must have an active bitmap in the current layer, in addition to whatever bitmaps it had in earlier layers. Presumably, upcoming patches will tweak libvirt to actually create these bitmaps as part of creating an external snapshot, and I also see how you could utilize qemu adding block-dirty-bitmap-populate as a convenient way for re-creating such bitmaps after the fact (any bitmap that exists in the backing chain should basically mirror the allocation of the current layer in the chain). Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Thu, Jun 18, 2020 at 09:58:35 -0500, Eric Blake wrote:
On 6/15/20 12:10 PM, Peter Krempa wrote:
Use test data which conforms to the new semantics which changed in the previous patch.
Well, a recent patch (the previous patch only touched tests, rather than changing semantics)
The test data was created by the same set of commands as originally in commit 0b27b655b1bac480186ce80457113cd5dc34e6a1
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++---------- tests/qemublocktestdata/bitmap/snapshots.out | 13 +- 2 files changed, 588 insertions(+), 807 deletions(-)
+++ b/tests/qemublocktestdata/bitmap/snapshots.out @@ -1,12 +1,17 @@ libvirt-1-format: - d: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + b: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + c: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 current: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0
So, where the old paradigm stitched bitmaps across snapshots (no need to add a bitmap to a new snapshot layer if it was already inactive in the old layer), the new paradigm says that every reachable checkpoint must have an active bitmap in the current layer, in addition to whatever bitmaps it had in earlier layers.
Presumably, upcoming patches will tweak libvirt to actually create these bitmaps as part of creating an external snapshot, and I also see how you could utilize qemu adding block-dirty-bitmap-populate as a convenient way for re-creating such bitmaps after the fact (any bitmap that exists in the backing chain should basically mirror the allocation of the current layer in the chain).
They are actually already being created by 'qemuDomainSnapshotDiskBitmapsPropagate', and that code is still valid. But I've noticed that the comment needs to be fixed probably.
Reject duplicates and other problematic bitmaps according to the new semantics of bitmap use in libvirt. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-) diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 6f9c7071c9..83e3df9601 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -2850,41 +2850,49 @@ qemuBlockGetNamedNodeData(virDomainObjPtr vm, /** * qemuBlockBitmapChainIsValid: * - * Validates that the backing chain of @src contains proper consistent bitmap - * data for a chain of bitmaps named @bitmapname. + * Validates that the backing chain of @src contains bitmaps which libvirt will + * consider as properly corresponding to a checkpoint named @bitmapname. * - * A valid chain: - * 1) bitmaps of same name are in a consecutive subset of images without gap - * 2) don't have any inconsistent bitmaps + * The bitmaps need to: + * 1) start from the top image @src + * 2) must be present in consecutive layers + * 3) all must be active, persistent and not incosistent */ bool qemuBlockBitmapChainIsValid(virStorageSourcePtr src, const char *bitmapname, virHashTablePtr blockNamedNodeData) { - qemuBlockNamedNodeDataBitmapPtr bitmap; virStorageSourcePtr n; - bool chain_started = false; + bool found = false; bool chain_ended = false; - for (n = src; n; n = n->backingStore) { - if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, n, bitmapname))) { - if (chain_started) - chain_ended = true; + for (n = src; virStorageSourceIsBacking(n); n = n->backingStore) { + qemuBlockNamedNodeDataBitmapPtr bitmap; + + if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + n, bitmapname))) { + /* rule 1, must start from top */ + if (!found) + return false; + + chain_ended = true; continue; } + /* rule 2, no-gaps */ if (chain_ended) return false; - chain_started = true; - - if (bitmap->inconsistent) + /* rule 3 */ + if (bitmap->inconsistent || !bitmap->persistent || !bitmap->recording) return false; + + found = true; } - return chain_started; + return found; } -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Reject duplicates and other problematic bitmaps according to the new semantics of bitmap use in libvirt.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 38 +++++++++++++++++++++++--------------- 1 file changed, 23 insertions(+), 15 deletions(-)
diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 6f9c7071c9..83e3df9601 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -2850,41 +2850,49 @@ qemuBlockGetNamedNodeData(virDomainObjPtr vm, /** * qemuBlockBitmapChainIsValid: * - * Validates that the backing chain of @src contains proper consistent bitmap - * data for a chain of bitmaps named @bitmapname. + * Validates that the backing chain of @src contains bitmaps which libvirt will + * consider as properly corresponding to a checkpoint named @bitmapname. * - * A valid chain: - * 1) bitmaps of same name are in a consecutive subset of images without gap - * 2) don't have any inconsistent bitmaps + * The bitmaps need to: + * 1) start from the top image @src + * 2) must be present in consecutive layers + * 3) all must be active, persistent and not incosistent
inconsistent Otherwise, Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Now that we've updated both the test data and the validator to new semantics we can start testing again. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 0ed9b99bc4..6d1bf3f250 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1346,6 +1346,18 @@ mymain(void) TEST_BITMAP_VALIDATE("empty", "a", false); + TEST_BITMAP_VALIDATE("basic", "a", true); + TEST_BITMAP_VALIDATE("basic", "b", true); + TEST_BITMAP_VALIDATE("basic", "c", true); + TEST_BITMAP_VALIDATE("basic", "d", true); + TEST_BITMAP_VALIDATE("basic", "current", true); + + TEST_BITMAP_VALIDATE("snapshots", "a", true); + TEST_BITMAP_VALIDATE("snapshots", "b", true); + TEST_BITMAP_VALIDATE("snapshots", "c", true); + TEST_BITMAP_VALIDATE("snapshots", "d", true); + TEST_BITMAP_VALIDATE("snapshots", "current", true); + #define TEST_BITMAP_BLOCKCOPY(testname, shllw, ndf) \ do { \ blockbitmapblockcopydata.name = testname; \ -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Now that we've updated both the test data and the validator to new semantics we can start testing again.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 12 ++++++++++++ 1 file changed, 12 insertions(+)
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Based on the 'snapshots' example with manual tweaks to introduce inactive, transient, inconsistent and duplicate bitmaps in various parts of the chain to excercise detection and new validation code. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 11 + tests/qemublocktestdata/bitmap/synthetic.json | 506 ++++++++++++++++++ tests/qemublocktestdata/bitmap/synthetic.out | 15 + 3 files changed, 532 insertions(+) create mode 100644 tests/qemublocktestdata/bitmap/synthetic.json create mode 100644 tests/qemublocktestdata/bitmap/synthetic.out diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 6d1bf3f250..47e9f0f363 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1303,6 +1303,7 @@ mymain(void) TEST_BITMAP_DETECT("basic"); TEST_BITMAP_DETECT("snapshots"); + TEST_BITMAP_DETECT("synthetic"); #define TEST_BACKUP_BITMAP_CALCULATE(testname, source, incrbackup, named) \ do { \ @@ -1358,6 +1359,16 @@ mymain(void) TEST_BITMAP_VALIDATE("snapshots", "d", true); TEST_BITMAP_VALIDATE("snapshots", "current", true); + TEST_BITMAP_VALIDATE("synthetic", "a", false); + TEST_BITMAP_VALIDATE("synthetic", "b", false); + TEST_BITMAP_VALIDATE("synthetic", "c", false); + TEST_BITMAP_VALIDATE("synthetic", "d", false); + TEST_BITMAP_VALIDATE("synthetic", "current", false); + TEST_BITMAP_VALIDATE("synthetic", "top-ok", true); + TEST_BITMAP_VALIDATE("synthetic", "top-inactive", false); + TEST_BITMAP_VALIDATE("synthetic", "top-transient", false); + TEST_BITMAP_VALIDATE("synthetic", "top-inactive-transient", false); + #define TEST_BITMAP_BLOCKCOPY(testname, shllw, ndf) \ do { \ blockbitmapblockcopydata.name = testname; \ diff --git a/tests/qemublocktestdata/bitmap/synthetic.json b/tests/qemublocktestdata/bitmap/synthetic.json new file mode 100644 index 0000000000..3712c8e5fc --- /dev/null +++ b/tests/qemublocktestdata/bitmap/synthetic.json @@ -0,0 +1,506 @@ +[ + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 10485760, + "filename": "/tmp/bitmaps.1590749073", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 208896, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ + { + "flags": [ + "in-use", + "auto" + ], + "name": "current", + "granularity": 65536 + } + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.1590749012", + "backing-filename": "/tmp/bitmaps.1590749012", + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.1590749012", + "dirty-bitmaps": [ + { + "name": "current", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "inconsistent": true, + "granularity": 65536, + "count": 0 + }, + { + "name": "top-ok", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "top-inactive", + "recording": false, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "top-transient", + "recording": true, + "persistent": false, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "top-transient-inactive", + "recording": false, + "persistent": false, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590749073", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 328192, + "filename": "/tmp/bitmaps.1590749073", + "format": "file", + "actual-size": 208896, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-1-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590749073", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 10485760, + "filename": "/tmp/bitmaps.1590749012", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 208896, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ + { + "flags": [ + "auto" + ], + "name": "d", + "granularity": 65536 + } + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.1590748995", + "backing-filename": "/tmp/bitmaps.1590748995", + "dirty-flag": false + }, + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-2-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.1590748995", + "dirty-bitmaps": [ + { + "name": "d", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590749012", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 328192, + "filename": "/tmp/bitmaps.1590749012", + "format": "file", + "actual-size": 208896, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-2-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590749012", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 10485760, + "filename": "/tmp/bitmaps.1590748995", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 212992, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ + { + "flags": [ + "auto" + ], + "name": "c", + "granularity": 65536 + }, + { + "flags": [ + "auto" + ], + "name": "b", + "granularity": 65536 + } + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.1590748992", + "backing-filename": "/tmp/bitmaps.1590748992", + "dirty-flag": false + }, + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-3-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.1590748992", + "dirty-bitmaps": [ + { + "name": "b", + "recording": true, + "persistent": false, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "c", + "recording": false, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + }, + { + "name": "d", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590748995", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 393728, + "filename": "/tmp/bitmaps.1590748995", + "format": "file", + "actual-size": 212992, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-3-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590748995", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "backing-filename-format": "qcow2", + "virtual-size": 10485760, + "filename": "/tmp/bitmaps.1590748992", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 200704, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "refcount-bits": 16, + "corrupt": false + } + }, + "full-backing-filename": "/tmp/bitmaps.qcow2", + "backing-filename": "/tmp/bitmaps.qcow2", + "dirty-flag": false + }, + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-4-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "backing_file": "/tmp/bitmaps.qcow2", + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590748992", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 197120, + "filename": "/tmp/bitmaps.1590748992", + "format": "file", + "actual-size": 200704, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-4-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.1590748992", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 10485760, + "filename": "/tmp/bitmaps.qcow2", + "cluster-size": 65536, + "format": "qcow2", + "actual-size": 208896, + "format-specific": { + "type": "qcow2", + "data": { + "compat": "1.1", + "compression-type": "zlib", + "lazy-refcounts": false, + "bitmaps": [ + { + "flags": [ + "auto" + ], + "name": "a", + "granularity": 65536 + } + ], + "refcount-bits": 16, + "corrupt": false + } + }, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": true, + "node-name": "libvirt-5-format", + "backing_file_depth": 0, + "drv": "qcow2", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "dirty-bitmaps": [ + { + "name": "a", + "recording": true, + "persistent": true, + "busy": false, + "status": "active", + "granularity": 65536, + "count": 0 + } + ], + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.qcow2", + "encryption_key_missing": false + }, + { + "iops_rd": 0, + "detect_zeroes": "off", + "image": { + "virtual-size": 328192, + "filename": "/tmp/bitmaps.qcow2", + "format": "file", + "actual-size": 208896, + "dirty-flag": false + }, + "iops_wr": 0, + "ro": false, + "node-name": "libvirt-5-storage", + "backing_file_depth": 0, + "drv": "file", + "iops": 0, + "bps_wr": 0, + "write_threshold": 0, + "encrypted": false, + "bps": 0, + "bps_rd": 0, + "cache": { + "no-flush": false, + "direct": false, + "writeback": true + }, + "file": "/tmp/bitmaps.qcow2", + "encryption_key_missing": false + } +] diff --git a/tests/qemublocktestdata/bitmap/synthetic.out b/tests/qemublocktestdata/bitmap/synthetic.out new file mode 100644 index 0000000000..cde7228e01 --- /dev/null +++ b/tests/qemublocktestdata/bitmap/synthetic.out @@ -0,0 +1,15 @@ +libvirt-1-format: + current: record:1 busy:0 persist:1 inconsist:1 gran:65536 dirty:0 + top-ok: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + top-inactive: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + top-transient: record:1 busy:0 persist:0 inconsist:0 gran:65536 dirty:0 + top-transient-inactive: record:0 busy:0 persist:0 inconsist:0 gran:65536 dirty:0 +libvirt-2-format: + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 +libvirt-3-format: + b: record:1 busy:0 persist:0 inconsist:0 gran:65536 dirty:0 + c: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 +libvirt-4-format: +libvirt-5-format: + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Based on the 'snapshots' example with manual tweaks to introduce inactive, transient, inconsistent and duplicate bitmaps in various parts of the chain to excercise detection and new validation code.
exercise
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 11 + tests/qemublocktestdata/bitmap/synthetic.json | 506 ++++++++++++++++++ tests/qemublocktestdata/bitmap/synthetic.out | 15 + 3 files changed, 532 insertions(+) create mode 100644 tests/qemublocktestdata/bitmap/synthetic.json create mode 100644 tests/qemublocktestdata/bitmap/synthetic.out
+++ b/tests/qemublocktestdata/bitmap/synthetic.out @@ -0,0 +1,15 @@ +libvirt-1-format: + current: record:1 busy:0 persist:1 inconsist:1 gran:65536 dirty:0 + top-ok: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + top-inactive: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + top-transient: record:1 busy:0 persist:0 inconsist:0 gran:65536 dirty:0 + top-transient-inactive: record:0 busy:0 persist:0 inconsist:0 gran:65536 dirty:0 +libvirt-2-format: + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 +libvirt-3-format: + b: record:1 busy:0 persist:0 inconsist:0 gran:65536 dirty:0 + c: record:0 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 + d: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0 +libvirt-4-format: +libvirt-5-format: + a: record:1 busy:0 persist:1 inconsist:0 gran:65536 dirty:0
Looks like a pretty decent set of awkward setups; libvirt should not ever produce these, but being robust to whatever happened to the qcow2 files externally to libvirt is worthwhile. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Now that we've switched to the simple handling, the first thing that can be massively simplified is checkpoint deletion. We now need to only go through the backing chain and find the appropriately named bitmaps and delete them, no complex lookups or merging. Note that compared to other functions this deletes the bitmap in all layers compared to others where we expect only exactly 1 bitmap of a name in the backing chain to prevent potential problems. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_checkpoint.c | 153 ++++++------------------------------- src/qemu/qemu_checkpoint.h | 1 - tests/qemublocktest.c | 7 +- 3 files changed, 25 insertions(+), 136 deletions(-) diff --git a/src/qemu/qemu_checkpoint.c b/src/qemu/qemu_checkpoint.c index 9b4f3ad396..c24d97443c 100644 --- a/src/qemu/qemu_checkpoint.c +++ b/src/qemu/qemu_checkpoint.c @@ -105,140 +105,41 @@ qemuCheckpointWriteMetadata(virDomainObjPtr vm, } -/** - * qemuCheckpointFindActiveDiskInParent: - * @vm: domain object - * @from: starting moment object - * @diskname: name (target) of the disk to find - * - * Find the first checkpoint starting from @from continuing through parents - * of the checkpoint which describes disk @diskname. Return the pointer to the - * definition of the disk. - */ -static virDomainCheckpointDiskDef * -qemuCheckpointFindActiveDiskInParent(virDomainObjPtr vm, - virDomainMomentObjPtr from, - const char *diskname) -{ - virDomainMomentObjPtr parent = from; - virDomainCheckpointDefPtr parentdef = NULL; - size_t i; - - while (parent) { - parentdef = virDomainCheckpointObjGetDef(parent); - - for (i = 0; i < parentdef->ndisks; i++) { - virDomainCheckpointDiskDef *chkdisk = &parentdef->disks[i]; - - if (STRNEQ(chkdisk->name, diskname)) - continue; - - /* currently inspected checkpoint doesn't describe the disk, - * continue into parent checkpoint */ - if (chkdisk->type != VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP) - break; - - return chkdisk; - } - - parent = virDomainCheckpointFindByName(vm->checkpoints, - parentdef->parent.parent_name); - } - - return NULL; -} - - int qemuCheckpointDiscardDiskBitmaps(virStorageSourcePtr src, virHashTablePtr blockNamedNodeData, const char *delbitmap, - const char *parentbitmap, virJSONValuePtr actions, const char *diskdst, GSList **reopenimages) { - virStorageSourcePtr n = src; + virStorageSourcePtr n; + bool found = false; /* find the backing chain entry with bitmap named '@delbitmap' */ - while (n) { - qemuBlockNamedNodeDataBitmapPtr tmp; - - if ((tmp = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - n, delbitmap))) { - break; - } - - n = n->backingStore; - } - - if (!n) { - virReportError(VIR_ERR_INTERNAL_ERROR, - _("bitmap '%s' not found in backing chain of '%s'"), - delbitmap, diskdst); - return -1; - } - - while (n) { - qemuBlockNamedNodeDataBitmapPtr srcbitmap; - - if (!(srcbitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - n, delbitmap))) - break; - - /* For the actual checkpoint deletion we will merge any bitmap into the - * bitmap of the parent checkpoint (@parentbitmap) or for any image - * where the parent checkpoint bitmap is not present we must rename - * the bitmap of the deleted checkpoint into the bitmap of the parent - * checkpoint as qemu can't currently take the allocation map and turn - * it into a bitmap and thus we wouldn't be able to do a backup. */ - if (parentbitmap) { - qemuBlockNamedNodeDataBitmapPtr dstbitmap; - g_autoptr(virJSONValue) arr = NULL; - - dstbitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - n, parentbitmap); - - if (dstbitmap) { - if (srcbitmap->recording && !dstbitmap->recording) { - if (qemuMonitorTransactionBitmapEnable(actions, - n->nodeformat, - dstbitmap->name) < 0) - return -1; - } - - } else { - if (qemuMonitorTransactionBitmapAdd(actions, - n->nodeformat, - parentbitmap, - true, - !srcbitmap->recording, - srcbitmap->granularity) < 0) - return -1; - } + for (n = src; virStorageSourceIsBacking(n); n = n->backingStore) { + qemuBlockNamedNodeDataBitmapPtr bitmapdata; - arr = virJSONValueNewArray(); - - if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(arr, - n->nodeformat, - srcbitmap->name) < 0) - return -1; + if (!(bitmapdata = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + n, delbitmap))) + continue; - if (qemuMonitorTransactionBitmapMerge(actions, - n->nodeformat, - parentbitmap, &arr) < 0) - return -1; - } + found = true; if (qemuMonitorTransactionBitmapRemove(actions, n->nodeformat, - srcbitmap->name) < 0) + bitmapdata->name) < 0) return -1; if (n != src) *reopenimages = g_slist_prepend(*reopenimages, n); + } - n = n->backingStore; + if (!found) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("bitmap '%s' not found in backing chain of '%s'"), + delbitmap, diskdst); + return -1; } return 0; @@ -247,8 +148,7 @@ qemuCheckpointDiscardDiskBitmaps(virStorageSourcePtr src, static int qemuCheckpointDiscardBitmaps(virDomainObjPtr vm, - virDomainCheckpointDefPtr chkdef, - virDomainMomentObjPtr parent) + virDomainCheckpointDefPtr chkdef) { qemuDomainObjPrivatePtr priv = vm->privateData; virQEMUDriverPtr driver = priv->driver; @@ -268,8 +168,6 @@ qemuCheckpointDiscardBitmaps(virDomainObjPtr vm, for (i = 0; i < chkdef->ndisks; i++) { virDomainCheckpointDiskDef *chkdisk = &chkdef->disks[i]; virDomainDiskDefPtr domdisk = virDomainDiskByTarget(vm->def, chkdisk->name); - virDomainCheckpointDiskDef *parentchkdisk = NULL; - const char *parentbitmap = NULL; /* domdisk can be missing e.g. when it was unplugged */ if (!domdisk) @@ -278,15 +176,8 @@ qemuCheckpointDiscardBitmaps(virDomainObjPtr vm, if (chkdisk->type != VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP) continue; - /* If any ancestor checkpoint has a bitmap for the same - * disk, then this bitmap must be merged to the - * ancestor. */ - if ((parentchkdisk = qemuCheckpointFindActiveDiskInParent(vm, parent, - chkdisk->name))) - parentbitmap = parentchkdisk->bitmap; - if (qemuCheckpointDiscardDiskBitmaps(domdisk->src, blockNamedNodeData, - chkdisk->bitmap, parentbitmap, + chkdisk->bitmap, actions, domdisk->dst, &reopenimages) < 0) return -1; @@ -334,7 +225,6 @@ qemuCheckpointDiscard(virQEMUDriverPtr driver, bool update_parent, bool metadata_only) { - virDomainMomentObjPtr parent = NULL; g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver); g_autofree char *chkFile = NULL; bool chkcurrent = chk == virDomainCheckpointGetCurrent(vm->checkpoints); @@ -350,14 +240,17 @@ qemuCheckpointDiscard(virQEMUDriverPtr driver, if (!metadata_only) { virDomainCheckpointDefPtr chkdef = virDomainCheckpointObjGetDef(chk); - parent = virDomainCheckpointFindByName(vm->checkpoints, - chk->def->parent_name); - if (qemuCheckpointDiscardBitmaps(vm, chkdef, parent) < 0) + if (qemuCheckpointDiscardBitmaps(vm, chkdef) < 0) return -1; } if (chkcurrent) { + virDomainMomentObjPtr parent = NULL; + virDomainCheckpointSetCurrent(vm->checkpoints, NULL); + parent = virDomainCheckpointFindByName(vm->checkpoints, + chk->def->parent_name); + if (update_parent && parent) { virDomainCheckpointSetCurrent(vm->checkpoints, parent); if (qemuCheckpointWriteMetadata(vm, parent, diff --git a/src/qemu/qemu_checkpoint.h b/src/qemu/qemu_checkpoint.h index cf1e9e46cb..0d267a188c 100644 --- a/src/qemu/qemu_checkpoint.h +++ b/src/qemu/qemu_checkpoint.h @@ -76,7 +76,6 @@ int qemuCheckpointDiscardDiskBitmaps(virStorageSourcePtr src, virHashTablePtr blockNamedNodeData, const char *delbitmap, - const char *parentbitmap, virJSONValuePtr actions, const char *diskdst, GSList **reopenimages); diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 47e9f0f363..336ff4b3ef 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -830,7 +830,6 @@ struct testQemuCheckpointDeleteMergeData { const char *name; virStorageSourcePtr chain; const char *deletebitmap; - const char *parentbitmap; const char *nodedatafile; }; @@ -864,7 +863,6 @@ testQemuCheckpointDeleteMerge(const void *opaque) if (qemuCheckpointDiscardDiskBitmaps(data->chain, nodedata, data->deletebitmap, - data->parentbitmap, actions, "testdisk", &reopenimages) >= 0) { @@ -1319,19 +1317,18 @@ mymain(void) TEST_BACKUP_BITMAP_CALCULATE("empty", bitmapSourceChain, "a", "empty"); -#define TEST_CHECKPOINT_DELETE_MERGE(testname, delbmp, parbmp, named) \ +#define TEST_CHECKPOINT_DELETE_MERGE(testname, delbmp, named) \ do { \ checkpointdeletedata.name = testname; \ checkpointdeletedata.chain = bitmapSourceChain; \ checkpointdeletedata.deletebitmap = delbmp; \ - checkpointdeletedata.parentbitmap = parbmp; \ checkpointdeletedata.nodedatafile = named; \ if (virTestRun("checkpoint delete " testname, \ testQemuCheckpointDeleteMerge, &checkpointdeletedata) < 0) \ ret = -1; \ } while (0) - TEST_CHECKPOINT_DELETE_MERGE("empty", "a", NULL, "empty"); + TEST_CHECKPOINT_DELETE_MERGE("empty", "a", "empty"); #define TEST_BITMAP_VALIDATE(testname, bitmap, rc) \ do { \ -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Now that we've switched to the simple handling, the first thing that can be massively simplified is checkpoint deletion. We now need to only go through the backing chain and find the appropriately named bitmaps and delete them, no complex lookups or merging.
Note that compared to other functions this deletes the bitmap in all layers compared to others where we expect only exactly 1 bitmap of a name in the backing chain to prevent potential problems.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_checkpoint.c | 153 ++++++------------------------------- src/qemu/qemu_checkpoint.h | 1 - tests/qemublocktest.c | 7 +- 3 files changed, 25 insertions(+), 136 deletions(-)
No kidding about the simplification. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Also rename the helper struct and function. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 336ff4b3ef..82297d0a8b 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -826,7 +826,7 @@ testQemuBackupIncrementalBitmapCalculate(const void *opaque) static const char *checkpointDeletePrefix = "qemublocktestdata/checkpointdelete/"; -struct testQemuCheckpointDeleteMergeData { +struct testQemuCheckpointDeleteData { const char *name; virStorageSourcePtr chain; const char *deletebitmap; @@ -835,9 +835,9 @@ struct testQemuCheckpointDeleteMergeData { static int -testQemuCheckpointDeleteMerge(const void *opaque) +testQemuCheckpointDelete(const void *opaque) { - const struct testQemuCheckpointDeleteMergeData *data = opaque; + const struct testQemuCheckpointDeleteData *data = opaque; g_autofree char *actual = NULL; g_autofree char *expectpath = NULL; g_autoptr(virJSONValue) actions = NULL; @@ -1037,7 +1037,7 @@ mymain(void) struct testJSONtoJSONData jsontojsondata; struct testQemuImageCreateData imagecreatedata; struct testQemuBackupIncrementalBitmapCalculateData backupbitmapcalcdata; - struct testQemuCheckpointDeleteMergeData checkpointdeletedata; + struct testQemuCheckpointDeleteData checkpointdeletedata; struct testQemuBlockBitmapValidateData blockbitmapvalidatedata; struct testQemuBlockBitmapBlockcopyData blockbitmapblockcopydata; struct testQemuBlockBitmapBlockcommitData blockbitmapblockcommitdata; @@ -1317,18 +1317,18 @@ mymain(void) TEST_BACKUP_BITMAP_CALCULATE("empty", bitmapSourceChain, "a", "empty"); -#define TEST_CHECKPOINT_DELETE_MERGE(testname, delbmp, named) \ +#define TEST_CHECKPOINT_DELETE(testname, delbmp, named) \ do { \ checkpointdeletedata.name = testname; \ checkpointdeletedata.chain = bitmapSourceChain; \ checkpointdeletedata.deletebitmap = delbmp; \ checkpointdeletedata.nodedatafile = named; \ if (virTestRun("checkpoint delete " testname, \ - testQemuCheckpointDeleteMerge, &checkpointdeletedata) < 0) \ + testQemuCheckpointDelete, &checkpointdeletedata) < 0) \ ret = -1; \ } while (0) - TEST_CHECKPOINT_DELETE_MERGE("empty", "a", "empty"); + TEST_CHECKPOINT_DELETE("empty", "a", "empty"); #define TEST_BITMAP_VALIDATE(testname, bitmap, rc) \ do { \ -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Also rename the helper struct and function.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-)
Mechanical. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Exercise the now arguably simpler checkpoint deletion code on the 'basic', 'snapshots', and 'synthetic' test data sets. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 18 ++++++++ .../checkpointdelete/basic-current-out.json | 9 ++++ .../basic-intermediate1-out.json | 9 ++++ .../basic-intermediate2-out.json | 9 ++++ .../basic-intermediate3-out.json | 9 ++++ .../checkpointdelete/basic-noparent-out.json | 9 ++++ .../snapshots-current-out.json | 9 ++++ .../snapshots-intermediate1-out.json | 26 ++++++++++++ .../snapshots-intermediate2-out.json | 26 ++++++++++++ .../snapshots-intermediate3-out.json | 18 ++++++++ .../snapshots-noparent-out.json | 42 +++++++++++++++++++ .../synthetic-current-out.json | 9 ++++ .../synthetic-intermediate1-out.json | 11 +++++ .../synthetic-intermediate2-out.json | 11 +++++ .../synthetic-intermediate3-out.json | 19 +++++++++ .../synthetic-noparent-out.json | 11 +++++ 16 files changed, 245 insertions(+) create mode 100644 tests/qemublocktestdata/checkpointdelete/basic-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 82297d0a8b..9e142a8851 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1330,6 +1330,24 @@ mymain(void) TEST_CHECKPOINT_DELETE("empty", "a", "empty"); + TEST_CHECKPOINT_DELETE("basic-noparent", "a", "basic"); + TEST_CHECKPOINT_DELETE("basic-intermediate1", "b", "basic"); + TEST_CHECKPOINT_DELETE("basic-intermediate2", "c", "basic"); + TEST_CHECKPOINT_DELETE("basic-intermediate3", "d", "basic"); + TEST_CHECKPOINT_DELETE("basic-current", "current", "basic"); + + TEST_CHECKPOINT_DELETE("snapshots-noparent", "a", "snapshots"); + TEST_CHECKPOINT_DELETE("snapshots-intermediate1", "b", "snapshots"); + TEST_CHECKPOINT_DELETE("snapshots-intermediate2", "c", "snapshots"); + TEST_CHECKPOINT_DELETE("snapshots-intermediate3", "d", "snapshots"); + TEST_CHECKPOINT_DELETE("snapshots-current", "current", "snapshots"); + + TEST_CHECKPOINT_DELETE("synthetic-noparent", "a", "synthetic"); + TEST_CHECKPOINT_DELETE("synthetic-intermediate1", "b", "synthetic"); + TEST_CHECKPOINT_DELETE("synthetic-intermediate2", "c", "synthetic"); + TEST_CHECKPOINT_DELETE("synthetic-intermediate3", "d", "synthetic"); + TEST_CHECKPOINT_DELETE("synthetic-current", "current", "synthetic"); + #define TEST_BITMAP_VALIDATE(testname, bitmap, rc) \ do { \ blockbitmapvalidatedata.name = testname; \ diff --git a/tests/qemublocktestdata/checkpointdelete/basic-current-out.json b/tests/qemublocktestdata/checkpointdelete/basic-current-out.json new file mode 100644 index 0000000000..6ed1b63b66 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/basic-current-out.json @@ -0,0 +1,9 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "current" + } + } +] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json b/tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json new file mode 100644 index 0000000000..e1dd4920cd --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/basic-intermediate1-out.json @@ -0,0 +1,9 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "b" + } + } +] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json b/tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json new file mode 100644 index 0000000000..5638532d35 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/basic-intermediate2-out.json @@ -0,0 +1,9 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "c" + } + } +] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json b/tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json new file mode 100644 index 0000000000..8c56b0e689 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/basic-intermediate3-out.json @@ -0,0 +1,9 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "d" + } + } +] diff --git a/tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json b/tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json new file mode 100644 index 0000000000..e87382fdb4 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/basic-noparent-out.json @@ -0,0 +1,9 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "a" + } + } +] diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json new file mode 100644 index 0000000000..6ed1b63b66 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/snapshots-current-out.json @@ -0,0 +1,9 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "current" + } + } +] diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json new file mode 100644 index 0000000000..1cde09131e --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate1-out.json @@ -0,0 +1,26 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "b" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-2-format", + "name": "b" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "b" + } + } +] +reopen nodes: +libvirt-3-format +libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json new file mode 100644 index 0000000000..6fcd7ee0d1 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate2-out.json @@ -0,0 +1,26 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "c" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-2-format", + "name": "c" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "c" + } + } +] +reopen nodes: +libvirt-3-format +libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json new file mode 100644 index 0000000000..2977dc3042 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/snapshots-intermediate3-out.json @@ -0,0 +1,18 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "d" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-2-format", + "name": "d" + } + } +] +reopen nodes: +libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json b/tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json new file mode 100644 index 0000000000..e4a6a413a3 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/snapshots-noparent-out.json @@ -0,0 +1,42 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "a" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-2-format", + "name": "a" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "a" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-4-format", + "name": "a" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-5-format", + "name": "a" + } + } +] +reopen nodes: +libvirt-5-format +libvirt-4-format +libvirt-3-format +libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json b/tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json new file mode 100644 index 0000000000..6ed1b63b66 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json @@ -0,0 +1,9 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-1-format", + "name": "current" + } + } +] diff --git a/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json b/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json new file mode 100644 index 0000000000..6eb9716aaa --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json @@ -0,0 +1,11 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "b" + } + } +] +reopen nodes: +libvirt-3-format diff --git a/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json b/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json new file mode 100644 index 0000000000..92b849cfc9 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json @@ -0,0 +1,11 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "c" + } + } +] +reopen nodes: +libvirt-3-format diff --git a/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json b/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json new file mode 100644 index 0000000000..466df52007 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json @@ -0,0 +1,19 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-2-format", + "name": "d" + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "d" + } + } +] +reopen nodes: +libvirt-3-format +libvirt-2-format diff --git a/tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json b/tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json new file mode 100644 index 0000000000..0479815f36 --- /dev/null +++ b/tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json @@ -0,0 +1,11 @@ +[ + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-5-format", + "name": "a" + } + } +] +reopen nodes: +libvirt-5-format -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Exercise the now arguably simpler checkpoint deletion code on the 'basic', 'snapshots', and 'synthetic' test data sets.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 18 ++++++++ .../checkpointdelete/basic-current-out.json | 9 ++++ .../basic-intermediate1-out.json | 9 ++++ .../basic-intermediate2-out.json | 9 ++++ .../basic-intermediate3-out.json | 9 ++++ .../checkpointdelete/basic-noparent-out.json | 9 ++++ .../snapshots-current-out.json | 9 ++++ .../snapshots-intermediate1-out.json | 26 ++++++++++++ .../snapshots-intermediate2-out.json | 26 ++++++++++++ .../snapshots-intermediate3-out.json | 18 ++++++++ .../snapshots-noparent-out.json | 42 +++++++++++++++++++ .../synthetic-current-out.json | 9 ++++ .../synthetic-intermediate1-out.json | 11 +++++ .../synthetic-intermediate2-out.json | 11 +++++ .../synthetic-intermediate3-out.json | 19 +++++++++ .../synthetic-noparent-out.json | 11 +++++
Looks reasonable; you're deleting the bitmap across all files in the chain where it appears, and don't have to mess around with which bitmaps to re-enable or merge into. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Add a function which allows to merge bitmaps according to the new semantics and will allow to replace all the specific ad-hoc functions currently in use for 'backup', 'block commit', 'block copy' and will also be usable in the future for 'block pull' and non-shared storage migration. The semantics are a bit quirky for the 'backup' case but these quirks are documented and will prevent us from having two slightly different algorithms. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 172 ++++++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_block.h | 10 +++ 2 files changed, 182 insertions(+) diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 83e3df9601..86f8410ce7 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -2847,6 +2847,178 @@ qemuBlockGetNamedNodeData(virDomainObjPtr vm, } +/** + * qemuBlockGetBitmapMergeActionsGetBitmaps: + * + * Collect a list of bitmaps which need to be handled in + * qemuBlockGetBitmapMergeActions. The list contains only valid bitmaps in the + * sub-chain which is being processed. + * + * Note that the returned GSList contains bitmap names string pointers borrowed + * from @blockNamedNodeData so they must not be freed. + */ +static GSList * +qemuBlockGetBitmapMergeActionsGetBitmaps(virStorageSourcePtr topsrc, + const char *bitmapname, + virHashTablePtr blockNamedNodeData) +{ + g_autoptr(GSList) ret = NULL; + qemuBlockNamedNodeDataPtr entry; + size_t i; + + /* for now it doesn't make sense to consider bitmaps which are not present + * in @topsrc as we can't recreate a bitmap for a layer if it's missing */ + + if (!(entry = virHashLookup(blockNamedNodeData, topsrc->nodeformat))) + return NULL; + + for (i = 0; i < entry->nbitmaps; i++) { + qemuBlockNamedNodeDataBitmapPtr bitmap = entry->bitmaps[i]; + + if (bitmapname && + STRNEQ(bitmapname, bitmap->name)) + continue; + + if (!qemuBlockBitmapChainIsValid(topsrc, bitmap->name, blockNamedNodeData)) + continue; + + ret = g_slist_prepend(ret, bitmap->name); + } + + return g_steal_pointer(&ret); +} + + +/** + * qemuBlockGetBitmapMergeActions: + * @topsrc: top of the chain to merge bitmaps in + * @basesrc: bottom of the chain to merge bitmaps in (NULL for full chain) + * @target: destination storage source of the merge (may be part of original chain) + * @bitmapname: name of bitmap to perform the merge (NULL for all bitmaps) + * @dstbitmapname: name of destination bitmap of the merge (see below for caveats) + * @writebitmapsrc: storage source corresponding to the node containing the write temporary bitmap + * @actions: returns actions for a 'transaction' QMP command for executing the merge + * @blockNamedNodeData: hash table filled with qemuBlockNamedNodeData + * + * Calculate handling of dirty block bitmaps between @topsrc and @basesrc. If + * @basesrc is NULL the end of the chain is considered. @target is the destination + * storage source definition of the merge and may or may not be part of the + * merged chain. + * + * Specifically the merging algorithm ensures that each considered bitmap is + * merged with the appropriate bitmaps so that it properly describes + * the state of dirty blocks when looked at from @topsrc based on the depth + * of the backing chain where the bitmap is placed. + * + * If @bitmapname is non-NULL only bitmaps with that name are handled, otherwise + * all bitmaps are considered. + * + * If @dstbitmap is non-NULL everything is merged into a bitmap with that name, + * otherwise each bitmap is merged into a bitmap with the same name into @target. + * Additionally if @dstbitmap is non-NULL the target bitmap is created as 'inactive' + * and 'transient' as a special case for the backup operation. + * + * If @writebitmapsrc is non-NULL, the 'libvirt-tmp-activewrite' bitmap from + * given node is merged along with others. This bitmap corresponds to the writes + * which occured between an active layer job finished and the rest of the bitmap + * merging. + * + * If the bitmap is not valid somehow (see qemuBlockBitmapChainIsValid) given + * bitmap is silently skipped, so callers must ensure that given bitmap is valid + * if they care about it. + * + * The resulting 'transacion' QMP command actions are filled in and returned via + * @actions. + * + * Note that @actions may be NULL if no merging is required. + */ +int +qemuBlockGetBitmapMergeActions(virStorageSourcePtr topsrc, + virStorageSourcePtr basesrc, + virStorageSourcePtr target, + const char *bitmapname, + const char *dstbitmapname, + virStorageSourcePtr writebitmapsrc, + virJSONValuePtr *actions, + virHashTablePtr blockNamedNodeData) +{ + g_autoptr(virJSONValue) act = virJSONValueNewArray(); + virStorageSourcePtr n; + + g_autoptr(GSList) bitmaps = NULL; + GSList *next; + + if (!(bitmaps = qemuBlockGetBitmapMergeActionsGetBitmaps(topsrc, bitmapname, + blockNamedNodeData))) + return 0; + + for (next = bitmaps; next; next = next->next) { + const char *curbitmap = next->data; + const char *mergebitmapname = dstbitmapname; + bool mergebitmappersistent = false; + bool mergebitmapdisabled = true; + g_autoptr(virJSONValue) merge = virJSONValueNewArray(); + unsigned long long granularity = 0; + qemuBlockNamedNodeDataBitmapPtr bitmap; + + /* explicitly named destinations mean that we want a temporary + * disabled bitmap only, so undo the default for non-explicit cases */ + if (!mergebitmapname) { + mergebitmapname = curbitmap; + mergebitmappersistent = true; + mergebitmapdisabled = false; + } + + for (n = topsrc; virStorageSourceIsBacking(n) && n != basesrc; n = n->backingStore) { + if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + n, curbitmap))) + continue; + + if (granularity == 0) + granularity = bitmap->granularity; + + if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(merge, + n->nodeformat, + bitmap->name) < 0) + return -1; + } + + if (dstbitmapname || + !(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + target, curbitmap))) { + + if (qemuMonitorTransactionBitmapAdd(act, + target->nodeformat, + mergebitmapname, + mergebitmappersistent, + mergebitmapdisabled, + granularity) < 0) + return -1; + } + + if (writebitmapsrc && + qemuMonitorTransactionBitmapMergeSourceAddBitmap(merge, + writebitmapsrc->nodeformat, + "libvirt-tmp-activewrite") < 0) + return -1; + + if (qemuMonitorTransactionBitmapMerge(act, target->nodeformat, + mergebitmapname, &merge) < 0) + return -1; + } + + if (writebitmapsrc && + qemuMonitorTransactionBitmapRemove(act, writebitmapsrc->nodeformat, + "libvirt-tmp-activewrite") < 0) + return -1; + + if (virJSONValueArraySize(act) > 0) + *actions = g_steal_pointer(&act); + + return 0; +} + + /** * qemuBlockBitmapChainIsValid: * diff --git a/src/qemu/qemu_block.h b/src/qemu/qemu_block.h index 2ad2ce1a1f..0f5b4cfaa9 100644 --- a/src/qemu/qemu_block.h +++ b/src/qemu/qemu_block.h @@ -221,6 +221,16 @@ virHashTablePtr qemuBlockGetNamedNodeData(virDomainObjPtr vm, qemuDomainAsyncJob asyncJob); +int +qemuBlockGetBitmapMergeActions(virStorageSourcePtr topsrc, + virStorageSourcePtr basesrc, + virStorageSourcePtr target, + const char *bitmapname, + const char *dstbitmapname, + virStorageSourcePtr writebitmapsrc, + virJSONValuePtr *actions, + virHashTablePtr blockNamedNodeData); + bool qemuBlockBitmapChainIsValid(virStorageSourcePtr src, const char *bitmapname, -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Add a function which allows to merge bitmaps according to the new
Another reminder about "allows to $VERB" being non-idiomatic :) s/allows to merge/allows merging/
semantics and will allow to replace all the specific ad-hoc functions
s/allow to replace/allow replacing/
currently in use for 'backup', 'block commit', 'block copy' and will also be usable in the future for 'block pull' and non-shared storage migration.
The semantics are a bit quirky for the 'backup' case but these quirks are documented and will prevent us from having two slightly different algorithms.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 172 ++++++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_block.h | 10 +++ 2 files changed, 182 insertions(+)
I'll have to read the later patches that start using this in place of their current ad-hoc usage, but in isolation, the new function looks reasonable.
diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 83e3df9601..86f8410ce7 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -2847,6 +2847,178 @@ qemuBlockGetNamedNodeData(virDomainObjPtr vm, }
+/** + * qemuBlockGetBitmapMergeActionsGetBitmaps: + * + * Collect a list of bitmaps which need to be handled in + * qemuBlockGetBitmapMergeActions. The list contains only valid bitmaps in the + * sub-chain which is being processed. + * + * Note that the returned GSList contains bitmap names string pointers borrowed + * from @blockNamedNodeData so they must not be freed. + */ +static GSList * +qemuBlockGetBitmapMergeActionsGetBitmaps(virStorageSourcePtr topsrc, + const char *bitmapname, + virHashTablePtr blockNamedNodeData) +{ + g_autoptr(GSList) ret = NULL; + qemuBlockNamedNodeDataPtr entry; + size_t i; + + /* for now it doesn't make sense to consider bitmaps which are not present + * in @topsrc as we can't recreate a bitmap for a layer if it's missing */
I'm guessing qemu adding block-dirty-bitmap-populate will somewhat help here (if it's missing in top, but present in a backing layer, the fix is to populate the bitmap in the top layer according to the allocation map).
+ + if (!(entry = virHashLookup(blockNamedNodeData, topsrc->nodeformat))) + return NULL; + + for (i = 0; i < entry->nbitmaps; i++) { + qemuBlockNamedNodeDataBitmapPtr bitmap = entry->bitmaps[i]; + + if (bitmapname && + STRNEQ(bitmapname, bitmap->name)) + continue; + + if (!qemuBlockBitmapChainIsValid(topsrc, bitmap->name, blockNamedNodeData)) + continue; + + ret = g_slist_prepend(ret, bitmap->name); + } + + return g_steal_pointer(&ret); +} + + +/** + * qemuBlockGetBitmapMergeActions: + * @topsrc: top of the chain to merge bitmaps in + * @basesrc: bottom of the chain to merge bitmaps in (NULL for full chain) + * @target: destination storage source of the merge (may be part of original chain) + * @bitmapname: name of bitmap to perform the merge (NULL for all bitmaps) + * @dstbitmapname: name of destination bitmap of the merge (see below for caveats) + * @writebitmapsrc: storage source corresponding to the node containing the write temporary bitmap + * @actions: returns actions for a 'transaction' QMP command for executing the merge + * @blockNamedNodeData: hash table filled with qemuBlockNamedNodeData + * + * Calculate handling of dirty block bitmaps between @topsrc and @basesrc. If + * @basesrc is NULL the end of the chain is considered. @target is the destination + * storage source definition of the merge and may or may not be part of the + * merged chain. + * + * Specifically the merging algorithm ensures that each considered bitmap is + * merged with the appropriate bitmaps so that it properly describes + * the state of dirty blocks when looked at from @topsrc based on the depth + * of the backing chain where the bitmap is placed. + * + * If @bitmapname is non-NULL only bitmaps with that name are handled, otherwise + * all bitmaps are considered. + * + * If @dstbitmap is non-NULL everything is merged into a bitmap with that name, + * otherwise each bitmap is merged into a bitmap with the same name into @target. + * Additionally if @dstbitmap is non-NULL the target bitmap is created as 'inactive' + * and 'transient' as a special case for the backup operation. + * + * If @writebitmapsrc is non-NULL, the 'libvirt-tmp-activewrite' bitmap from + * given node is merged along with others. This bitmap corresponds to the writes + * which occured between an active layer job finished and the rest of the bitmap
occurred
+ * merging. + * + * If the bitmap is not valid somehow (see qemuBlockBitmapChainIsValid) given + * bitmap is silently skipped, so callers must ensure that given bitmap is valid + * if they care about it. + * + * The resulting 'transacion' QMP command actions are filled in and returned via
transaction
+ * @actions. + * + * Note that @actions may be NULL if no merging is required. + */ +int +qemuBlockGetBitmapMergeActions(virStorageSourcePtr topsrc, + virStorageSourcePtr basesrc, + virStorageSourcePtr target, + const char *bitmapname, + const char *dstbitmapname, + virStorageSourcePtr writebitmapsrc, + virJSONValuePtr *actions, + virHashTablePtr blockNamedNodeData)
Looks a bit odd to have an output parameter (actions) prior to an input parameter (blockNamedNodeData), but not fatal.
+{ + g_autoptr(virJSONValue) act = virJSONValueNewArray(); + virStorageSourcePtr n; + + g_autoptr(GSList) bitmaps = NULL; + GSList *next; + + if (!(bitmaps = qemuBlockGetBitmapMergeActionsGetBitmaps(topsrc, bitmapname, + blockNamedNodeData))) + return 0;
Look up one or all bitmaps in the top layer (based on whether bitmapname is set),
+ + for (next = bitmaps; next; next = next->next) { + const char *curbitmap = next->data; + const char *mergebitmapname = dstbitmapname; + bool mergebitmappersistent = false; + bool mergebitmapdisabled = true; + g_autoptr(virJSONValue) merge = virJSONValueNewArray(); + unsigned long long granularity = 0; + qemuBlockNamedNodeDataBitmapPtr bitmap; + + /* explicitly named destinations mean that we want a temporary + * disabled bitmap only, so undo the default for non-explicit cases */ + if (!mergebitmapname) { + mergebitmapname = curbitmap; + mergebitmappersistent = true; + mergebitmapdisabled = false; + } + + for (n = topsrc; virStorageSourceIsBacking(n) && n != basesrc; n = n->backingStore) { + if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + n, curbitmap))) + continue; + + if (granularity == 0) + granularity = bitmap->granularity; + + if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(merge, + n->nodeformat, + bitmap->name) < 0) + return -1; + }
So this loop merges the same bitmap name along all disks in the backing chain where it is present,
+ + if (dstbitmapname || + !(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, + target, curbitmap))) { + + if (qemuMonitorTransactionBitmapAdd(act, + target->nodeformat, + mergebitmapname, + mergebitmappersistent, + mergebitmapdisabled, + granularity) < 0) + return -1; + }
if a destination was set or if the bitmap does not exist in the target layer, then we add in a bitmap for the destination,
+ + if (writebitmapsrc && + qemuMonitorTransactionBitmapMergeSourceAddBitmap(merge, + writebitmapsrc->nodeformat, + "libvirt-tmp-activewrite") < 0) + return -1;
if a temporary bitmap was needed, we add that in too,
+ + if (qemuMonitorTransactionBitmapMerge(act, target->nodeformat, + mergebitmapname, &merge) < 0) + return -1;
and repeat a merge inside the single transaction for each bitmap involved.
+ } + + if (writebitmapsrc && + qemuMonitorTransactionBitmapRemove(act, writebitmapsrc->nodeformat, + "libvirt-tmp-activewrite") < 0) + return -1;
The transaction also removes any temporary bitmap. You only do this once instead of per-bitmap; I'm guessing that means that writebitmapsrc can only be used when bitmapname is non-NULL. Should there be a check for that?
+ + if (virJSONValueArraySize(act) > 0) + *actions = g_steal_pointer(&act); + + return 0; +} + +
Otherwise, it seems to make sense as a generalized tool. We'll see how later patches use it. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc iplementatio of bitmap merging for backup. The new approach is simpler and also more robust in case some of the bitmaps break as they remove the dependancy on the whole chain of bitmaps working. The new approach also allows backups if a snapshot is created outside of libvirt. Additionally the code is greatly simplified. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 220 +++--------------- src/qemu/qemu_backup.h | 12 +- tests/qemublocktest.c | 86 ++----- .../backupmerge/empty-out.json | 4 +- 4 files changed, 63 insertions(+), 259 deletions(-) diff --git a/src/qemu/qemu_backup.c b/src/qemu/qemu_backup.c index 495da86ff0..7c337846dc 100644 --- a/src/qemu/qemu_backup.c +++ b/src/qemu/qemu_backup.c @@ -173,199 +173,58 @@ qemuBackupDiskDataCleanup(virDomainObjPtr vm, } -/** - * qemuBackupBeginCollectIncrementalCheckpoints: - * @vm: domain object - * @incrFrom: name of checkpoint representing starting point of incremental backup - * - * Returns a NULL terminated list of pointers to checkpoint definitions in - * chronological order starting from the 'current' checkpoint until reaching - * @incrFrom. - */ -static virDomainMomentDefPtr * -qemuBackupBeginCollectIncrementalCheckpoints(virDomainObjPtr vm, - const char *incrFrom) -{ - virDomainMomentObjPtr n = virDomainCheckpointGetCurrent(vm->checkpoints); - g_autofree virDomainMomentDefPtr *incr = NULL; - size_t nincr = 0; - - while (n) { - virDomainMomentDefPtr def = n->def; - - if (VIR_APPEND_ELEMENT_COPY(incr, nincr, def) < 0) - return NULL; - - if (STREQ(def->name, incrFrom)) { - def = NULL; - if (VIR_APPEND_ELEMENT_COPY(incr, nincr, def) < 0) - return NULL; - - return g_steal_pointer(&incr); - } - - if (!n->def->parent_name) - break; - - n = virDomainCheckpointFindByName(vm->checkpoints, n->def->parent_name); - } - - virReportError(VIR_ERR_OPERATION_INVALID, - _("could not locate checkpoint '%s' for incremental backup"), - incrFrom); - return NULL; -} - - -static int -qemuBackupGetBitmapMergeRange(virStorageSourcePtr from, - const char *bitmapname, - virJSONValuePtr *actions, - virStorageSourcePtr *to, - const char *diskdst, - virHashTablePtr blockNamedNodeData) +int +qemuBackupDiskPrepareOneBitmapsChain(virStorageSourcePtr backingChain, + virStorageSourcePtr targetsrc, + const char *targetbitmap, + const char *incremental, + virJSONValuePtr actions, + virHashTablePtr blockNamedNodeData) { - g_autoptr(virJSONValue) ret = virJSONValueNewArray(); - virStorageSourcePtr tmpsrc = NULL; - virStorageSourcePtr n; - bool foundbitmap = false; + g_autoptr(virJSONValue) tmpactions = NULL; - for (n = from; virStorageSourceIsBacking(n); n = n->backingStore) { - qemuBlockNamedNodeDataBitmapPtr bitmap = NULL; - - if (!(bitmap = qemuBlockNamedNodeDataGetBitmapByName(blockNamedNodeData, - n, - bitmapname))) - break; - - foundbitmap = true; - - if (bitmap->inconsistent) { - virReportError(VIR_ERR_INVALID_ARG, - _("bitmap '%s' for image '%s%u' is inconsistent"), - bitmap->name, diskdst, n->id); - return -1; - } - - if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(ret, - n->nodeformat, - bitmapname) < 0) - return -1; - - tmpsrc = n; - } - - if (!foundbitmap) { - virReportError(VIR_ERR_INVALID_ARG, - _("failed to find bitmap '%s' in image '%s%u'"), - bitmapname, diskdst, from->id); + if (qemuBlockGetBitmapMergeActions(backingChain, NULL, targetsrc, + incremental, targetbitmap, NULL, + &tmpactions, + blockNamedNodeData) < 0) return -1; - } - *actions = g_steal_pointer(&ret); - *to = tmpsrc; + if (tmpactions && + virJSONValueArrayConcat(actions, tmpactions) < 0) + return -1; return 0; } -virJSONValuePtr -qemuBackupDiskPrepareOneBitmapsChain(virDomainMomentDefPtr *incremental, - virStorageSourcePtr backingChain, - virHashTablePtr blockNamedNodeData, - const char *diskdst) -{ - g_autoptr(virJSONValue) ret = NULL; - size_t incridx = 0; - virStorageSourcePtr n = backingChain; - - ret = virJSONValueNewArray(); - - for (incridx = 0; incremental[incridx]; incridx++) { - g_autoptr(virJSONValue) tmp = virJSONValueNewArray(); - virStorageSourcePtr tmpsrc = NULL; - virDomainCheckpointDefPtr chkdef = (virDomainCheckpointDefPtr) incremental[incridx]; - bool checkpoint_has_disk = false; - size_t i; - - for (i = 0; i < chkdef->ndisks; i++) { - if (STRNEQ_NULLABLE(diskdst, chkdef->disks[i].name)) - continue; - - if (chkdef->disks[i].type == VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP) - checkpoint_has_disk = true; - - break; - } - - if (!checkpoint_has_disk) { - if (!incremental[incridx + 1]) { - virReportError(VIR_ERR_INVALID_ARG, - _("disk '%s' not found in checkpoint '%s'"), - diskdst, incremental[incridx]->name); - return NULL; - } - - continue; - } - - if (qemuBackupGetBitmapMergeRange(n, incremental[incridx]->name, - &tmp, &tmpsrc, diskdst, - blockNamedNodeData) < 0) - return NULL; - - if (virJSONValueArrayConcat(ret, tmp) < 0) - return NULL; - - n = tmpsrc; - } - - return g_steal_pointer(&ret); -} - - static int qemuBackupDiskPrepareOneBitmaps(struct qemuBackupDiskData *dd, virJSONValuePtr actions, - virDomainMomentDefPtr *incremental, virHashTablePtr blockNamedNodeData) { - g_autoptr(virJSONValue) mergebitmaps = NULL; - g_autoptr(virJSONValue) mergebitmapsstore = NULL; - - if (!(mergebitmaps = qemuBackupDiskPrepareOneBitmapsChain(incremental, - dd->domdisk->src, - blockNamedNodeData, - dd->domdisk->dst))) - return -1; - - if (!(mergebitmapsstore = virJSONValueCopy(mergebitmaps))) - return -1; - - if (qemuMonitorTransactionBitmapAdd(actions, - dd->domdisk->src->nodeformat, - dd->incrementalBitmap, - false, - true, 0) < 0) - return -1; - - if (qemuMonitorTransactionBitmapMerge(actions, - dd->domdisk->src->nodeformat, - dd->incrementalBitmap, - &mergebitmaps) < 0) + if (!qemuBlockBitmapChainIsValid(dd->domdisk->src, + dd->backupdisk->incremental, + blockNamedNodeData)) { + virReportError(VIR_ERR_INVALID_ARG, + _("missing or broken bitmap '%s' for disk '%s'"), + dd->backupdisk->incremental, dd->domdisk->dst); return -1; + } - if (qemuMonitorTransactionBitmapAdd(actions, - dd->store->nodeformat, - dd->incrementalBitmap, - false, - true, 0) < 0) + if (qemuBackupDiskPrepareOneBitmapsChain(dd->domdisk->src, + dd->domdisk->src, + dd->incrementalBitmap, + dd->backupdisk->incremental, + actions, + blockNamedNodeData) < 0) return -1; - if (qemuMonitorTransactionBitmapMerge(actions, - dd->store->nodeformat, - dd->incrementalBitmap, - &mergebitmapsstore) < 0) + if (qemuBackupDiskPrepareOneBitmapsChain(dd->domdisk->src, + dd->store, + dd->incrementalBitmap, + dd->backupdisk->incremental, + actions, + blockNamedNodeData) < 0) return -1; return 0; @@ -382,7 +241,6 @@ qemuBackupDiskPrepareDataOne(virDomainObjPtr vm, virQEMUDriverConfigPtr cfg) { qemuDomainObjPrivatePtr priv = vm->privateData; - g_autofree virDomainMomentDefPtr *incremental = NULL; /* set data structure */ dd->backupdisk = backupdisk; @@ -420,16 +278,12 @@ qemuBackupDiskPrepareDataOne(virDomainObjPtr vm, return -1; if (dd->backupdisk->incremental) { - if (!(incremental = qemuBackupBeginCollectIncrementalCheckpoints(vm, dd->backupdisk->incremental))) - return -1; - if (dd->backupdisk->exportbitmap) dd->incrementalBitmap = g_strdup(dd->backupdisk->exportbitmap); else dd->incrementalBitmap = g_strdup_printf("backup-%s", dd->domdisk->dst); - if (qemuBackupDiskPrepareOneBitmaps(dd, actions, incremental, - blockNamedNodeData) < 0) + if (qemuBackupDiskPrepareOneBitmaps(dd, actions, blockNamedNodeData) < 0) return -1; } @@ -883,8 +737,8 @@ qemuBackupBegin(virDomainObjPtr vm, if (!(blockNamedNodeData = qemuBlockGetNamedNodeData(vm, QEMU_ASYNC_JOB_BACKUP))) goto endjob; - if ((ndd = qemuBackupDiskPrepareData(vm, def, blockNamedNodeData, - actions, cfg, &dd)) <= 0) { + if ((ndd = qemuBackupDiskPrepareData(vm, def, blockNamedNodeData, actions, + cfg, &dd)) <= 0) { if (ndd == 0) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("no disks selected for backup")); diff --git a/src/qemu/qemu_backup.h b/src/qemu/qemu_backup.h index b19c3bf1c9..075fde709b 100644 --- a/src/qemu/qemu_backup.h +++ b/src/qemu/qemu_backup.h @@ -53,8 +53,10 @@ qemuBackupGetJobInfoStats(virQEMUDriverPtr driver, qemuDomainJobInfoPtr jobInfo); /* exported for testing */ -virJSONValuePtr -qemuBackupDiskPrepareOneBitmapsChain(virDomainMomentDefPtr *incremental, - virStorageSourcePtr backingChain, - virHashTablePtr blockNamedNodeData, - const char *diskdst); +int +qemuBackupDiskPrepareOneBitmapsChain(virStorageSourcePtr backingChain, + virStorageSourcePtr targetsrc, + const char *targetbitmap, + const char *incremental, + virJSONValuePtr actions, + virHashTablePtr blockNamedNodeData); diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 9e142a8851..30662f6f79 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -716,65 +716,6 @@ testQemuBitmapGetFakeChainEntry(virStorageSourcePtr src, } -typedef virDomainMomentDefPtr testMomentList; - -static void -testMomentListFree(testMomentList *list) -{ - testMomentList *tmp = list; - - if (!list) - return; - - while (*tmp) { - virObjectUnref(*tmp); - tmp++; - } - - g_free(list); -} - -G_DEFINE_AUTOPTR_CLEANUP_FUNC(testMomentList, testMomentListFree); - -static virDomainMomentDefPtr -testQemuBackupGetIncrementalMoment(const char *name) -{ - virDomainCheckpointDefPtr checkpoint = NULL; - - if (!(checkpoint = virDomainCheckpointDefNew())) - abort(); - - checkpoint->disks = g_new0(virDomainCheckpointDiskDef, 1); - checkpoint->ndisks = 1; - - checkpoint->disks[0].name = g_strdup("testdisk"); - checkpoint->disks[0].type = VIR_DOMAIN_CHECKPOINT_TYPE_BITMAP; - - checkpoint->parent.name = g_strdup(name); - - return (virDomainMomentDefPtr) checkpoint; -} - - -static virDomainMomentDefPtr * -testQemuBackupGetIncremental(const char *incFrom) -{ - const char *checkpoints[] = {"current", "d", "c", "b", "a"}; - virDomainMomentDefPtr *incr; - size_t i; - - incr = g_new0(virDomainMomentDefPtr, G_N_ELEMENTS(checkpoints) + 1); - - for (i = 0; i < G_N_ELEMENTS(checkpoints); i++) { - incr[i] = testQemuBackupGetIncrementalMoment(checkpoints[i]); - - if (STREQ(incFrom, checkpoints[i])) - break; - } - - return incr; -} - static const char *backupDataPrefix = "qemublocktestdata/backupmerge/"; struct testQemuBackupIncrementalBitmapCalculateData { @@ -791,10 +732,10 @@ testQemuBackupIncrementalBitmapCalculate(const void *opaque) const struct testQemuBackupIncrementalBitmapCalculateData *data = opaque; g_autoptr(virJSONValue) nodedatajson = NULL; g_autoptr(virHashTable) nodedata = NULL; - g_autoptr(virJSONValue) mergebitmaps = NULL; - g_autofree char *actual = NULL; + g_autoptr(virJSONValue) actions = virJSONValueNewArray(); g_autofree char *expectpath = NULL; - g_autoptr(testMomentList) incremental = NULL; + g_autoptr(virStorageSource) target = NULL; + g_auto(virBuffer) buf = VIR_BUFFER_INITIALIZER; expectpath = g_strdup_printf("%s/%s%s-out.json", abs_srcdir, backupDataPrefix, data->name); @@ -808,19 +749,24 @@ testQemuBackupIncrementalBitmapCalculate(const void *opaque) return -1; } - incremental = testQemuBackupGetIncremental(data->incremental); + if (!(target = virStorageSourceNew())) + return -1; - if ((mergebitmaps = qemuBackupDiskPrepareOneBitmapsChain(incremental, - data->chain, - nodedata, - "testdisk"))) { - if (!(actual = virJSONValueToString(mergebitmaps, true))) + target->nodeformat = g_strdup_printf("target_node"); + + if (qemuBackupDiskPrepareOneBitmapsChain(data->chain, + target, + "target-bitmap-name", + data->incremental, + actions, + nodedata) >= 0) { + if (virJSONValueToBuffer(actions, &buf, true) < 0) return -1; } else { - actual = g_strdup("NULL\n"); + virBufferAddLit(&buf, "NULL\n"); } - return virTestCompareToFile(actual, expectpath); + return virTestCompareToFile(virBufferCurrentContent(&buf), expectpath); } diff --git a/tests/qemublocktestdata/backupmerge/empty-out.json b/tests/qemublocktestdata/backupmerge/empty-out.json index 7951defec1..41b42e677b 100644 --- a/tests/qemublocktestdata/backupmerge/empty-out.json +++ b/tests/qemublocktestdata/backupmerge/empty-out.json @@ -1 +1,3 @@ -NULL +[ + +] -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc
allows the removal of the
iplementatio of bitmap merging for backup. The new approach is simpler
implementation
and also more robust in case some of the bitmaps break as they remove the dependancy on the whole chain of bitmaps working.
dependency
The new approach also allows backups if a snapshot is created outside of libvirt.
Additionally the code is greatly simplified.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_backup.c | 220 +++--------------- src/qemu/qemu_backup.h | 12 +- tests/qemublocktest.c | 86 ++----- .../backupmerge/empty-out.json | 4 +- 4 files changed, 63 insertions(+), 259 deletions(-)
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
The 'basic' case is just a single backing store layer containing the bitmaps so we just copy the bitmaps over to the backup bitmap. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 +++ .../backupmerge/basic-deep-out.json | 25 +++++++++++++++++++ .../backupmerge/basic-flat-out.json | 25 +++++++++++++++++++ .../backupmerge/basic-intermediate-out.json | 25 +++++++++++++++++++ 4 files changed, 79 insertions(+) create mode 100644 tests/qemublocktestdata/backupmerge/basic-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/basic-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/basic-intermediate-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 30662f6f79..1300e0cbe9 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1263,6 +1263,10 @@ mymain(void) TEST_BACKUP_BITMAP_CALCULATE("empty", bitmapSourceChain, "a", "empty"); + TEST_BACKUP_BITMAP_CALCULATE("basic-flat", bitmapSourceChain, "current", "basic"); + TEST_BACKUP_BITMAP_CALCULATE("basic-intermediate", bitmapSourceChain, "d", "basic"); + TEST_BACKUP_BITMAP_CALCULATE("basic-deep", bitmapSourceChain, "a", "basic"); + #define TEST_CHECKPOINT_DELETE(testname, delbmp, named) \ do { \ checkpointdeletedata.name = testname; \ diff --git a/tests/qemublocktestdata/backupmerge/basic-deep-out.json b/tests/qemublocktestdata/backupmerge/basic-deep-out.json new file mode 100644 index 0000000000..ff77af789b --- /dev/null +++ b/tests/qemublocktestdata/backupmerge/basic-deep-out.json @@ -0,0 +1,25 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "target_node", + "name": "target-bitmap-name", + "persistent": false, + "disabled": true, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "target_node", + "target": "target-bitmap-name", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/backupmerge/basic-flat-out.json b/tests/qemublocktestdata/backupmerge/basic-flat-out.json new file mode 100644 index 0000000000..4637bbc377 --- /dev/null +++ b/tests/qemublocktestdata/backupmerge/basic-flat-out.json @@ -0,0 +1,25 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "target_node", + "name": "target-bitmap-name", + "persistent": false, + "disabled": true, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "target_node", + "target": "target-bitmap-name", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/backupmerge/basic-intermediate-out.json b/tests/qemublocktestdata/backupmerge/basic-intermediate-out.json new file mode 100644 index 0000000000..f2f3b3f568 --- /dev/null +++ b/tests/qemublocktestdata/backupmerge/basic-intermediate-out.json @@ -0,0 +1,25 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "target_node", + "name": "target-bitmap-name", + "persistent": false, + "disabled": true, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "target_node", + "target": "target-bitmap-name", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + } + ] + } + } +] -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
The 'basic' case is just a single backing store layer containing the bitmaps so we just copy the bitmaps over to the backup bitmap.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 +++ .../backupmerge/basic-deep-out.json | 25 +++++++++++++++++++ .../backupmerge/basic-flat-out.json | 25 +++++++++++++++++++ .../backupmerge/basic-intermediate-out.json | 25 +++++++++++++++++++ 4 files changed, 79 insertions(+) create mode 100644 tests/qemublocktestdata/backupmerge/basic-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/basic-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/basic-intermediate-out.json
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
The 'snapshots' case has multiple layers so we need to make sure that the bitmaps are merged with the appropriate temporary bitmaps formatted from the allocation bitmap for any backing chain layer above. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 ++ .../backupmerge/snapshots-deep-out.json | 41 +++++++++++++++++++ .../backupmerge/snapshots-flat-out.json | 25 +++++++++++ .../snapshots-intermediate-out.json | 29 +++++++++++++ 4 files changed, 99 insertions(+) create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 1300e0cbe9..fd6dff82f9 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1267,6 +1267,10 @@ mymain(void) TEST_BACKUP_BITMAP_CALCULATE("basic-intermediate", bitmapSourceChain, "d", "basic"); TEST_BACKUP_BITMAP_CALCULATE("basic-deep", bitmapSourceChain, "a", "basic"); + TEST_BACKUP_BITMAP_CALCULATE("snapshots-flat", bitmapSourceChain, "current", "snapshots"); + TEST_BACKUP_BITMAP_CALCULATE("snapshots-intermediate", bitmapSourceChain, "d", "snapshots"); + TEST_BACKUP_BITMAP_CALCULATE("snapshots-deep", bitmapSourceChain, "a", "snapshots"); + #define TEST_CHECKPOINT_DELETE(testname, delbmp, named) \ do { \ checkpointdeletedata.name = testname; \ diff --git a/tests/qemublocktestdata/backupmerge/snapshots-deep-out.json b/tests/qemublocktestdata/backupmerge/snapshots-deep-out.json new file mode 100644 index 0000000000..d678507b85 --- /dev/null +++ b/tests/qemublocktestdata/backupmerge/snapshots-deep-out.json @@ -0,0 +1,41 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "target_node", + "name": "target-bitmap-name", + "persistent": false, + "disabled": true, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "target_node", + "target": "target-bitmap-name", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "a" + }, + { + "node": "libvirt-4-format", + "name": "a" + }, + { + "node": "libvirt-5-format", + "name": "a" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/backupmerge/snapshots-flat-out.json b/tests/qemublocktestdata/backupmerge/snapshots-flat-out.json new file mode 100644 index 0000000000..4637bbc377 --- /dev/null +++ b/tests/qemublocktestdata/backupmerge/snapshots-flat-out.json @@ -0,0 +1,25 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "target_node", + "name": "target-bitmap-name", + "persistent": false, + "disabled": true, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "target_node", + "target": "target-bitmap-name", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json b/tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json new file mode 100644 index 0000000000..30d8bcd8b7 --- /dev/null +++ b/tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json @@ -0,0 +1,29 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "target_node", + "name": "target-bitmap-name", + "persistent": false, + "disabled": true, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "target_node", + "target": "target-bitmap-name", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-2-format", + "name": "d" + } + ] + } + } +] -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
The 'snapshots' case has multiple layers so we need to make sure that the bitmaps are merged with the appropriate temporary bitmaps formatted from the allocation bitmap for any backing chain layer above.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 ++ .../backupmerge/snapshots-deep-out.json | 41 +++++++++++++++++++ .../backupmerge/snapshots-flat-out.json | 25 +++++++++++ .../snapshots-intermediate-out.json | 29 +++++++++++++ 4 files changed, 99 insertions(+) create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc implementatio of bitmap merging for block commit. The new approach is way simpler and more robust and also allows us to get rid of the disabling of bitmaps done prior to the start as we actually do want to update the bitmaps in the base. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 203 +----------------- src/qemu/qemu_block.h | 11 +- src/qemu/qemu_blockjob.c | 25 ++- src/qemu/qemu_driver.c | 56 +---- tests/qemublocktest.c | 18 +- .../qemublocktestdata/bitmapblockcommit/empty | 1 - 6 files changed, 42 insertions(+), 272 deletions(-) diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 86f8410ce7..e958fb71fa 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -3209,117 +3209,7 @@ qemuBlockBitmapsHandleBlockcopy(virStorageSourcePtr src, /** * @topsrc: virStorageSource representing 'top' of the job * @basesrc: virStorageSource representing 'base' of the job - * @blockNamedNodeData: hash table containing data about bitmaps - * @actions: filled with arguments for a 'transaction' command - * @disabledBitmapsBase: filled with a list of bitmap names which must be disabled - * - * Prepares data for correctly handling bitmaps during the start of a commit - * job. The bitmaps in the 'base' image must be disabled, so that the writes - * done by the blockjob don't dirty the enabled bitmaps. - * - * @actions and @disabledBitmapsBase are untouched if no bitmaps need - * to be disabled. - */ -int -qemuBlockBitmapsHandleCommitStart(virStorageSourcePtr topsrc, - virStorageSourcePtr basesrc, - virHashTablePtr blockNamedNodeData, - virJSONValuePtr *actions, - char ***disabledBitmapsBase) -{ - g_autoptr(virJSONValue) act = virJSONValueNewArray(); - VIR_AUTOSTRINGLIST bitmaplist = NULL; - size_t curbitmapstr = 0; - qemuBlockNamedNodeDataPtr entry; - bool disable_bitmaps = false; - size_t i; - - if (!(entry = virHashLookup(blockNamedNodeData, basesrc->nodeformat))) - return 0; - - bitmaplist = g_new0(char *, entry->nbitmaps + 1); - - for (i = 0; i < entry->nbitmaps; i++) { - qemuBlockNamedNodeDataBitmapPtr bitmap = entry->bitmaps[i]; - - if (!bitmap->recording || bitmap->inconsistent || - !qemuBlockBitmapChainIsValid(topsrc, bitmap->name, blockNamedNodeData)) - continue; - - disable_bitmaps = true; - - if (qemuMonitorTransactionBitmapDisable(act, basesrc->nodeformat, - bitmap->name) < 0) - return -1; - - bitmaplist[curbitmapstr++] = g_strdup(bitmap->name); - } - - if (disable_bitmaps) { - *actions = g_steal_pointer(&act); - *disabledBitmapsBase = g_steal_pointer(&bitmaplist); - } - - return 0; -} - - -struct qemuBlockBitmapsHandleCommitData { - bool skip; - bool create; - bool enable; - const char *basenode; - virJSONValuePtr merge; - unsigned long long granularity; - bool persistent; -}; - - -static void -qemuBlockBitmapsHandleCommitDataFree(void *opaque) -{ - struct qemuBlockBitmapsHandleCommitData *data = opaque; - - virJSONValueFree(data->merge); - g_free(data); -} - - -static int -qemuBlockBitmapsHandleCommitFinishIterate(void *payload, - const void *entryname, - void *opaque) -{ - struct qemuBlockBitmapsHandleCommitData *data = payload; - const char *bitmapname = entryname; - virJSONValuePtr actions = opaque; - - if (data->skip) - return 0; - - if (data->create) { - if (qemuMonitorTransactionBitmapAdd(actions, data->basenode, bitmapname, - data->persistent, !data->enable, - data->granularity) < 0) - return -1; - } else { - if (data->enable && - qemuMonitorTransactionBitmapEnable(actions, data->basenode, bitmapname) < 0) - return -1; - } - - if (data->merge && - qemuMonitorTransactionBitmapMerge(actions, data->basenode, bitmapname, - &data->merge) < 0) - return -1; - - return 0; -} - - -/** - * @topsrc: virStorageSource representing 'top' of the job - * @basesrc: virStorageSource representing 'base' of the job + * @active: commit job is an active layer block-commit * @blockNamedNodeData: hash table containing data about bitmaps * @actions: filled with arguments for a 'transaction' command * @disabledBitmapsBase: bitmap names which were disabled @@ -3332,95 +3222,20 @@ qemuBlockBitmapsHandleCommitFinishIterate(void *payload, int qemuBlockBitmapsHandleCommitFinish(virStorageSourcePtr topsrc, virStorageSourcePtr basesrc, + bool active, virHashTablePtr blockNamedNodeData, - virJSONValuePtr *actions, - char **disabledBitmapsBase) + virJSONValuePtr *actions) { - g_autoptr(virJSONValue) act = virJSONValueNewArray(); - virStorageSourcePtr n; - qemuBlockNamedNodeDataPtr entry; - g_autoptr(virHashTable) commitdata = NULL; - struct qemuBlockBitmapsHandleCommitData *bitmapdata; - size_t i; - - commitdata = virHashNew(qemuBlockBitmapsHandleCommitDataFree); - - for (n = topsrc; n != basesrc; n = n->backingStore) { - if (!(entry = virHashLookup(blockNamedNodeData, n->nodeformat))) - continue; - - for (i = 0; i < entry->nbitmaps; i++) { - qemuBlockNamedNodeDataBitmapPtr bitmap = entry->bitmaps[i]; - - if (!(bitmapdata = virHashLookup(commitdata, bitmap->name))) { - bitmapdata = g_new0(struct qemuBlockBitmapsHandleCommitData, 1); + virStorageSourcePtr writebitmapsrc = NULL; - /* we must mirror the state of the topmost bitmap and merge - * everything else */ - bitmapdata->create = true; - bitmapdata->enable = bitmap->recording; - bitmapdata->basenode = basesrc->nodeformat; - bitmapdata->merge = virJSONValueNewArray(); - bitmapdata->granularity = bitmap->granularity; - bitmapdata->persistent = bitmap->persistent; + if (active) + writebitmapsrc = basesrc; - if (virHashAddEntry(commitdata, bitmap->name, bitmapdata) < 0) { - qemuBlockBitmapsHandleCommitDataFree(bitmapdata); - return -1; - } - } - - if (bitmap->inconsistent || - !qemuBlockBitmapChainIsValid(topsrc, bitmap->name, blockNamedNodeData)) - bitmapdata->skip = true; - - if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(bitmapdata->merge, - n->nodeformat, - bitmap->name) < 0) - return -1; - } - } - - if ((entry = virHashLookup(blockNamedNodeData, basesrc->nodeformat))) { - /* note that all bitmaps in 'base' were disabled when commit was started */ - for (i = 0; i < entry->nbitmaps; i++) { - qemuBlockNamedNodeDataBitmapPtr bitmap = entry->bitmaps[i]; - - if ((bitmapdata = virHashLookup(commitdata, bitmap->name))) { - bitmapdata->create = false; - } else { - if (disabledBitmapsBase) { - char **disabledbitmaps; - - for (disabledbitmaps = disabledBitmapsBase; *disabledbitmaps; disabledbitmaps++) { - if (STREQ(*disabledbitmaps, bitmap->name)) { - bitmapdata = g_new0(struct qemuBlockBitmapsHandleCommitData, 1); - - bitmapdata->create = false; - bitmapdata->enable = true; - bitmapdata->basenode = basesrc->nodeformat; - bitmapdata->granularity = bitmap->granularity; - bitmapdata->persistent = bitmap->persistent; - - if (virHashAddEntry(commitdata, bitmap->name, bitmapdata) < 0) { - qemuBlockBitmapsHandleCommitDataFree(bitmapdata); - return -1; - } - - break; - } - } - } - } - } - } - - if (virHashForEach(commitdata, qemuBlockBitmapsHandleCommitFinishIterate, act) < 0) + if (qemuBlockGetBitmapMergeActions(topsrc, basesrc, basesrc, NULL, NULL, + writebitmapsrc, actions, + blockNamedNodeData) < 0) return -1; - if (virJSONValueArraySize(act) > 0) - *actions = g_steal_pointer(&act); - return 0; } diff --git a/src/qemu/qemu_block.h b/src/qemu/qemu_block.h index 0f5b4cfaa9..24b87e79db 100644 --- a/src/qemu/qemu_block.h +++ b/src/qemu/qemu_block.h @@ -243,19 +243,12 @@ qemuBlockBitmapsHandleBlockcopy(virStorageSourcePtr src, bool shallow, virJSONValuePtr *actions); -int -qemuBlockBitmapsHandleCommitStart(virStorageSourcePtr topsrc, - virStorageSourcePtr basesrc, - virHashTablePtr blockNamedNodeData, - virJSONValuePtr *actions, - char ***disabledBitmapsBase); - int qemuBlockBitmapsHandleCommitFinish(virStorageSourcePtr topsrc, virStorageSourcePtr basesrc, + bool active, virHashTablePtr blockNamedNodeData, - virJSONValuePtr *actions, - char **disabledBitmapsBase); + virJSONValuePtr *actions); int qemuBlockReopenReadWrite(virDomainObjPtr vm, diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c index 17dc08476b..8e81b142f4 100644 --- a/src/qemu/qemu_blockjob.c +++ b/src/qemu/qemu_blockjob.c @@ -1069,6 +1069,7 @@ qemuBlockJobProcessEventCompletedCommitBitmaps(virDomainObjPtr vm, qemuDomainObjPrivatePtr priv = vm->privateData; g_autoptr(virHashTable) blockNamedNodeData = NULL; g_autoptr(virJSONValue) actions = NULL; + bool active = job->type == QEMU_BLOCKJOB_TYPE_ACTIVE_COMMIT; if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV_REOPEN)) return 0; @@ -1078,16 +1079,18 @@ qemuBlockJobProcessEventCompletedCommitBitmaps(virDomainObjPtr vm, if (qemuBlockBitmapsHandleCommitFinish(job->data.commit.top, job->data.commit.base, + active, blockNamedNodeData, - &actions, - job->data.commit.disabledBitmapsBase) < 0) + &actions) < 0) return 0; if (!actions) return 0; - if (qemuBlockReopenReadWrite(vm, job->data.commit.base, asyncJob) < 0) - return -1; + if (!active) { + if (qemuBlockReopenReadWrite(vm, job->data.commit.base, asyncJob) < 0) + return -1; + } if (qemuDomainObjEnterMonitorAsync(priv->driver, vm, asyncJob) < 0) return -1; @@ -1097,8 +1100,10 @@ qemuBlockJobProcessEventCompletedCommitBitmaps(virDomainObjPtr vm, if (qemuDomainObjExitMonitor(priv->driver, vm) < 0) return -1; - if (qemuBlockReopenReadOnly(vm, job->data.commit.base, asyncJob) < 0) - return -1; + if (!active) { + if (qemuBlockReopenReadOnly(vm, job->data.commit.base, asyncJob) < 0) + return -1; + } return 0; } @@ -1268,6 +1273,9 @@ qemuBlockJobProcessEventCompletedActiveCommit(virQEMUDriverPtr driver, job->disk->src = job->data.commit.base; job->disk->src->readonly = job->data.commit.top->readonly; + if (qemuBlockJobProcessEventCompletedCommitBitmaps(vm, job, asyncJob) < 0) + return; + qemuBlockJobEventProcessConcludedRemoveChain(driver, vm, asyncJob, job->data.commit.top); if (job->data.commit.deleteCommittedImages) @@ -1339,6 +1347,7 @@ qemuBlockJobProcessEventFailedActiveCommit(virQEMUDriverPtr driver, virDomainObjPtr vm, qemuBlockJobDataPtr job) { + g_autoptr(virJSONValue) actions = virJSONValueNewArray(); virDomainDiskDefPtr disk = job->disk; VIR_DEBUG("active commit job '%s' on VM '%s' failed", job->name, vm->def->name); @@ -1346,6 +1355,10 @@ qemuBlockJobProcessEventFailedActiveCommit(virQEMUDriverPtr driver, if (!disk) return; + ignore_value(qemuMonitorTransactionBitmapRemove(actions, disk->mirror->nodeformat, + "libvirt-tmp-activewrite")); + + /* Ideally, we would make the backing chain read only again (yes, SELinux * can do that using different labels). But that is not implemented yet and * not leaking security driver metadata is more important. */ diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 88517ba6a7..6256046762 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -17312,9 +17312,9 @@ qemuDomainBlockPivot(virQEMUDriverPtr driver, case QEMU_BLOCKJOB_TYPE_COPY: if (blockdev && !job->jobflagsmissing) { - g_autoptr(virHashTable) blockNamedNodeData = NULL; bool shallow = job->jobflags & VIR_DOMAIN_BLOCK_COPY_SHALLOW; bool reuse = job->jobflags & VIR_DOMAIN_BLOCK_COPY_REUSE_EXT; + g_autoptr(virHashTable) blockNamedNodeData = NULL; if (!(blockNamedNodeData = qemuBlockGetNamedNodeData(vm, QEMU_ASYNC_JOB_NONE))) return -1; @@ -17352,16 +17352,15 @@ qemuDomainBlockPivot(virQEMUDriverPtr driver, * the bitmaps if it wasn't present thus must skip this */ if (blockdev && virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV_REOPEN)) { - g_autoptr(virHashTable) blockNamedNodeData = NULL; - if (!(blockNamedNodeData = qemuBlockGetNamedNodeData(vm, QEMU_ASYNC_JOB_NONE))) - return -1; + actions = virJSONValueNewArray(); - if (qemuBlockBitmapsHandleCommitFinish(job->data.commit.top, - job->data.commit.base, - blockNamedNodeData, - &actions, - job->data.commit.disabledBitmapsBase) < 0) + if (qemuMonitorTransactionBitmapAdd(actions, + job->data.commit.base->nodeformat, + "libvirt-tmp-activewrite", + false, + false, + 0) < 0) return -1; } @@ -18482,7 +18481,6 @@ qemuDomainBlockCommit(virDomainPtr dom, const char *nodebase = NULL; bool persistjob = false; bool blockdev = false; - g_autoptr(virJSONValue) bitmapDisableActions = NULL; VIR_AUTOSTRINGLIST bitmapDisableList = NULL; virCheckFlags(VIR_DOMAIN_BLOCK_COMMIT_SHALLOW | @@ -18647,27 +18645,6 @@ qemuDomainBlockCommit(virDomainPtr dom, goto endjob; } - if (blockdev && - virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV_REOPEN)) { - g_autoptr(virHashTable) blockNamedNodeData = NULL; - if (!(blockNamedNodeData = qemuBlockGetNamedNodeData(vm, QEMU_ASYNC_JOB_NONE))) - goto endjob; - - if (qemuBlockBitmapsHandleCommitStart(topSource, baseSource, - blockNamedNodeData, - &bitmapDisableActions, - &bitmapDisableList) < 0) - goto endjob; - - /* if we don't have terminator on 'base' we can't reopen it */ - if (bitmapDisableActions && !baseSource->backingStore) { - virReportError(VIR_ERR_OPERATION_UNSUPPORTED, - _("can't handle bitmaps on unterminated backing image '%s'"), - base); - goto endjob; - } - } - if (!(job = qemuBlockJobDiskNewCommit(vm, disk, top_parent, topSource, baseSource, &bitmapDisableList, flags & VIR_DOMAIN_BLOCK_COMMIT_DELETE, @@ -18692,23 +18669,6 @@ qemuDomainBlockCommit(virDomainPtr dom, !(backingPath = qemuBlockGetBackingStoreString(baseSource, false))) goto endjob; - if (bitmapDisableActions) { - int rc; - - if (qemuBlockReopenReadWrite(vm, baseSource, QEMU_ASYNC_JOB_NONE) < 0) - goto endjob; - - qemuDomainObjEnterMonitor(driver, vm); - rc = qemuMonitorTransaction(priv->mon, &bitmapDisableActions); - if (qemuDomainObjExitMonitor(driver, vm) < 0) - goto endjob; - - if (qemuBlockReopenReadOnly(vm, baseSource, QEMU_ASYNC_JOB_NONE) < 0) - goto endjob; - - if (rc < 0) - goto endjob; - } } else { device = job->name; } diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index fd6dff82f9..5be98b9320 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -928,12 +928,11 @@ testQemuBlockBitmapBlockcommit(const void *opaque) g_autofree char *actual = NULL; g_autofree char *expectpath = NULL; - g_autoptr(virJSONValue) actionsDisable = NULL; g_autoptr(virJSONValue) actionsMerge = NULL; g_autoptr(virJSONValue) nodedatajson = NULL; g_autoptr(virHashTable) nodedata = NULL; g_auto(virBuffer) buf = VIR_BUFFER_INITIALIZER; - VIR_AUTOSTRINGLIST bitmapsDisable = NULL; + bool active = data->top == data->chain; expectpath = g_strdup_printf("%s/%s%s", abs_srcdir, blockcommitPrefix, data->name); @@ -947,20 +946,10 @@ testQemuBlockBitmapBlockcommit(const void *opaque) return -1; } - if (qemuBlockBitmapsHandleCommitStart(data->top, data->base, nodedata, - &actionsDisable, &bitmapsDisable) < 0) - return -1; - - virBufferAddLit(&buf, "pre job bitmap disable:\n"); - - if (actionsDisable && - virJSONValueToBuffer(actionsDisable, &buf, true) < 0) - return -1; - virBufferAddLit(&buf, "merge bitmpas:\n"); - if (qemuBlockBitmapsHandleCommitFinish(data->top, data->base, nodedata, - &actionsMerge, bitmapsDisable) < 0) + if (qemuBlockBitmapsHandleCommitFinish(data->top, data->base, active, nodedata, + &actionsMerge) < 0) return -1; if (actionsMerge && @@ -1357,6 +1346,7 @@ mymain(void) #define TEST_BITMAP_BLOCKCOMMIT(testname, topimg, baseimg, ndf) \ do {\ blockbitmapblockcommitdata.name = testname; \ + blockbitmapblockcommitdata.chain = bitmapSourceChain; \ blockbitmapblockcommitdata.top = testQemuBitmapGetFakeChainEntry(bitmapSourceChain, topimg); \ blockbitmapblockcommitdata.base = testQemuBitmapGetFakeChainEntry(bitmapSourceChain, baseimg); \ blockbitmapblockcommitdata.nodedatafile = ndf; \ diff --git a/tests/qemublocktestdata/bitmapblockcommit/empty b/tests/qemublocktestdata/bitmapblockcommit/empty index bfc58f994e..9260011852 100644 --- a/tests/qemublocktestdata/bitmapblockcommit/empty +++ b/tests/qemublocktestdata/bitmapblockcommit/empty @@ -1,2 +1 @@ -pre job bitmap disable: merge bitmpas: -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc
s/allows to remove/allows the removal of/
implementatio of bitmap merging for block commit. The new approach is
implementation
way simpler and more robust and also allows us to get rid of the disabling of bitmaps done prior to the start as we actually do want to update the bitmaps in the base.
What if the commit fails, but we have merged in portions of the bitmap? In practice, this merely means that a future attempt to use that bitmap in the backing file will now copy more data than previously necessary; what's more, if we try a commit again (if the failure was transient), we'll be setting those same bits again. So yeah, I don't think it hurts to not disable the bitmaps in the backing files.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 203 +----------------- src/qemu/qemu_block.h | 11 +- src/qemu/qemu_blockjob.c | 25 ++- src/qemu/qemu_driver.c | 56 +---- tests/qemublocktest.c | 18 +- .../qemublocktestdata/bitmapblockcommit/empty | 1 - 6 files changed, 42 insertions(+), 272 deletions(-)
@@ -3332,95 +3222,20 @@ qemuBlockBitmapsHandleCommitFinishIterate(void *payload, int qemuBlockBitmapsHandleCommitFinish(virStorageSourcePtr topsrc, virStorageSourcePtr basesrc, + bool active, virHashTablePtr blockNamedNodeData, - virJSONValuePtr *actions, - char **disabledBitmapsBase) + virJSONValuePtr *actions) {
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Thu, Jun 18, 2020 at 18:56:15 -0500, Eric Blake wrote:
On 6/15/20 12:10 PM, Peter Krempa wrote:
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc
s/allows to remove/allows the removal of/
implementatio of bitmap merging for block commit. The new approach is
implementation
way simpler and more robust and also allows us to get rid of the disabling of bitmaps done prior to the start as we actually do want to update the bitmaps in the base.
What if the commit fails, but we have merged in portions of the bitmap? In practice, this merely means that a future attempt to use that bitmap in the backing file will now copy more data than previously necessary; what's more, if we try a commit again (if the failure was transient), we'll be setting those same bits again. So yeah, I don't think it hurts to not disable the bitmaps in the backing files.
Well actually the bitmaps are merged only after the commit has finished so we update them only once it's done. In the meanwhile all of them which were present in the image before will get auto-updated with the commited blocks which is IMO also a good thing. Regardless, from the full chain point of view it will still look correct and after a commit failure, the partial chain between the comitted images is not valid anyways so we can't do much.
In the 'basic case we have few bitmaps in only the top layer. Simulate commit into the backing of the top layer and also 2 levels deep. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 + .../bitmapblockcommit/basic-1-2 | 145 ++++++++++++++++++ .../bitmapblockcommit/basic-1-3 | 145 ++++++++++++++++++ .../bitmapblockcommit/basic-2-3 | 1 + 4 files changed, 295 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-1-2 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-1-3 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-2-3 diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 5be98b9320..1a8ce3b73d 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1358,6 +1358,10 @@ mymain(void) TEST_BITMAP_BLOCKCOMMIT("empty", 1, 2, "empty"); + TEST_BITMAP_BLOCKCOMMIT("basic-1-2", 1, 2, "basic"); + TEST_BITMAP_BLOCKCOMMIT("basic-1-3", 1, 3, "basic"); + TEST_BITMAP_BLOCKCOMMIT("basic-2-3", 2, 3, "basic"); + cleanup: qemuTestDriverFree(&driver); VIR_FREE(capslatest_x86_64); diff --git a/tests/qemublocktestdata/bitmapblockcommit/basic-1-2 b/tests/qemublocktestdata/bitmapblockcommit/basic-1-2 new file mode 100644 index 0000000000..d393d31044 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/basic-1-2 @@ -0,0 +1,145 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "a", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/basic-1-3 b/tests/qemublocktestdata/bitmapblockcommit/basic-1-3 new file mode 100644 index 0000000000..3fa902c354 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/basic-1-3 @@ -0,0 +1,145 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "a", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/basic-2-3 b/tests/qemublocktestdata/bitmapblockcommit/basic-2-3 new file mode 100644 index 0000000000..9260011852 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/basic-2-3 @@ -0,0 +1 @@ +merge bitmpas: -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
In the 'basic case we have few bitmaps in only the top layer. Simulate
stray or missing '
commit into the backing of the top layer and also 2 levels deep.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 4 + .../bitmapblockcommit/basic-1-2 | 145 ++++++++++++++++++ .../bitmapblockcommit/basic-1-3 | 145 ++++++++++++++++++ .../bitmapblockcommit/basic-2-3 | 1 + 4 files changed, 295 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-1-2 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-1-3 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/basic-2-3
+++ b/tests/qemublocktestdata/bitmapblockcommit/basic-1-2 @@ -0,0 +1,145 @@ +merge bitmpas:
bitmaps (fixing the source of the typo probably has ripple effects in expected output, and might even be in a prior patch...)
+[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "a", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + }
Hmm, so you are merging the 'libvirt-tmp-activewrite' bitmap into bitmap (libvirt-2-format,a)...
+ ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite"
...and (libvirt-2-format,b) simultaneously. But that still makes sense: if libvirt-tmp-activewrite contains bits for everything that has changed since a certain point in time, and both bitmap a and b will be left active when the operation is done, they will both need those same bits merged in. It took me a while to see why you were reusing a bitmap as a source to more than one destination, but it looks correct. With the typo fixed, Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Simulate commit between all the combinations of layers in the 'snapshots' case to see whether the code merges the correct bitmaps with the correct depth of temporary bitmaps. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 14 ++ .../bitmapblockcommit/snapshots-1-2 | 105 +++++++++++ .../bitmapblockcommit/snapshots-1-3 | 131 ++++++++++++++ .../bitmapblockcommit/snapshots-1-4 | 163 +++++++++++++++++ .../bitmapblockcommit/snapshots-1-5 | 167 ++++++++++++++++++ .../bitmapblockcommit/snapshots-2-3 | 65 +++++++ .../bitmapblockcommit/snapshots-2-4 | 97 ++++++++++ .../bitmapblockcommit/snapshots-2-5 | 101 +++++++++++ .../bitmapblockcommit/snapshots-3-4 | 62 +++++++ .../bitmapblockcommit/snapshots-3-5 | 66 +++++++ .../bitmapblockcommit/snapshots-4-5 | 16 ++ 11 files changed, 987 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 1a8ce3b73d..4a737e19d3 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1362,6 +1362,20 @@ mymain(void) TEST_BITMAP_BLOCKCOMMIT("basic-1-3", 1, 3, "basic"); TEST_BITMAP_BLOCKCOMMIT("basic-2-3", 2, 3, "basic"); + TEST_BITMAP_BLOCKCOMMIT("snapshots-1-2", 1, 2, "snapshots"); + TEST_BITMAP_BLOCKCOMMIT("snapshots-1-3", 1, 3, "snapshots"); + TEST_BITMAP_BLOCKCOMMIT("snapshots-1-4", 1, 4, "snapshots"); + TEST_BITMAP_BLOCKCOMMIT("snapshots-1-5", 1, 5, "snapshots"); + + TEST_BITMAP_BLOCKCOMMIT("snapshots-2-3", 2, 3, "snapshots"); + TEST_BITMAP_BLOCKCOMMIT("snapshots-2-4", 2, 4, "snapshots"); + TEST_BITMAP_BLOCKCOMMIT("snapshots-2-5", 2, 5, "snapshots"); + + TEST_BITMAP_BLOCKCOMMIT("snapshots-3-4", 3, 4, "snapshots"); + TEST_BITMAP_BLOCKCOMMIT("snapshots-3-5", 3, 5, "snapshots"); + + TEST_BITMAP_BLOCKCOMMIT("snapshots-4-5", 4, 5, "snapshots"); + cleanup: qemuTestDriverFree(&driver); VIR_FREE(capslatest_x86_64); diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 new file mode 100644 index 0000000000..7d5b0a4402 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 @@ -0,0 +1,105 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-2-format", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-2-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-2-format", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 new file mode 100644 index 0000000000..17a020ba20 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 @@ -0,0 +1,131 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "libvirt-2-format", + "name": "c" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-2-format", + "name": "b" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-2-format", + "name": "d" + }, + { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-3-format", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 new file mode 100644 index 0000000000..972d30b2e1 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 @@ -0,0 +1,163 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "libvirt-4-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "libvirt-2-format", + "name": "c" + }, + { + "node": "libvirt-3-format", + "name": "c" + }, + { + "node": "libvirt-4-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-2-format", + "name": "b" + }, + { + "node": "libvirt-3-format", + "name": "b" + }, + { + "node": "libvirt-4-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "a" + }, + { + "node": "libvirt-4-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-2-format", + "name": "d" + }, + { + "node": "libvirt-4-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-4-format", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 new file mode 100644 index 0000000000..3cda5b4132 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 @@ -0,0 +1,167 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "libvirt-5-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "libvirt-2-format", + "name": "c" + }, + { + "node": "libvirt-3-format", + "name": "c" + }, + { + "node": "libvirt-5-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-2-format", + "name": "b" + }, + { + "node": "libvirt-3-format", + "name": "b" + }, + { + "node": "libvirt-5-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "a" + }, + { + "node": "libvirt-4-format", + "name": "a" + }, + { + "node": "libvirt-5-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-2-format", + "name": "d" + }, + { + "node": "libvirt-5-format", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "libvirt-5-format", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 new file mode 100644 index 0000000000..cb952eeb06 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 @@ -0,0 +1,65 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-3-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "d" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "a" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "b" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-3-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "c" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 new file mode 100644 index 0000000000..a540ca42e3 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 @@ -0,0 +1,97 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "d" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "a" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "b" + }, + { + "node": "libvirt-3-format", + "name": "b" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "c" + }, + { + "node": "libvirt-3-format", + "name": "c" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 new file mode 100644 index 0000000000..8ea6036a33 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 @@ -0,0 +1,101 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "d" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "a" + }, + { + "node": "libvirt-4-format", + "name": "a" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "b" + }, + { + "node": "libvirt-3-format", + "name": "b" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-2-format", + "name": "c" + }, + { + "node": "libvirt-3-format", + "name": "c" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 new file mode 100644 index 0000000000..736d4fd9df --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 @@ -0,0 +1,62 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-3-format", + "name": "c" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-4-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-3-format", + "name": "b" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-4-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-3-format", + "name": "a" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 new file mode 100644 index 0000000000..207d82e6b6 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 @@ -0,0 +1,66 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-3-format", + "name": "c" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "libvirt-5-format", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-3-format", + "name": "b" + } + ] + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-3-format", + "name": "a" + }, + { + "node": "libvirt-4-format", + "name": "a" + } + ] + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 b/tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 new file mode 100644 index 0000000000..4a8e6c5cff --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5 @@ -0,0 +1,16 @@ +merge bitmpas: +[ + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "libvirt-5-format", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-4-format", + "name": "a" + } + ] + } + } +] -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Simulate commit between all the combinations of layers in the 'snapshots' case to see whether the code merges the correct bitmaps with the correct depth of temporary bitmaps.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 14 ++ .../bitmapblockcommit/snapshots-1-2 | 105 +++++++++++ .../bitmapblockcommit/snapshots-1-3 | 131 ++++++++++++++ .../bitmapblockcommit/snapshots-1-4 | 163 +++++++++++++++++ .../bitmapblockcommit/snapshots-1-5 | 167 ++++++++++++++++++ .../bitmapblockcommit/snapshots-2-3 | 65 +++++++ .../bitmapblockcommit/snapshots-2-4 | 97 ++++++++++ .../bitmapblockcommit/snapshots-2-5 | 101 +++++++++++ .../bitmapblockcommit/snapshots-3-4 | 62 +++++++ .../bitmapblockcommit/snapshots-3-5 | 66 +++++++ .../bitmapblockcommit/snapshots-4-5 | 16 ++ 11 files changed, 987 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-2 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-3 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-4 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-1-5 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-3 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-4 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-2-5 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-3-4 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-3-5 create mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-4-5
Lots of combinations, but that serves for good regression prevention. Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
New semantics of the bitmap handling don't need this. Remove the field and all uses of it including the status XML. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_blockjob.c | 44 ------------------- src/qemu/qemu_blockjob.h | 3 -- src/qemu/qemu_domain.c | 22 ---------- src/qemu/qemu_driver.c | 3 +- .../blockjob-blockdev-in.xml | 4 -- 5 files changed, 1 insertion(+), 75 deletions(-) diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c index 8e81b142f4..58ba3e6e96 100644 --- a/src/qemu/qemu_blockjob.c +++ b/src/qemu/qemu_blockjob.c @@ -84,11 +84,6 @@ qemuBlockJobDataDisposeJobdata(qemuBlockJobDataPtr job) virObjectUnref(job->data.backup.store); g_free(job->data.backup.bitmap); } - - if (job->type == QEMU_BLOCKJOB_TYPE_COMMIT || - job->type == QEMU_BLOCKJOB_TYPE_ACTIVE_COMMIT) { - virStringListFree(job->data.commit.disabledBitmapsBase); - } } @@ -290,7 +285,6 @@ qemuBlockJobDiskNewCommit(virDomainObjPtr vm, virStorageSourcePtr topparent, virStorageSourcePtr top, virStorageSourcePtr base, - char ***disabledBitmapsBase, bool delete_imgs, unsigned int jobflags) { @@ -316,7 +310,6 @@ qemuBlockJobDiskNewCommit(virDomainObjPtr vm, job->data.commit.top = top; job->data.commit.base = base; job->data.commit.deleteCommittedImages = delete_imgs; - job->data.commit.disabledBitmapsBase = g_steal_pointer(disabledBitmapsBase); job->jobflags = jobflags; if (qemuBlockJobRegister(job, vm, disk, true) < 0) @@ -1369,40 +1362,6 @@ qemuBlockJobProcessEventFailedActiveCommit(virQEMUDriverPtr driver, } -static void -qemuBlockJobProcessEventFailedCommitCommon(virDomainObjPtr vm, - qemuBlockJobDataPtr job, - qemuDomainAsyncJob asyncJob) -{ - qemuDomainObjPrivatePtr priv = vm->privateData; - g_autoptr(virJSONValue) actions = virJSONValueNewArray(); - char **disabledBitmaps = job->data.commit.disabledBitmapsBase; - - if (!disabledBitmaps || !*disabledBitmaps) - return; - - for (; *disabledBitmaps; disabledBitmaps++) { - qemuMonitorTransactionBitmapEnable(actions, - job->data.commit.base->nodeformat, - *disabledBitmaps); - } - - if (qemuBlockReopenReadWrite(vm, job->data.commit.base, asyncJob) < 0) - return; - - if (qemuDomainObjEnterMonitorAsync(priv->driver, vm, asyncJob) < 0) - return; - - qemuMonitorTransaction(priv->mon, &actions); - - if (qemuDomainObjExitMonitor(priv->driver, vm) < 0) - return; - - if (qemuBlockReopenReadOnly(vm, job->data.commit.base, asyncJob) < 0) - return; -} - - static void qemuBlockJobProcessEventConcludedCreate(virQEMUDriverPtr driver, virDomainObjPtr vm, @@ -1510,8 +1469,6 @@ qemuBlockJobEventProcessConcludedTransition(qemuBlockJobDataPtr job, case QEMU_BLOCKJOB_TYPE_COMMIT: if (success) qemuBlockJobProcessEventCompletedCommit(driver, vm, job, asyncJob); - else - qemuBlockJobProcessEventFailedCommitCommon(vm, job, asyncJob); break; case QEMU_BLOCKJOB_TYPE_ACTIVE_COMMIT: @@ -1519,7 +1476,6 @@ qemuBlockJobEventProcessConcludedTransition(qemuBlockJobDataPtr job, qemuBlockJobProcessEventCompletedActiveCommit(driver, vm, job, asyncJob); } else { qemuBlockJobProcessEventFailedActiveCommit(driver, vm, job); - qemuBlockJobProcessEventFailedCommitCommon(vm, job, asyncJob); } break; diff --git a/src/qemu/qemu_blockjob.h b/src/qemu/qemu_blockjob.h index 19498b5bd8..9f73a3547c 100644 --- a/src/qemu/qemu_blockjob.h +++ b/src/qemu/qemu_blockjob.h @@ -88,8 +88,6 @@ struct _qemuBlockJobCommitData { virStorageSourcePtr top; virStorageSourcePtr base; bool deleteCommittedImages; - char **disabledBitmapsBase; /* a NULL-terminated list of bitmap names which - were disabled in @base for the commit job */ }; @@ -187,7 +185,6 @@ qemuBlockJobDiskNewCommit(virDomainObjPtr vm, virStorageSourcePtr topparent, virStorageSourcePtr top, virStorageSourcePtr base, - char ***disabledBitmapsBase, bool delete_imgs, unsigned int jobflags); diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 2dad823a86..0548e9c2e6 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -2623,7 +2623,6 @@ qemuDomainPrivateBlockJobFormatCommit(qemuBlockJobDataPtr job, virBufferPtr buf) { g_auto(virBuffer) disabledBitmapsBuf = VIR_BUFFER_INIT_CHILD(buf); - char **bitmaps = job->data.commit.disabledBitmapsBase; if (job->data.commit.base) virBufferAsprintf(buf, "<base node='%s'/>\n", job->data.commit.base->nodeformat); @@ -2637,9 +2636,6 @@ qemuDomainPrivateBlockJobFormatCommit(qemuBlockJobDataPtr job, if (job->data.commit.deleteCommittedImages) virBufferAddLit(buf, "<deleteCommittedImages/>\n"); - while (bitmaps && *bitmaps) - virBufferEscapeString(&disabledBitmapsBuf, "<bitmap name='%s'/>\n", *(bitmaps++)); - virXMLFormatElement(buf, "disabledBaseBitmaps", NULL, &disabledBitmapsBuf); } @@ -3260,9 +3256,6 @@ static int qemuDomainObjPrivateXMLParseBlockjobDataCommit(qemuBlockJobDataPtr job, xmlXPathContextPtr ctxt) { - g_autofree xmlNodePtr *nodes = NULL; - ssize_t nnodes; - if (job->type == QEMU_BLOCKJOB_TYPE_COMMIT) { qemuDomainObjPrivateXMLParseBlockjobNodename(job, "string(./topparent/@node)", @@ -3289,21 +3282,6 @@ qemuDomainObjPrivateXMLParseBlockjobDataCommit(qemuBlockJobDataPtr job, !job->data.commit.base) return -1; - if ((nnodes = virXPathNodeSet("./disabledBaseBitmaps/bitmap", ctxt, &nodes)) > 0) { - size_t i; - - job->data.commit.disabledBitmapsBase = g_new0(char *, nnodes + 1); - - for (i = 0; i < nnodes; i++) { - char *tmp; - - if (!(tmp = virXMLPropString(nodes[i], "name"))) - return -1; - - job->data.commit.disabledBitmapsBase[i] = g_steal_pointer(&tmp); - } - } - return 0; } diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 6256046762..8cf3629a80 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -18481,7 +18481,6 @@ qemuDomainBlockCommit(virDomainPtr dom, const char *nodebase = NULL; bool persistjob = false; bool blockdev = false; - VIR_AUTOSTRINGLIST bitmapDisableList = NULL; virCheckFlags(VIR_DOMAIN_BLOCK_COMMIT_SHALLOW | VIR_DOMAIN_BLOCK_COMMIT_ACTIVE | @@ -18646,7 +18645,7 @@ qemuDomainBlockCommit(virDomainPtr dom, } if (!(job = qemuBlockJobDiskNewCommit(vm, disk, top_parent, topSource, - baseSource, &bitmapDisableList, + baseSource, flags & VIR_DOMAIN_BLOCK_COMMIT_DELETE, flags))) goto endjob; diff --git a/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml b/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml index cc17a17ff4..ca6d110179 100644 --- a/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml +++ b/tests/qemustatusxml2xmldata/blockjob-blockdev-in.xml @@ -243,10 +243,6 @@ <base node='libvirt-19-format'/> <top node='libvirt-17-format'/> <deleteCommittedImages/> - <disabledBaseBitmaps> - <bitmap name='test'/> - <bitmap name='test1'/> - </disabledBaseBitmaps> </blockjob> <blockjob name='create-libvirt-1337-storage' type='create' state='running'> <create mode='storage'/> -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
New semantics of the bitmap handling don't need this. Remove the field and all uses of it including the status XML.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_blockjob.c | 44 ------------------- src/qemu/qemu_blockjob.h | 3 -- src/qemu/qemu_domain.c | 22 ---------- src/qemu/qemu_driver.c | 3 +- .../blockjob-blockdev-in.xml | 4 -- 5 files changed, 1 insertion(+), 75 deletions(-)
Reviewed-by: Eric Blake <eblake@redhat.com> Another sign of how the new paradigm is simpler. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc implementatio of bitmap merging for block copy. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 113 ++------------------------------------- src/qemu/qemu_blockjob.c | 40 ++++++++++++++ src/qemu/qemu_driver.c | 13 ++--- tests/qemublocktest.c | 6 ++- 4 files changed, 57 insertions(+), 115 deletions(-) diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index e958fb71fa..0480f6b97d 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -3068,38 +3068,6 @@ qemuBlockBitmapChainIsValid(virStorageSourcePtr src, } -struct qemuBlockBitmapsHandleBlockcopyConcatData { - virHashTablePtr bitmaps_merge; - virJSONValuePtr actions; - const char *mirrornodeformat; - bool has_bitmaps; -}; - - -static int -qemuBlockBitmapsHandleBlockcopyConcatActions(void *payload, - const void *name, - void *opaque) -{ - struct qemuBlockBitmapsHandleBlockcopyConcatData *data = opaque; - virJSONValuePtr createactions = payload; - const char *bitmapname = name; - g_autoptr(virJSONValue) mergebitmaps = virHashSteal(data->bitmaps_merge, bitmapname); - - data->has_bitmaps = true; - - virJSONValueArrayConcat(data->actions, createactions); - - if (qemuMonitorTransactionBitmapMerge(data->actions, - data->mirrornodeformat, - bitmapname, - &mergebitmaps) < 0) - return -1; - - return 0; -} - - /** * qemuBlockBitmapsHandleBlockcopy: * @src: disk source @@ -3122,86 +3090,15 @@ qemuBlockBitmapsHandleBlockcopy(virStorageSourcePtr src, bool shallow, virJSONValuePtr *actions) { - g_autoptr(virHashTable) bitmaps = virHashNew(virJSONValueHashFree); - g_autoptr(virHashTable) bitmaps_merge = virHashNew(virJSONValueHashFree); - g_autoptr(virHashTable) bitmaps_skip = virHashNew(NULL); - g_autoptr(virJSONValue) tmpactions = virJSONValueNewArray(); - qemuBlockNamedNodeDataPtr entry; - virStorageSourcePtr n; - size_t i; - struct qemuBlockBitmapsHandleBlockcopyConcatData data = { .bitmaps_merge = bitmaps_merge, - .actions = tmpactions, - .mirrornodeformat = mirror->nodeformat, - .has_bitmaps = false, }; - - for (n = src; n; n = n->backingStore) { - if (!(entry = virHashLookup(blockNamedNodeData, n->nodeformat))) - continue; - - for (i = 0; i < entry->nbitmaps; i++) { - qemuBlockNamedNodeDataBitmapPtr bitmap = entry->bitmaps[i]; - virJSONValuePtr bitmap_merge; - - if (virHashHasEntry(bitmaps_skip, bitmap->name)) - continue; - - if (!(bitmap_merge = virHashLookup(bitmaps_merge, bitmap->name))) { - g_autoptr(virJSONValue) tmp = NULL; - bool disabled = !bitmap->recording; - - /* disable any non top-layer bitmaps */ - if (n != src) - disabled = true; - - if (!bitmap->persistent || - !(qemuBlockBitmapChainIsValid(n, bitmap->name, - blockNamedNodeData))) { - ignore_value(virHashAddEntry(bitmaps_skip, bitmap->name, NULL)); - continue; - } - - /* prepare the data for adding the bitmap to the mirror */ - tmp = virJSONValueNewArray(); - - if (qemuMonitorTransactionBitmapAdd(tmp, - mirror->nodeformat, - bitmap->name, - true, - disabled, - bitmap->granularity) < 0) - return -1; + virStorageSourcePtr base = NULL; - if (virHashAddEntry(bitmaps, bitmap->name, tmp) < 0) - return -1; - - tmp = NULL; - - /* prepare array for merging all the bitmaps from the original chain */ - tmp = virJSONValueNewArray(); - - if (virHashAddEntry(bitmaps_merge, bitmap->name, tmp) < 0) - return -1; - - bitmap_merge = g_steal_pointer(&tmp); - } + if (shallow) + base = src->backingStore; - if (qemuMonitorTransactionBitmapMergeSourceAddBitmap(bitmap_merge, - n->nodeformat, - bitmap->name) < 0) - return -1; - } - - if (shallow) - break; - } - - if (virHashForEach(bitmaps, qemuBlockBitmapsHandleBlockcopyConcatActions, - &data) < 0) + if (qemuBlockGetBitmapMergeActions(src, base, mirror, NULL, NULL, mirror, actions, + blockNamedNodeData) < 0) return -1; - if (data.has_bitmaps) - *actions = g_steal_pointer(&tmpactions); - return 0; } diff --git a/src/qemu/qemu_blockjob.c b/src/qemu/qemu_blockjob.c index 58ba3e6e96..7e4530f48b 100644 --- a/src/qemu/qemu_blockjob.c +++ b/src/qemu/qemu_blockjob.c @@ -1282,6 +1282,43 @@ qemuBlockJobProcessEventCompletedActiveCommit(virQEMUDriverPtr driver, } +static int +qemuBlockJobProcessEventCompletedCopyBitmaps(virDomainObjPtr vm, + qemuBlockJobDataPtr job, + qemuDomainAsyncJob asyncJob) +{ + qemuDomainObjPrivatePtr priv = vm->privateData; + g_autoptr(virHashTable) blockNamedNodeData = NULL; + g_autoptr(virJSONValue) actions = NULL; + bool shallow = job->jobflags & VIR_DOMAIN_BLOCK_COPY_SHALLOW; + + if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV_REOPEN)) + return 0; + + if (!(blockNamedNodeData = qemuBlockGetNamedNodeData(vm, asyncJob))) + return -1; + + if (qemuBlockBitmapsHandleBlockcopy(job->disk->src, + job->disk->mirror, + blockNamedNodeData, + shallow, + &actions) < 0) + return 0; + + if (!actions) + return 0; + + if (qemuDomainObjEnterMonitorAsync(priv->driver, vm, asyncJob) < 0) + return -1; + + qemuMonitorTransaction(priv->mon, &actions); + + if (qemuDomainObjExitMonitor(priv->driver, vm) < 0) + return -1; + + return 0; +} + static void qemuBlockJobProcessEventConcludedCopyPivot(virQEMUDriverPtr driver, virDomainObjPtr vm, @@ -1296,6 +1333,8 @@ qemuBlockJobProcessEventConcludedCopyPivot(virQEMUDriverPtr driver, !job->disk->mirror) return; + qemuBlockJobProcessEventCompletedCopyBitmaps(vm, job, asyncJob); + /* for shallow copy without reusing external image the user can either not * specify the backing chain in which case libvirt will open and use the * chain the user provided or not specify a chain in which case we'll @@ -1329,6 +1368,7 @@ qemuBlockJobProcessEventConcludedCopyAbort(virQEMUDriverPtr driver, !job->disk->mirror) return; + /* activeWrite bitmap is removed automatically here */ qemuBlockJobEventProcessConcludedRemoveChain(driver, vm, asyncJob, job->disk->mirror); virObjectUnref(job->disk->mirror); job->disk->mirror = NULL; diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 8cf3629a80..c4d8a13004 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -17314,14 +17314,15 @@ qemuDomainBlockPivot(virQEMUDriverPtr driver, if (blockdev && !job->jobflagsmissing) { bool shallow = job->jobflags & VIR_DOMAIN_BLOCK_COPY_SHALLOW; bool reuse = job->jobflags & VIR_DOMAIN_BLOCK_COPY_REUSE_EXT; - g_autoptr(virHashTable) blockNamedNodeData = NULL; - if (!(blockNamedNodeData = qemuBlockGetNamedNodeData(vm, QEMU_ASYNC_JOB_NONE))) - return -1; + actions = virJSONValueNewArray(); - if (qemuBlockBitmapsHandleBlockcopy(disk->src, disk->mirror, - blockNamedNodeData, - shallow, &actions) < 0) + if (qemuMonitorTransactionBitmapAdd(actions, + disk->mirror->nodeformat, + "libvirt-tmp-activewrite", + false, + false, + 0) < 0) return -1; /* Open and install the backing chain of 'mirror' late if we can use diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 4a737e19d3..1c6517646b 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -881,6 +881,7 @@ testQemuBlockBitmapBlockcopy(const void *opaque) g_autoptr(virJSONValue) nodedatajson = NULL; g_autoptr(virHashTable) nodedata = NULL; g_autoptr(virStorageSource) fakemirror = virStorageSourceNew(); + g_auto(virBuffer) buf = VIR_BUFFER_INITIALIZER; if (!fakemirror) return -1; @@ -903,10 +904,13 @@ testQemuBlockBitmapBlockcopy(const void *opaque) data->shallow, &actions) < 0) return -1; + if (actions && - !(actual = virJSONValueToString(actions, true))) + virJSONValueToBuffer(actions, &buf, true) < 0) return -1; + actual = virBufferContentAndReset(&buf); + return virTestCompareToFile(actual, expectpath); } -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc
allows the removal of
implementatio of bitmap merging for block copy.
implementation
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- src/qemu/qemu_block.c | 113 ++------------------------------------- src/qemu/qemu_blockjob.c | 40 ++++++++++++++ src/qemu/qemu_driver.c | 13 ++--- tests/qemublocktest.c | 6 ++- 4 files changed, 57 insertions(+), 115 deletions(-)
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Fri, Jun 19, 2020 at 09:50:16 -0500, Eric Blake wrote:
On 6/15/20 12:10 PM, Peter Krempa wrote:
Reuse qemuBlockGetBitmapMergeActions which allows to remove the ad-hoc
allows the removal of
implementatio of bitmap merging for block copy.
implementation
Too much copy and paste :)
Test both 'basic' and 'snapshots' cases on shallow and deep copy modes. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 5 + .../bitmapblockcopy/basic-deep-out.json | 144 ++++++++++++++ .../bitmapblockcopy/basic-shallow-out.json | 144 ++++++++++++++ .../bitmapblockcopy/snapshots-deep-out.json | 180 ++++++++++++++++++ .../snapshots-shallow-out.json | 144 ++++++++++++++ 5 files changed, 617 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json diff --git a/tests/qemublocktest.c b/tests/qemublocktest.c index 1c6517646b..0ab599dd67 100644 --- a/tests/qemublocktest.c +++ b/tests/qemublocktest.c @@ -1346,6 +1346,11 @@ mymain(void) TEST_BITMAP_BLOCKCOPY("empty-shallow", true, "empty"); TEST_BITMAP_BLOCKCOPY("empty-deep", false, "empty"); + TEST_BITMAP_BLOCKCOPY("basic-shallow", true, "basic"); + TEST_BITMAP_BLOCKCOPY("basic-deep", false, "basic"); + + TEST_BITMAP_BLOCKCOPY("snapshots-shallow", true, "snapshots"); + TEST_BITMAP_BLOCKCOPY("snapshots-deep", false, "snapshots"); #define TEST_BITMAP_BLOCKCOMMIT(testname, topimg, baseimg, ndf) \ do {\ diff --git a/tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json b/tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json new file mode 100644 index 0000000000..a8c2749879 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json @@ -0,0 +1,144 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "a", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json b/tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json new file mode 100644 index 0000000000..a8c2749879 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json @@ -0,0 +1,144 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "a", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json b/tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json new file mode 100644 index 0000000000..ac2ffb4ce2 --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json @@ -0,0 +1,180 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "libvirt-2-format", + "name": "c" + }, + { + "node": "libvirt-3-format", + "name": "c" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "libvirt-2-format", + "name": "b" + }, + { + "node": "libvirt-3-format", + "name": "b" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "a", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "libvirt-2-format", + "name": "a" + }, + { + "node": "libvirt-3-format", + "name": "a" + }, + { + "node": "libvirt-4-format", + "name": "a" + }, + { + "node": "libvirt-5-format", + "name": "a" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "libvirt-2-format", + "name": "d" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + } +] diff --git a/tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json b/tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json new file mode 100644 index 0000000000..3a9942569a --- /dev/null +++ b/tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json @@ -0,0 +1,144 @@ +[ + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "current", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "current", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "current" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "c", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "c", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "c" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "b", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "b", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "b" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "a", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "a", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "a" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-add", + "data": { + "node": "mirror-format-node", + "name": "d", + "persistent": true, + "disabled": false, + "granularity": 65536 + } + }, + { + "type": "block-dirty-bitmap-merge", + "data": { + "node": "mirror-format-node", + "target": "d", + "bitmaps": [ + { + "node": "libvirt-1-format", + "name": "d" + }, + { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + ] + } + }, + { + "type": "block-dirty-bitmap-remove", + "data": { + "node": "mirror-format-node", + "name": "libvirt-tmp-activewrite" + } + } +] -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Test both 'basic' and 'snapshots' cases on shallow and deep copy modes.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- tests/qemublocktest.c | 5 + .../bitmapblockcopy/basic-deep-out.json | 144 ++++++++++++++ .../bitmapblockcopy/basic-shallow-out.json | 144 ++++++++++++++ .../bitmapblockcopy/snapshots-deep-out.json | 180 ++++++++++++++++++ .../snapshots-shallow-out.json | 144 ++++++++++++++ 5 files changed, 617 insertions(+) create mode 100644 tests/qemublocktestdata/bitmapblockcopy/basic-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/basic-shallow-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/snapshots-shallow-out.json
Reviewed-by: Eric Blake <eblake@redhat.com> -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
Outline the basics and how to integrate with externally created overlays. Other topics will continue later. Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 ++++++++++++++++++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/kbase/incrementalbackupinternals.rst diff --git a/docs/kbase.html.in b/docs/kbase.html.in index c586e0f676..4257e52b7e 100644 --- a/docs/kbase.html.in +++ b/docs/kbase.html.in @@ -36,6 +36,9 @@ <dt><a href="kbase/virtiofs.html">Virtio-FS</a></dt> <dd>Share a filesystem between the guest and the host</dd> + + <dt><a href="kbase/incrementalbackupinternals.html">Incremental backup internals</a></dt> + <dd>Incremental backup implementation details relevant for users</dd> </dl> </div> diff --git a/docs/kbase/incrementalbackupinternals.rst b/docs/kbase/incrementalbackupinternals.rst new file mode 100644 index 0000000000..adf12002d2 --- /dev/null +++ b/docs/kbase/incrementalbackupinternals.rst @@ -0,0 +1,210 @@ +================================================ +Internals of incremental backup handling in qemu +================================================ + +.. contents:: + +Libvirt's implementation of incremental backups in the ``qemu`` driver uses +qemu's ``block-dirty-bitmaps`` under the hood to track the guest visible disk +state changes correspoiding to the points in time described by a libvirt +checkpoint. + +There are some semantical implications how libvirt creates and manages the +bitmaps which de-facto become API as they are written into the disk images and +this document will try to sumarize them. + +Glossary +======== + +Checkpoint + + A libvirt object which represents a named point in time of the life of the + vm where libvirt tracks writes the VM has done and allows then a backup of + block which changed. Note that state of the VM memory is _not_ captured. + + A checkpoint can be created either explicitly via the corresponding API + which isn't very useful or is created as part of creating an + incremental or full backup of the VM using the ``virDomainBackupBegin`` API + which allows a next backup to only copy the differences. + +Backup + + A copy of either all blocks of selected disks (full backup) or blocks changed + since a checkpoint (incremental backup) at the time the backup job was + started. (Blocks modified while the backup job is running are not part of the + backup!) + +Snapshot + + Similarly to a checkpoint it's a point in time in the lifecycle of the VM + but the state of the VM including memory is captured at that point allowing + returning to the state later. + +Blockjob + + A long running job which modifies the shape and/or location of the disk + backing chain (images storing the disk contents). Libvirt supports + ``block pull`` where data is moved up the chain towards the active layer, + ``block commit`` where data is moved down the chain towards the base/oldest + image. These blockjobs always remove images from the backing chain. Lastly + ``block copy`` where image is moved to a different location (and possibly + collapsed moving all of the data into the new location into the one image). + +block-dirty-bitmap (bitmap) + + A data structure in qemu tracking which blocks were written by the guest + OS since the bitmap was created. + +Relationships of bitmaps, checkpoints and VM disks +================================================== + +When a checkpoint is created libvirt creates a block-dirty-bitmap for every +configured VM disk named the same way as chcheckpoint. The bitmap is actively +recording which blocks were changed by the guest OS from that point on. Other +bitmaps are not impacted by any way as they are self-contained: + +:: + + +----------------+ +----------------+ + | disk: vda | | disk: vdb | + +--------+-------+ +--------+-------+ + | | + +--------v-------+ +--------v-------+ + | vda-1.qcow2 | | vdb-1.qcow2 | + | | | | + | bitmaps: chk-a | | bitmaps: chk-a | + | chk-b | | chk-b | + | | | | + +----------------+ +----------------+ + +Bitmaps are created at the same time to track changes to all disks in sync and +are active and persisted in the QCOW2 image. Oter formats currently don't +support this feature. + +Modification of bitmaps outside of libvirt is not recommended, but when adrering +to the same semantics which the document will describe it should be safe to do +so but obviously we can't guarantee that. + + +Integration with external snapshots +=================================== + +Handling of bitmaps +------------------- + +Creating an external snapshot involves adding a new layer to the backing chain +on top of the previous chain. In this step there are no new bitmaps created by +default, which would mean that backups become impossible after this step. + +To prevent this from happening we need to re-create the active bitmaps in the +new top/active layer of the backing chain which allows us to continue tracking +the changes with same granularity as before and also allows libvirt to stitch +together all the corresponding bitmaps to do a backup acorss snapshots. + +After taking a snapshot of the ``vda`` disk from the example above placed into +``vda-2.qcow2`` the following topology will be created: + +:: + + +----------------+ + | disk: vda | + +-------+--------+ + | + +-------v--------+ +----------------+ + | vda-2.qcow2 | | vda-1.qcow2 | + | | | | + | bitmaps: chk-a +----> bitmaps: chk-a | + | chk-b | | chk-b | + | | | | + +----------------+ +----------------+ + +Checking bitmap health +---------------------- + +QEMU optimizes disk writes by only updating the bitmaps in certain cases. This +also can cause problems in cases when e.g. QEMU crashes. + +For a chain of bitmaps corresponding in a backing chain to be considered valid +and eligible for use with ``virDomainBackupBegin`` it must conform to the +following rules: + +1) Top image must contain the bitmap +2) If any of the backing images in the chain contain the bitmap too all + contiguous images must have the bitmap (no gaps) +3) all of the above bitmaps must be marked as active + (``auto`` flag in ``qemu-img`` output, ``recording`` in qemu) +4) none of the above bitmaps can be inconsistent + (``in-use`` flag in ``qemu-img`` provided that it's not used on image which + is currently in use by a qemu instance, or ``inconsistent`` in qemu) + +:: + + # check that image has bitmaps + $ qemu-img info vda-1.qcow2 + image: vda-1.qcow2 + file format: qcow2 + virtual size: 100 MiB (104857600 bytes) + disk size: 220 KiB + cluster_size: 65536 + Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + bitmaps: + [0]: + flags: + [0]: in-use + [1]: auto + name: chk-a + granularity: 65536 + [1]: + flags: + [0]: auto + name: chk-b + granularity: 65536 + refcount bits: 16 + corrupt: false + +(See also the ``qemuBlockBitmapChainIsValid`` helper method in +``src/qemu/qemu_block.c``) + +Creating external checkpoints manually +-------------------------------------- + +To create the same topology outside of libvirt (e.g when doing snapshots offline) +a new ``qemu-img`` which supports the ``bitmap`` subcomand is necessary. The +following algorithm then ensures that the new image after snapshot will work +with backups (note that ``jq`` is a JSON processor): + +:: + + # arguments + SNAP_IMG="vda-2.qcow2" + BACKING_IMG="vda-1.qcow2" + + # constants - snapshots and bitmaps work only with qcow2 + SNAP_FMT="qcow2" + BACKING_IMG_FMT="qcow2" + + # create snapshot overlay + qemu-img create -f "$SNAP_FMT" -F "$BACKING_IMG_FMT" -b "$BACKING_IMG" "$SNAP_IMG" + + BACKING_IMG_INFO=$(qemu-img info --output=json -f "$BACKING_IMG_FMT" "$BACKING_IMG") + BACKING_BITMAPS=$(jq '."format-specific".data.bitmaps' <<< "$BACKING_IMG_INFO") + + if [ "x$BACKING_BITMAPS" == "xnull" ]; then + exit 0 + fi + + for BACKING_BITMAP_ in $(jq -c '.[]' <<< "$BACKING_BITMAPS"); do + BITMAP_FLAGS=$(jq -c -r '.flags[]' <<< "$BACKING_BITMAP_") + BITMAP_NAME=$(jq -r '.name' <<< "$BACKING_BITMAP_") + + if grep 'in-use' <<< "$BITMAP_FLAGS" || + grep -v 'auto' <<< "$BITMAP_FLAGS"; then + continue + fi + + qemu-img bitmap -f "$SNAP_FMT" "$SNAP_IMG" --add "$BITMAP_NAME" + + done -- 2.26.2
On 6/15/20 12:10 PM, Peter Krempa wrote:
Outline the basics and how to integrate with externally created overlays. Other topics will continue later.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 ++++++++++++++++++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/kbase/incrementalbackupinternals.rst
+++ b/docs/kbase/incrementalbackupinternals.rst @@ -0,0 +1,210 @@ +================================================ +Internals of incremental backup handling in qemu +================================================ + +.. contents:: + +Libvirt's implementation of incremental backups in the ``qemu`` driver uses +qemu's ``block-dirty-bitmaps`` under the hood to track the guest visible disk +state changes correspoiding to the points in time described by a libvirt
corresponding
+checkpoint. + +There are some semantical implications how libvirt creates and manages the
semantic implications with how
+bitmaps which de-facto become API as they are written into the disk images and
images,
+this document will try to sumarize them.
summarize
+ +Glossary +======== + +Checkpoint + + A libvirt object which represents a named point in time of the life of the + vm where libvirt tracks writes the VM has done and allows then a backup of
has done, thereby allowing a backup of only the blocks which changed
+ block which changed. Note that state of the VM memory is _not_ captured. + + A checkpoint can be created either explicitly via the corresponding API + which isn't very useful or is created as part of creating an + incremental or full backup of the VM using the ``virDomainBackupBegin`` API + which allows a next backup to only copy the differences.
Maybe: A checkpoint can be created either explicitly via the corresponding API (although this isn't very useful on its own), or simultaneously with an incremental or full backup of the VM
+ +Backup + + A copy of either all blocks of selected disks (full backup) or blocks changed + since a checkpoint (incremental backup) at the time the backup job was + started. (Blocks modified while the backup job is running are not part of the + backup!) + +Snapshot + + Similarly to a checkpoint it's a point in time in the lifecycle of the VM + but the state of the VM including memory is captured at that point allowing + returning to the state later.
Hmm. We have disk-only snapshots which do not save the state of memory. Does this paragraph need adjustment to mention the difference between a disk-only snapshot and a full state capture? Are we redefining any of the terms in domainstatecapture.rst, and/or should those two documents have cross-references?
+ +Blockjob + + A long running job which modifies the shape and/or location of the disk + backing chain (images storing the disk contents). Libvirt supports
If qemu adds block-dirty-bitmap-populate, blockjobs can also manipulate just bitmaps.
+ ``block pull`` where data is moved up the chain towards the active layer, + ``block commit`` where data is moved down the chain towards the base/oldest + image. These blockjobs always remove images from the backing chain. Lastly + ``block copy`` where image is moved to a different location (and possibly + collapsed moving all of the data into the new location into the one image). + +block-dirty-bitmap (bitmap) + + A data structure in qemu tracking which blocks were written by the guest + OS since the bitmap was created. + +Relationships of bitmaps, checkpoints and VM disks +================================================== + +When a checkpoint is created libvirt creates a block-dirty-bitmap for every +configured VM disk named the same way as chcheckpoint. The bitmap is actively
s/chcheckpoint/the checkpoint/
+recording which blocks were changed by the guest OS from that point on. Other +bitmaps are not impacted by any way as they are self-contained: + +:: + + +----------------+ +----------------+ + | disk: vda | | disk: vdb | + +--------+-------+ +--------+-------+ + | | + +--------v-------+ +--------v-------+ + | vda-1.qcow2 | | vdb-1.qcow2 | + | | | | + | bitmaps: chk-a | | bitmaps: chk-a | + | chk-b | | chk-b | + | | | | + +----------------+ +----------------+ + +Bitmaps are created at the same time to track changes to all disks in sync and +are active and persisted in the QCOW2 image. Oter formats currently don't
Other
+support this feature. + +Modification of bitmaps outside of libvirt is not recommended, but when adrering
adhering
+to the same semantics which the document will describe it should be safe to do +so but obviously we can't guarantee that.
do so, even if we obviously can't guarantee that
+ + +Integration with external snapshots +=================================== + +Handling of bitmaps +------------------- + +Creating an external snapshot involves adding a new layer to the backing chain +on top of the previous chain. In this step there are no new bitmaps created by +default, which would mean that backups become impossible after this step. + +To prevent this from happening we need to re-create the active bitmaps in the +new top/active layer of the backing chain which allows us to continue tracking +the changes with same granularity as before and also allows libvirt to stitch +together all the corresponding bitmaps to do a backup acorss snapshots.
across
+ +After taking a snapshot of the ``vda`` disk from the example above placed into +``vda-2.qcow2`` the following topology will be created: + +:: + + +----------------+ + | disk: vda | + +-------+--------+ + | + +-------v--------+ +----------------+ + | vda-2.qcow2 | | vda-1.qcow2 | + | | | | + | bitmaps: chk-a +----> bitmaps: chk-a | + | chk-b | | chk-b | + | | | | + +----------------+ +----------------+ + +Checking bitmap health +---------------------- + +QEMU optimizes disk writes by only updating the bitmaps in certain cases. This +also can cause problems in cases when e.g. QEMU crashes. + +For a chain of bitmaps corresponding in a backing chain to be considered valid
corresponding bitmaps
+and eligible for use with ``virDomainBackupBegin`` it must conform to the +following rules: + +1) Top image must contain the bitmap +2) If any of the backing images in the chain contain the bitmap too all
too,
+ contiguous images must have the bitmap (no gaps) +3) all of the above bitmaps must be marked as active + (``auto`` flag in ``qemu-img`` output, ``recording`` in qemu) +4) none of the above bitmaps can be inconsistent + (``in-use`` flag in ``qemu-img`` provided that it's not used on image which + is currently in use by a qemu instance, or ``inconsistent`` in qemu) + +:: + + # check that image has bitmaps + $ qemu-img info vda-1.qcow2 + image: vda-1.qcow2 + file format: qcow2 + virtual size: 100 MiB (104857600 bytes) + disk size: 220 KiB + cluster_size: 65536 + Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + bitmaps: + [0]: + flags: + [0]: in-use + [1]: auto + name: chk-a + granularity: 65536 + [1]: + flags: + [0]: auto + name: chk-b + granularity: 65536 + refcount bits: 16 + corrupt: false + +(See also the ``qemuBlockBitmapChainIsValid`` helper method in +``src/qemu/qemu_block.c``) + +Creating external checkpoints manually
s/checkpoints/snapshots/
+-------------------------------------- + +To create the same topology outside of libvirt (e.g when doing snapshots offline) +a new ``qemu-img`` which supports the ``bitmap`` subcomand is necessary. The
subcommand s/necessary/recommended/ (as it is also possible to use 'qemu-kvm -S' to do the same actions via QMP commands - although I'm not sure if it is worth documenting that fallback)
+following algorithm then ensures that the new image after snapshot will work +with backups (note that ``jq`` is a JSON processor): + +:: + + # arguments + SNAP_IMG="vda-2.qcow2" + BACKING_IMG="vda-1.qcow2" + + # constants - snapshots and bitmaps work only with qcow2 + SNAP_FMT="qcow2" + BACKING_IMG_FMT="qcow2" + + # create snapshot overlay + qemu-img create -f "$SNAP_FMT" -F "$BACKING_IMG_FMT" -b "$BACKING_IMG" "$SNAP_IMG" + + BACKING_IMG_INFO=$(qemu-img info --output=json -f "$BACKING_IMG_FMT" "$BACKING_IMG") + BACKING_BITMAPS=$(jq '."format-specific".data.bitmaps' <<< "$BACKING_IMG_INFO")
<<< is a bashism.
+ + if [ "x$BACKING_BITMAPS" == "xnull" ]; then
So is == instead of =. Either we should tweak this to be portable to dash, or you should add a #!/bin/bash line to the top of the example.
+ exit 0 + fi + + for BACKING_BITMAP_ in $(jq -c '.[]' <<< "$BACKING_BITMAPS"); do + BITMAP_FLAGS=$(jq -c -r '.flags[]' <<< "$BACKING_BITMAP_") + BITMAP_NAME=$(jq -r '.name' <<< "$BACKING_BITMAP_") + + if grep 'in-use' <<< "$BITMAP_FLAGS" || + grep -v 'auto' <<< "$BITMAP_FLAGS"; then + continue + fi + + qemu-img bitmap -f "$SNAP_FMT" "$SNAP_IMG" --add "$BITMAP_NAME"
Do you want to also copy the --granularity of the bitmaps being added?
+ + done
-- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
On Fri, Jun 19, 2020 at 10:10:36 -0500, Eric Blake wrote:
On 6/15/20 12:10 PM, Peter Krempa wrote:
Outline the basics and how to integrate with externally created overlays. Other topics will continue later.
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 ++++++++++++++++++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/kbase/incrementalbackupinternals.rst
[...]
+Snapshot + + Similarly to a checkpoint it's a point in time in the lifecycle of the VM + but the state of the VM including memory is captured at that point allowing + returning to the state later.
Hmm. We have disk-only snapshots which do not save the state of memory. Does this paragraph need adjustment to mention the difference between a disk-only snapshot and a full state capture? Are we redefining any of the terms in domainstatecapture.rst, and/or should those two documents have cross-references?
Good idea! I'll keep this as is, but I'll link to the domainstatecapture.rst document at the beginning of the glossary. [...]
+ fi + + for BACKING_BITMAP_ in $(jq -c '.[]' <<< "$BACKING_BITMAPS"); do + BITMAP_FLAGS=$(jq -c -r '.flags[]' <<< "$BACKING_BITMAP_") + BITMAP_NAME=$(jq -r '.name' <<< "$BACKING_BITMAP_") + + if grep 'in-use' <<< "$BITMAP_FLAGS" || + grep -v 'auto' <<< "$BITMAP_FLAGS"; then + continue + fi + + qemu-img bitmap -f "$SNAP_FMT" "$SNAP_IMG" --add "$BITMAP_NAME"
Do you want to also copy the --granularity of the bitmaps being added?
Hmm, yeah, that's what libvirt does.
On Mon, Jun 15, 2020 at 8:13 PM Peter Krempa <pkrempa@redhat.com> wrote:
Outline the basics and how to integrate with externally created overlays. Other topics will continue later.
Thanks, this is very helpful!
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 ++++++++++++++++++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/kbase/incrementalbackupinternals.rst
diff --git a/docs/kbase.html.in b/docs/kbase.html.in index c586e0f676..4257e52b7e 100644 --- a/docs/kbase.html.in +++ b/docs/kbase.html.in @@ -36,6 +36,9 @@
<dt><a href="kbase/virtiofs.html">Virtio-FS</a></dt> <dd>Share a filesystem between the guest and the host</dd> + + <dt><a href="kbase/incrementalbackupinternals.html">Incremental backup internals</a></dt> + <dd>Incremental backup implementation details relevant for users</dd> </dl> </div>
diff --git a/docs/kbase/incrementalbackupinternals.rst b/docs/kbase/incrementalbackupinternals.rst new file mode 100644 index 0000000000..adf12002d2 --- /dev/null +++ b/docs/kbase/incrementalbackupinternals.rst @@ -0,0 +1,210 @@ +================================================ +Internals of incremental backup handling in qemu +================================================ + +.. contents:: + +Libvirt's implementation of incremental backups in the ``qemu`` driver uses +qemu's ``block-dirty-bitmaps`` under the hood to track the guest visible disk +state changes correspoiding to the points in time described by a libvirt +checkpoint. + +There are some semantical implications how libvirt creates and manages the +bitmaps which de-facto become API as they are written into the disk images and +this document will try to sumarize them. + +Glossary +======== + +Checkpoint + + A libvirt object which represents a named point in time of the life of the + vm where libvirt tracks writes the VM has done and allows then a backup of + block which changed. Note that state of the VM memory is _not_ captured. + + A checkpoint can be created either explicitly via the corresponding API + which isn't very useful or is created as part of creating an + incremental or full backup of the VM using the ``virDomainBackupBegin`` API + which allows a next backup to only copy the differences. + +Backup + + A copy of either all blocks of selected disks (full backup) or blocks changed + since a checkpoint (incremental backup) at the time the backup job was + started. (Blocks modified while the backup job is running are not part of the + backup!) + +Snapshot + + Similarly to a checkpoint it's a point in time in the lifecycle of the VM + but the state of the VM including memory is captured at that point allowing + returning to the state later. + +Blockjob + + A long running job which modifies the shape and/or location of the disk + backing chain (images storing the disk contents). Libvirt supports + ``block pull`` where data is moved up the chain towards the active layer, + ``block commit`` where data is moved down the chain towards the base/oldest + image. These blockjobs always remove images from the backing chain. Lastly + ``block copy`` where image is moved to a different location (and possibly + collapsed moving all of the data into the new location into the one image). + +block-dirty-bitmap (bitmap) + + A data structure in qemu tracking which blocks were written by the guest + OS since the bitmap was created. + +Relationships of bitmaps, checkpoints and VM disks +================================================== + +When a checkpoint is created libvirt creates a block-dirty-bitmap for every +configured VM disk named the same way as chcheckpoint. The bitmap is actively +recording which blocks were changed by the guest OS from that point on. Other +bitmaps are not impacted by any way as they are self-contained: + +:: + + +----------------+ +----------------+ + | disk: vda | | disk: vdb | + +--------+-------+ +--------+-------+ + | | + +--------v-------+ +--------v-------+ + | vda-1.qcow2 | | vdb-1.qcow2 | + | | | | + | bitmaps: chk-a | | bitmaps: chk-a | + | chk-b | | chk-b | + | | | | + +----------------+ +----------------+ + +Bitmaps are created at the same time to track changes to all disks in sync and +are active and persisted in the QCOW2 image. Oter formats currently don't +support this feature. + +Modification of bitmaps outside of libvirt is not recommended, but when adrering +to the same semantics which the document will describe it should be safe to do +so but obviously we can't guarantee that. + + +Integration with external snapshots +=================================== + +Handling of bitmaps +------------------- + +Creating an external snapshot involves adding a new layer to the backing chain +on top of the previous chain. In this step there are no new bitmaps created by +default, which would mean that backups become impossible after this step. + +To prevent this from happening we need to re-create the active bitmaps in the +new top/active layer of the backing chain which allows us to continue tracking +the changes with same granularity as before and also allows libvirt to stitch +together all the corresponding bitmaps to do a backup acorss snapshots. + +After taking a snapshot of the ``vda`` disk from the example above placed into +``vda-2.qcow2`` the following topology will be created: + +:: + + +----------------+ + | disk: vda | + +-------+--------+ + | + +-------v--------+ +----------------+ + | vda-2.qcow2 | | vda-1.qcow2 | + | | | | + | bitmaps: chk-a +----> bitmaps: chk-a | + | chk-b | | chk-b | + | | | | + +----------------+ +----------------+ + +Checking bitmap health +---------------------- + +QEMU optimizes disk writes by only updating the bitmaps in certain cases. This +also can cause problems in cases when e.g. QEMU crashes. + +For a chain of bitmaps corresponding in a backing chain to be considered valid +and eligible for use with ``virDomainBackupBegin`` it must conform to the +following rules: + +1) Top image must contain the bitmap +2) If any of the backing images in the chain contain the bitmap too all + contiguous images must have the bitmap (no gaps) +3) all of the above bitmaps must be marked as active + (``auto`` flag in ``qemu-img`` output, ``recording`` in qemu) +4) none of the above bitmaps can be inconsistent + (``in-use`` flag in ``qemu-img`` provided that it's not used on image which + is currently in use by a qemu instance, or ``inconsistent`` in qemu)
Can you add a chapter of about the old format and how it was different from the new format? Looks like the differences are: - all bitmaps are always active, so no need to enable or disable them - based on next section, new snapshost contain all the bitmaps from the previous snapshots (since all of them active and we copy all of them to the new snapshot) How qemu knows which bitmap should track changes if all bitmaps are active? When libvirt starts a VM, it knows nothing about the checkpoints. We define the checkpoints right before the first backup after starting a VM. So both libvirt and qemu know nothing about the bitmaps at this point. Do you expect to have the checkpoints defined before the guest is started? If I understand this correctly, if we do: - create base image - do full backup (check-1) - do incremental backup 1 (check-2) - create snapshot-1 - do incremental backup 2 (check-3) - do incremental backup 3 (check-4) - create snapshot-2 - do incremental backup 4 (check-5) This will be the image structure: - base image - check-1 - check-2 - snapshot-1 - check-1 - check-2 - check-3 - check-4 - snapshot-2 - check-1 - check-2 - check-3 - check-4 - check-5 So we are duplicating bitmaps that have no content in all snapshot? Why not copy only the last (current) bitmap? - base image - check-1 - check-2 - snapshot-1 - check-2 - check-3 - check-4 - snapshot-2 - check-4 - check-5
+:: + + # check that image has bitmaps + $ qemu-img info vda-1.qcow2 + image: vda-1.qcow2 + file format: qcow2 + virtual size: 100 MiB (104857600 bytes) + disk size: 220 KiB + cluster_size: 65536 + Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + bitmaps: + [0]: + flags: + [0]: in-use + [1]: auto + name: chk-a + granularity: 65536 + [1]: + flags: + [0]: auto + name: chk-b + granularity: 65536 + refcount bits: 16 + corrupt: false + +(See also the ``qemuBlockBitmapChainIsValid`` helper method in +``src/qemu/qemu_block.c``)
Looks like oVirt needs to implement this a well, otherwise we will waste time creating bitmaps on a snapshot when libvirt will fail the backup later since there was an inconsistent or disabled bitmap.
+Creating external checkpoints manually +-------------------------------------- + +To create the same topology outside of libvirt (e.g when doing snapshots offline) +a new ``qemu-img`` which supports the ``bitmap`` subcomand is necessary. The +following algorithm then ensures that the new image after snapshot will work +with backups (note that ``jq`` is a JSON processor): + +:: + + # arguments + SNAP_IMG="vda-2.qcow2" + BACKING_IMG="vda-1.qcow2" + + # constants - snapshots and bitmaps work only with qcow2 + SNAP_FMT="qcow2" + BACKING_IMG_FMT="qcow2" + + # create snapshot overlay + qemu-img create -f "$SNAP_FMT" -F "$BACKING_IMG_FMT" -b "$BACKING_IMG" "$SNAP_IMG" + + BACKING_IMG_INFO=$(qemu-img info --output=json -f "$BACKING_IMG_FMT" "$BACKING_IMG") + BACKING_BITMAPS=$(jq '."format-specific".data.bitmaps' <<< "$BACKING_IMG_INFO") + + if [ "x$BACKING_BITMAPS" == "xnull" ]; then + exit 0 + fi + + for BACKING_BITMAP_ in $(jq -c '.[]' <<< "$BACKING_BITMAPS"); do + BITMAP_FLAGS=$(jq -c -r '.flags[]' <<< "$BACKING_BITMAP_") + BITMAP_NAME=$(jq -r '.name' <<< "$BACKING_BITMAP_") + + if grep 'in-use' <<< "$BITMAP_FLAGS" || + grep -v 'auto' <<< "$BITMAP_FLAGS"; then + continue + fi + + qemu-img bitmap -f "$SNAP_FMT" "$SNAP_IMG" --add "$BITMAP_NAME"
So what we have to do is: - get a list of bitmaps that should be in this disk from oVirt engine - get list of bitmaps with "auto" flag and without the "in-use" flag in the backing file - if the lists do not match we can delete all bitmaps and relevant checkpoints on engine side, since the next backup will fail anyway. - if the lists match, maybe verify that bitmaps are not missing in lower layers (gaps) - if the lists match, create empty bitmap with the same name and granularity in the top image What do we have to do for the old format? or we just not implement this until we get the new format? Can you add a section explaining how bitmaps should be handled in blockCommit? We may need to implement this for cold merge using qemu-img commit. Looking at the current structure, it looks like we have to do: 1. commit top layer to base layer (we support only one layer commit) 2. merge all bitmaps from top layer to base layer 3. copy all bitmaps in top layer that are not in base layer to base layer Nir
+ + done -- 2.26.2
On Mon, Jun 22, 2020 at 02:40:18 +0300, Nir Soffer wrote:
On Mon, Jun 15, 2020 at 8:13 PM Peter Krempa <pkrempa@redhat.com> wrote:
Outline the basics and how to integrate with externally created overlays. Other topics will continue later.
Thanks, this is very helpful!
Signed-off-by: Peter Krempa <pkrempa@redhat.com> --- docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 ++++++++++++++++++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/kbase/incrementalbackupinternals.rst
[...]
+ +Checking bitmap health +---------------------- + +QEMU optimizes disk writes by only updating the bitmaps in certain cases. This +also can cause problems in cases when e.g. QEMU crashes. + +For a chain of bitmaps corresponding in a backing chain to be considered valid +and eligible for use with ``virDomainBackupBegin`` it must conform to the +following rules: + +1) Top image must contain the bitmap +2) If any of the backing images in the chain contain the bitmap too all + contiguous images must have the bitmap (no gaps) +3) all of the above bitmaps must be marked as active + (``auto`` flag in ``qemu-img`` output, ``recording`` in qemu) +4) none of the above bitmaps can be inconsistent + (``in-use`` flag in ``qemu-img`` provided that it's not used on image which + is currently in use by a qemu instance, or ``inconsistent`` in qemu)
Can you add a chapter of about the old format and how it was different from the new format?
No, I don't plan to do that. This feature was never enabled upstream yet, so except for the users who hacked-in the support by adding the correct capability, nobody was exposed to bitmaps managed by libvirt yet. As of such I don't feel it's necessary or even worth documenting the old state.
Looks like the differences are: - all bitmaps are always active, so no need to enable or disable them - based on next section, new snapshost contain all the bitmaps from the previous snapshots (since all of them active and we copy all of them to the new snapshot)
Yes.
How qemu knows which bitmap should track changes if all bitmaps are active?
When libvirt starts a VM, it knows nothing about the checkpoints. We define the checkpoints right before the first backup after starting a VM. So both libvirt and qemu know nothing about the bitmaps at this point.
QEMU tracks changes for all active bitmaps, that's why they are active. Libvirt doesn't care at startup, but cares if you want to do a backup.
Do you expect to have the checkpoints defined before the guest is started?
If I understand this correctly, if we do:
- create base image - do full backup (check-1) - do incremental backup 1 (check-2) - create snapshot-1 - do incremental backup 2 (check-3) - do incremental backup 3 (check-4) - create snapshot-2 - do incremental backup 4 (check-5)
This will be the image structure:
- base image - check-1 - check-2 - snapshot-1 - check-1 - check-2 - check-3 - check-4 - snapshot-2 - check-1 - check-2 - check-3 - check-4 - check-5
So we are duplicating bitmaps that have no content in all snapshot?
Yes. It's way easier to manage them that way.
Why not copy only the last (current) bitmap?
- base image - check-1 - check-2 - snapshot-1 - check-2 - check-3 - check-4 - snapshot-2 - check-4 - check-5
This is too complicated to deal with. That's what we did for now and it's a nightmare to compute what to do. Ideally when 'block-dirty-bitmap-populate' materializes, I'll add a flag for API creating snapshots which will stop pulling bitmaps up, so you'll end up with: - base image - check-1 - check-2 - snapshot-1 - check-3 - check-4 - snapshot-2 - check-5 Since the rest can be re-calculated It actually doesn't make sense to use any bitmap after a snapshot.
+:: + + # check that image has bitmaps + $ qemu-img info vda-1.qcow2 + image: vda-1.qcow2 + file format: qcow2 + virtual size: 100 MiB (104857600 bytes) + disk size: 220 KiB + cluster_size: 65536 + Format specific information: + compat: 1.1 + compression type: zlib + lazy refcounts: false + bitmaps: + [0]: + flags: + [0]: in-use + [1]: auto + name: chk-a + granularity: 65536 + [1]: + flags: + [0]: auto + name: chk-b + granularity: 65536 + refcount bits: 16 + corrupt: false + +(See also the ``qemuBlockBitmapChainIsValid`` helper method in +``src/qemu/qemu_block.c``)
Looks like oVirt needs to implement this a well, otherwise we will waste time creating bitmaps on a snapshot when libvirt will fail the backup later since there was an inconsistent or disabled bitmap.
+Creating external checkpoints manually +-------------------------------------- + +To create the same topology outside of libvirt (e.g when doing snapshots offline) +a new ``qemu-img`` which supports the ``bitmap`` subcomand is necessary. The +following algorithm then ensures that the new image after snapshot will work +with backups (note that ``jq`` is a JSON processor): + +:: + + # arguments + SNAP_IMG="vda-2.qcow2" + BACKING_IMG="vda-1.qcow2" + + # constants - snapshots and bitmaps work only with qcow2 + SNAP_FMT="qcow2" + BACKING_IMG_FMT="qcow2" + + # create snapshot overlay + qemu-img create -f "$SNAP_FMT" -F "$BACKING_IMG_FMT" -b "$BACKING_IMG" "$SNAP_IMG" + + BACKING_IMG_INFO=$(qemu-img info --output=json -f "$BACKING_IMG_FMT" "$BACKING_IMG") + BACKING_BITMAPS=$(jq '."format-specific".data.bitmaps' <<< "$BACKING_IMG_INFO") + + if [ "x$BACKING_BITMAPS" == "xnull" ]; then + exit 0 + fi + + for BACKING_BITMAP_ in $(jq -c '.[]' <<< "$BACKING_BITMAPS"); do + BITMAP_FLAGS=$(jq -c -r '.flags[]' <<< "$BACKING_BITMAP_") + BITMAP_NAME=$(jq -r '.name' <<< "$BACKING_BITMAP_") + + if grep 'in-use' <<< "$BITMAP_FLAGS" || + grep -v 'auto' <<< "$BITMAP_FLAGS"; then + continue + fi + + qemu-img bitmap -f "$SNAP_FMT" "$SNAP_IMG" --add "$BITMAP_NAME"
So what we have to do is:
- get a list of bitmaps that should be in this disk from oVirt engine - get list of bitmaps with "auto" flag and without the "in-use" flag in the backing file - if the lists do not match we can delete all bitmaps and relevant checkpoints on engine side, since the next backup will fail anyway. - if the lists match, maybe verify that bitmaps are not missing in lower layers (gaps) - if the lists match, create empty bitmap with the same name and granularity in the top image
What do we have to do for the old format? or we just not implement this until we get the new format?
The above algorithm will work even for the old format, by the way. There's just one active bitmap at any point. The validation algorithm will not work obviously. The problem of the "old" format is not in snapshots, but rather in any other operation.
Can you add a section explaining how bitmaps should be handled in blockCommit? We may need to implement this for cold merge using qemu-img commit.
Yes, right after this get's merged. By the way, block commit is one of the things where the 'old' format was becoming insane.
Looking at the current structure, it looks like we have to do:
1. commit top layer to base layer (we support only one layer commit) 2. merge all bitmaps from top layer to base layer 3. copy all bitmaps in top layer that are not in base layer to base layer
Yes, for that special case it's true. Technically I'd write it as: 2) create bitmaps in base layer which are in top but not in base (they are empty now 3) merge all bitmaps from top into corresponding bitmap in base
participants (3)
-
Eric Blake -
Nir Soffer -
Peter Krempa