[PATCH 00/32] qemu: backup: Rewrite/simplify internals and document semantics

Simplify how we approach bitmap handling to increase robustness and understanding of the code. Additionally document intricacies of how we approach handling of bitmaps so that in cases when users decide to do snapshots outside of libvirt they can re-create the bitmaps so that they will still work with backups (requires new qemu-img). I'll also follow up documenting implicationsof blockjobs on bitmaps later. To fetch this including a commit which allows testing: git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-rewrite-and-document This doesn't require any special qemu patches. Note that this shares many patches from my previous posting which tried to make use of 'block-dirty-bitmap-populate', but keeps explicit bitmap tracking for now. We'll be able to revisit use of the blockjob if the qemu work will continue somehow. Peter Krempa (32): qemu: backup: Split up code traversing checkpoint list looking for bitmaps qemu: backup: Fix backup of disk skipped in an intermediate checkpoint conf: backup: Store incremental backup checkpoint name per-disk qemu: backup: Move fetching of checkpoint list for incremental backup qemublocktest: Add 'empty' test case for bitmaps qemublocktest: Add 'empty' case for incremental backup test qemublocktest: Add 'empty' case for checkpoint deletion qemublocktest: Add 'empty' case for blockcopy bitmap handling test qemublocktest: Add 'empty' case for checkpoint bitmap handling qemublocktest: Disable testcases for all bitmap handling qemublocktest: Delete 'synthetic' bitmap test cases qemublocktest: Extract printing of nodename list qemu: checkpoint: Don't chain bitmaps for checkpoints qemublocktest: Replace 'basic' bitmap detection test case data qemublocktest: Replace 'snapshots' bitmap detection test case data qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots' cases qemublocktest: Add new 'synthetic' bitmap detection and validation test case qemu: checkpoint: Don't merge checkpoints during deletion qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to TEST_CHECKPOINT_DELETE qemublocktest: Re-introduce testing of checkpoint deletion qemu: block: Add universal helper for merging dirty bitmaps for all scenarios qemu: backup: Rewrite backup bitmap handling to the new bitmap semantics qemublocktest: Add 'basic' tests for backup bitmap handling qemublocktest: Add 'snapshots' tests for backup bitmap handling qemu: Rewrite bitmap handling for block commit qemublocktest: Add 'basic' tests for commit bitmap handling qemublocktest: Add 'snapshots' tests for block commit bitmap handling qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job private data qemu: Rewrite bitmap handling for block copy qemublocktest: Add test cases for handling bitmaps during block-copy kbase: Add document outlining internals of incremental backup in qemu docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 +++ src/conf/backup_conf.c | 8 + src/conf/backup_conf.h | 1 + src/qemu/qemu_backup.c | 199 +-- src/qemu/qemu_backup.h | 12 +- src/qemu/qemu_block.c | 500 +++--- src/qemu/qemu_block.h | 21 +- src/qemu/qemu_blockjob.c | 109 +- src/qemu/qemu_blockjob.h | 3 - src/qemu/qemu_checkpoint.c | 174 +-- src/qemu/qemu_checkpoint.h | 1 - src/qemu/qemu_domain.c | 22 - src/qemu/qemu_driver.c | 70 +- tests/qemublocktest.c | 254 ++- .../backupmerge/basic-deep-out.json | 35 +- .../backupmerge/basic-flat-out.json | 23 +- .../backupmerge/basic-intermediate-out.json | 23 +- .../backupmerge/empty-out.json | 3 + .../backupmerge/snapshot-deep-out.json | 38 - .../backupmerge/snapshot-flat-out.json | 6 - .../snapshot-intermediate-out.json | 14 - .../backupmerge/snapshots-deep-out.json | 41 + .../backupmerge/snapshots-flat-out.json | 25 + .../snapshots-intermediate-out.json | 29 + tests/qemublocktestdata/bitmap/basic.json | 229 +-- tests/qemublocktestdata/bitmap/basic.out | 8 +- tests/qemublocktestdata/bitmap/empty.json | 70 + tests/qemublocktestdata/bitmap/empty.out | 1 + .../bitmap/snapshots-synthetic-broken.json | 837 ---------- .../bitmap/snapshots-synthetic-broken.out | 14 - .../snapshots-synthetic-checkpoint.json | 827 ---------- .../bitmap/snapshots-synthetic-checkpoint.out | 13 - tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++---------- tests/qemublocktestdata/bitmap/snapshots.out | 13 +- tests/qemublocktestdata/bitmap/synthetic.json | 606 ++++++-- tests/qemublocktestdata/bitmap/synthetic.out | 19 +- .../bitmapblockcommit/basic-1-2 | 54 +- .../bitmapblockcommit/basic-1-3 | 54 +- .../bitmapblockcommit/basic-2-3 | 1 - .../qemublocktestdata/bitmapblockcommit/empty | 1 + .../bitmapblockcommit/snapshots-1-2 | 76 +- .../bitmapblockcommit/snapshots-1-3 | 77 +- .../bitmapblockcommit/snapshots-1-4 | 99 +- .../bitmapblockcommit/snapshots-1-5 | 103 +- .../bitmapblockcommit/snapshots-2-3 | 48 +- .../bitmapblockcommit/snapshots-2-4 | 58 +- .../bitmapblockcommit/snapshots-2-5 | 62 +- .../bitmapblockcommit/snapshots-3-4 | 34 +- .../bitmapblockcommit/snapshots-3-5 | 42 +- .../bitmapblockcommit/snapshots-4-5 | 17 - .../snapshots-synthetic-broken-1-2 | 57 - .../snapshots-synthetic-broken-1-3 | 112 -- .../snapshots-synthetic-broken-1-4 | 119 -- .../snapshots-synthetic-broken-1-5 | 119 -- .../snapshots-synthetic-broken-2-3 | 89 -- .../snapshots-synthetic-broken-2-4 | 96 -- .../snapshots-synthetic-broken-2-5 | 96 -- .../snapshots-synthetic-broken-3-4 | 27 - .../snapshots-synthetic-broken-3-5 | 27 - .../snapshots-synthetic-broken-4-5 | 20 - .../bitmapblockcopy/basic-deep-out.json | 53 +- .../bitmapblockcopy/basic-shallow-out.json | 53 +- .../bitmapblockcopy/empty-deep-out.json | 0 .../bitmapblockcopy/empty-shallow-out.json | 0 .../bitmapblockcopy/snapshots-deep-out.json | 95 +- .../snapshots-shallow-out.json | 98 +- .../checkpointdelete/basic-current-out.json | 20 - .../basic-intermediate1-out.json | 13 - .../basic-intermediate2-out.json | 13 - .../basic-intermediate3-out.json | 13 - .../checkpointdelete/empty-out.json | 1 + .../snapshots-current-out.json | 20 - .../snapshots-intermediate1-out.json | 20 +- .../snapshots-intermediate2-out.json | 42 +- .../snapshots-intermediate3-out.json | 43 - .../snapshots-noparent-out.json | 15 + ...hots-synthetic-checkpoint-current-out.json | 29 - ...ynthetic-checkpoint-intermediate1-out.json | 31 - ...ynthetic-checkpoint-intermediate2-out.json | 34 - ...ynthetic-checkpoint-intermediate3-out.json | 61 - ...ots-synthetic-checkpoint-noparent-out.json | 27 - .../synthetic-current-out.json | 9 + .../synthetic-intermediate1-out.json | 11 + .../synthetic-intermediate2-out.json | 11 + .../synthetic-intermediate3-out.json | 19 + .../synthetic-noparent-out.json | 11 + .../blockjob-blockdev-in.xml | 4 - 88 files changed, 2927 insertions(+), 5150 deletions(-) create mode 100644 docs/kbase/incrementalbackupinternals.rst create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json create mode 100644 tests/qemublocktestdata/bitmap/empty.json create mode 100644 tests/qemublocktestdata/bitmap/empty.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json -- 2.26.2