[RFC PATCH 00/41] qemu: Rewrite checkpoint code for 'block-dirty-bitmap-populate'

Use 'block-dirty-bitmap-populate' and change how we create bitmaps corresponding to checkpoints to simplify the code and also properly integrate with backing chain images created outside of libvirt. This patchset changes how we approach checkpoints by keeping one bitmap per checkpoint and disk and not propagating the bitmaps into overlays on snapshots. This massively simplifies the code handling all the operations during blockjobs and backups. While the change isn't compatible with checkpoints created previously, we didn't yet enable the support for checkpoints/backups. Note that 'block-dirty-bitmap-populate' is _not_ in finished state in qemu yet, so I'm posting this as RFC and as reference for qemu developers of the usefulnes of it. The changes can be fetched by: git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-bitmap-populate Note that the above branch also contains a commit enabling incremental backup for simpler testing. I've pushed the appropriate qemu patches for convenience here: git fetch https://gitlab.com/pipo.sk/qemu.git block-dirty-bitmap-populate Peter Krempa (41): virErrorPreserveLast: Return the saved error object in addition to storing it util: error: Introduce VIR_ERROR_AUTOPRESERVE_LAST macro qemuBackupDiskDataCleanup: Use VIR_ERROR_AUTO_PRESERVE_LAST qemu: backup: Split up code traversing checkpoint list looking for bitmaps qemu: backup: Fix backup of disk skipped in an intermediate checkpoint conf: backup: Store incremental backup checkpoint name per-disk qemu: backup: Move fetching of checkpoint list for incremental backup qemublocktest: Add 'empty' test case for bitmaps qemublocktest: Add 'empty' case for incremental backup test qemublocktest: Add 'empty' case for checkpoint deletion qemublocktest: Add 'empty' case for blockcopy bitmap handling test qemublocktest: Add 'empty' case for checkpoint bitmap handling qemublocktest: Disable testcases for all bitmap handling qemublocktest: Delete 'synthetic' bitmap test cases qemublocktest: Extract printing of nodename list qemu: checkpoint: Don't chain bitmaps for checkpoints qemublocktest: Replace 'basic' bitmap detection test case data qemu: snapshot: Don't propagate bitmaps to upper layers qemublocktest: Replace 'snapshots' bitmap detection test case data qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots' cases qemublocktest: Add new 'synthetic' bitmap detection and validation test case qemu: checkpoint: Don't merge checkpoints during deletion qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to TEST_CHECKPOINT_DELETE qemublocktest: Re-introduce testing of checkpoint deletion qemucapabilitiestest: Bump qemu-5.1 caps on x86_64 for 'dirty-bitmap-populate' qemu: monitor: Add support for 'block-dirty-bitmap-populate' transaction member qemuDomainStorageSourcePrivate: Add per-source private blockjob qemu: blockjob: Introduce 'populate' blockjob qemu: domain: Introduce helper for always fetching virStorageSource private data qemu: block: Add helper to add temporary block bitmaps from allocation maps qemu: block: Add universal helper for merging dirty bitmaps for all scenarios qemu: backup: Rewrite backup bitmap handling to the new bitmap semantics qemublocktest: Add 'basic' tests for backup bitmap handling qemublocktest: Add 'snapshots' tests for backup bitmap handling qemu: Rewrite bitmap handling for block commit qemublocktest: Add 'basic' tests for commit bitmap handling qemublocktest: Add 'snapshots' tests for block commit bitmap handling qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job private data qemu: Rewrite bitmap handling for block copy qemublocktest: Add test cases for handling bitmaps during block-copy src/conf/backup_conf.c | 8 + src/conf/backup_conf.h | 1 + src/qemu/qemu_backup.c | 221 +- src/qemu/qemu_backup.h | 13 +- src/qemu/qemu_block.c | 674 ++-- src/qemu/qemu_block.h | 35 +- src/qemu/qemu_blockjob.c | 182 +- src/qemu/qemu_blockjob.h | 16 +- src/qemu/qemu_checkpoint.c | 174 +- src/qemu/qemu_checkpoint.h | 1 - src/qemu/qemu_domain.c | 46 +- src/qemu/qemu_domain.h | 6 + src/qemu/qemu_driver.c | 112 +- src/qemu/qemu_monitor.c | 11 + src/qemu/qemu_monitor.h | 7 + src/qemu/qemu_monitor_json.c | 18 + src/qemu/qemu_monitor_json.h | 6 + src/util/virerror.c | 18 +- src/util/virerror.h | 12 +- tests/qemublocktest.c | 267 +- .../backupmerge/basic-deep-out.json | 35 +- .../backupmerge/basic-flat-out.json | 23 +- .../backupmerge/basic-intermediate-out.json | 23 +- .../backupmerge/empty-out.json | 3 + .../backupmerge/snapshot-deep-out.json | 38 - .../backupmerge/snapshot-flat-out.json | 6 - .../snapshot-intermediate-out.json | 14 - .../backupmerge/snapshots-deep-out.json | 46 + .../backupmerge/snapshots-flat-out.json | 25 + .../snapshots-intermediate-out.json | 31 + tests/qemublocktestdata/bitmap/basic.json | 229 +- tests/qemublocktestdata/bitmap/basic.out | 8 +- tests/qemublocktestdata/bitmap/empty.json | 70 + tests/qemublocktestdata/bitmap/empty.out | 1 + .../bitmap/snapshots-synthetic-broken.json | 837 ----- .../bitmap/snapshots-synthetic-broken.out | 14 - .../snapshots-synthetic-checkpoint.json | 827 ----- .../bitmap/snapshots-synthetic-checkpoint.out | 13 - tests/qemublocktestdata/bitmap/snapshots.json | 1254 +++----- tests/qemublocktestdata/bitmap/snapshots.out | 6 +- tests/qemublocktestdata/bitmap/synthetic.json | 606 +++- tests/qemublocktestdata/bitmap/synthetic.out | 19 +- .../bitmapblockcommit/basic-1-2 | 66 +- .../bitmapblockcommit/basic-1-3 | 66 +- .../bitmapblockcommit/basic-2-3 | 1 - .../qemublocktestdata/bitmapblockcommit/empty | 1 + .../bitmapblockcommit/snapshots-1-2 | 24 +- .../bitmapblockcommit/snapshots-1-3 | 42 +- .../bitmapblockcommit/snapshots-1-4 | 97 +- .../bitmapblockcommit/snapshots-1-5 | 99 +- .../bitmapblockcommit/snapshots-2-3 | 23 - .../bitmapblockcommit/snapshots-2-4 | 55 +- .../bitmapblockcommit/snapshots-2-5 | 59 +- .../bitmapblockcommit/snapshots-3-4 | 25 +- .../bitmapblockcommit/snapshots-3-5 | 29 +- .../bitmapblockcommit/snapshots-4-5 | 32 - .../snapshots-synthetic-broken-1-2 | 57 - .../snapshots-synthetic-broken-1-3 | 112 - .../snapshots-synthetic-broken-1-4 | 119 - .../snapshots-synthetic-broken-1-5 | 119 - .../snapshots-synthetic-broken-2-3 | 89 - .../snapshots-synthetic-broken-2-4 | 96 - .../snapshots-synthetic-broken-2-5 | 96 - .../snapshots-synthetic-broken-3-4 | 27 - .../snapshots-synthetic-broken-3-5 | 27 - .../snapshots-synthetic-broken-4-5 | 20 - .../bitmapblockcopy/basic-deep-out.json | 65 +- .../bitmapblockcopy/basic-shallow-out.json | 65 +- .../bitmapblockcopy/empty-deep-out.json | 0 .../bitmapblockcopy/empty-shallow-out.json | 0 .../bitmapblockcopy/snapshots-deep-out.json | 104 +- .../snapshots-shallow-out.json | 24 +- .../checkpointdelete/basic-current-out.json | 20 - .../basic-intermediate1-out.json | 13 - .../basic-intermediate2-out.json | 13 - .../basic-intermediate3-out.json | 13 - .../checkpointdelete/empty-out.json | 1 + .../snapshots-current-out.json | 20 - .../snapshots-intermediate1-out.json | 13 - .../snapshots-intermediate2-out.json | 51 - .../snapshots-intermediate3-out.json | 50 - .../snapshots-noparent-out.json | 16 - ...hots-synthetic-checkpoint-current-out.json | 29 - ...ynthetic-checkpoint-intermediate1-out.json | 31 - ...ynthetic-checkpoint-intermediate2-out.json | 34 - ...ynthetic-checkpoint-intermediate3-out.json | 61 - ...ots-synthetic-checkpoint-noparent-out.json | 27 - .../synthetic-current-out.json | 9 + .../synthetic-intermediate1-out.json | 11 + .../synthetic-intermediate2-out.json | 11 + .../synthetic-intermediate3-out.json | 19 + .../synthetic-noparent-out.json | 11 + .../caps_5.1.0.x86_64.replies | 2737 +++++++++-------- .../caps_5.1.0.x86_64.xml | 2 +- tests/qemumonitorjsontest.c | 3 +- .../blockjob-blockdev-in.xml | 4 - 96 files changed, 4077 insertions(+), 6717 deletions(-) create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json create mode 100644 tests/qemublocktestdata/bitmap/empty.json create mode 100644 tests/qemublocktestdata/bitmap/empty.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json -- 2.26.2