June 2020 - Devel - libvirt List Archives

[PATCH 00/32] qemu: backup: Rewrite/simplify internals and document semantics
by Peter Krempa 22 Jun '20

22 Jun '20

Simplify how we approach bitmap handling to increase robustness and understanding of the code. Additionally document intricacies of how we approach handling of bitmaps so that in cases when users decide to do snapshots outside of libvirt they can re-create the bitmaps so that they will still work with backups (requires new qemu-img). I'll also follow up documenting implicationsof blockjobs on bitmaps later. To fetch this including a commit which allows testing: git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-rewrite-and-document This doesn't require any special qemu patches. Note that this shares many patches from my previous posting which tried to make use of 'block-dirty-bitmap-populate', but keeps explicit bitmap tracking for now. We'll be able to revisit use of the blockjob if the qemu work will continue somehow. Peter Krempa (32): qemu: backup: Split up code traversing checkpoint list looking for bitmaps qemu: backup: Fix backup of disk skipped in an intermediate checkpoint conf: backup: Store incremental backup checkpoint name per-disk qemu: backup: Move fetching of checkpoint list for incremental backup qemublocktest: Add 'empty' test case for bitmaps qemublocktest: Add 'empty' case for incremental backup test qemublocktest: Add 'empty' case for checkpoint deletion qemublocktest: Add 'empty' case for blockcopy bitmap handling test qemublocktest: Add 'empty' case for checkpoint bitmap handling qemublocktest: Disable testcases for all bitmap handling qemublocktest: Delete 'synthetic' bitmap test cases qemublocktest: Extract printing of nodename list qemu: checkpoint: Don't chain bitmaps for checkpoints qemublocktest: Replace 'basic' bitmap detection test case data qemublocktest: Replace 'snapshots' bitmap detection test case data qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots' cases qemublocktest: Add new 'synthetic' bitmap detection and validation test case qemu: checkpoint: Don't merge checkpoints during deletion qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to TEST_CHECKPOINT_DELETE qemublocktest: Re-introduce testing of checkpoint deletion qemu: block: Add universal helper for merging dirty bitmaps for all scenarios qemu: backup: Rewrite backup bitmap handling to the new bitmap semantics qemublocktest: Add 'basic' tests for backup bitmap handling qemublocktest: Add 'snapshots' tests for backup bitmap handling qemu: Rewrite bitmap handling for block commit qemublocktest: Add 'basic' tests for commit bitmap handling qemublocktest: Add 'snapshots' tests for block commit bitmap handling qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job private data qemu: Rewrite bitmap handling for block copy qemublocktest: Add test cases for handling bitmaps during block-copy kbase: Add document outlining internals of incremental backup in qemu docs/kbase.html.in | 3 + docs/kbase/incrementalbackupinternals.rst | 210 +++ src/conf/backup_conf.c | 8 + src/conf/backup_conf.h | 1 + src/qemu/qemu_backup.c | 199 +-- src/qemu/qemu_backup.h | 12 +- src/qemu/qemu_block.c | 500 +++--- src/qemu/qemu_block.h | 21 +- src/qemu/qemu_blockjob.c | 109 +- src/qemu/qemu_blockjob.h | 3 - src/qemu/qemu_checkpoint.c | 174 +-- src/qemu/qemu_checkpoint.h | 1 - src/qemu/qemu_domain.c | 22 - src/qemu/qemu_driver.c | 70 +- tests/qemublocktest.c | 254 ++- .../backupmerge/basic-deep-out.json | 35 +- .../backupmerge/basic-flat-out.json | 23 +- .../backupmerge/basic-intermediate-out.json | 23 +- .../backupmerge/empty-out.json | 3 + .../backupmerge/snapshot-deep-out.json | 38 - .../backupmerge/snapshot-flat-out.json | 6 - .../snapshot-intermediate-out.json | 14 - .../backupmerge/snapshots-deep-out.json | 41 + .../backupmerge/snapshots-flat-out.json | 25 + .../snapshots-intermediate-out.json | 29 + tests/qemublocktestdata/bitmap/basic.json | 229 +-- tests/qemublocktestdata/bitmap/basic.out | 8 +- tests/qemublocktestdata/bitmap/empty.json | 70 + tests/qemublocktestdata/bitmap/empty.out | 1 + .../bitmap/snapshots-synthetic-broken.json | 837 ---------- .../bitmap/snapshots-synthetic-broken.out | 14 - .../snapshots-synthetic-checkpoint.json | 827 ---------- .../bitmap/snapshots-synthetic-checkpoint.out | 13 - tests/qemublocktestdata/bitmap/snapshots.json | 1382 +++++++---------- tests/qemublocktestdata/bitmap/snapshots.out | 13 +- tests/qemublocktestdata/bitmap/synthetic.json | 606 ++++++-- tests/qemublocktestdata/bitmap/synthetic.out | 19 +- .../bitmapblockcommit/basic-1-2 | 54 +- .../bitmapblockcommit/basic-1-3 | 54 +- .../bitmapblockcommit/basic-2-3 | 1 - .../qemublocktestdata/bitmapblockcommit/empty | 1 + .../bitmapblockcommit/snapshots-1-2 | 76 +- .../bitmapblockcommit/snapshots-1-3 | 77 +- .../bitmapblockcommit/snapshots-1-4 | 99 +- .../bitmapblockcommit/snapshots-1-5 | 103 +- .../bitmapblockcommit/snapshots-2-3 | 48 +- .../bitmapblockcommit/snapshots-2-4 | 58 +- .../bitmapblockcommit/snapshots-2-5 | 62 +- .../bitmapblockcommit/snapshots-3-4 | 34 +- .../bitmapblockcommit/snapshots-3-5 | 42 +- .../bitmapblockcommit/snapshots-4-5 | 17 - .../snapshots-synthetic-broken-1-2 | 57 - .../snapshots-synthetic-broken-1-3 | 112 -- .../snapshots-synthetic-broken-1-4 | 119 -- .../snapshots-synthetic-broken-1-5 | 119 -- .../snapshots-synthetic-broken-2-3 | 89 -- .../snapshots-synthetic-broken-2-4 | 96 -- .../snapshots-synthetic-broken-2-5 | 96 -- .../snapshots-synthetic-broken-3-4 | 27 - .../snapshots-synthetic-broken-3-5 | 27 - .../snapshots-synthetic-broken-4-5 | 20 - .../bitmapblockcopy/basic-deep-out.json | 53 +- .../bitmapblockcopy/basic-shallow-out.json | 53 +- .../bitmapblockcopy/empty-deep-out.json | 0 .../bitmapblockcopy/empty-shallow-out.json | 0 .../bitmapblockcopy/snapshots-deep-out.json | 95 +- .../snapshots-shallow-out.json | 98 +- .../checkpointdelete/basic-current-out.json | 20 - .../basic-intermediate1-out.json | 13 - .../basic-intermediate2-out.json | 13 - .../basic-intermediate3-out.json | 13 - .../checkpointdelete/empty-out.json | 1 + .../snapshots-current-out.json | 20 - .../snapshots-intermediate1-out.json | 20 +- .../snapshots-intermediate2-out.json | 42 +- .../snapshots-intermediate3-out.json | 43 - .../snapshots-noparent-out.json | 15 + ...hots-synthetic-checkpoint-current-out.json | 29 - ...ynthetic-checkpoint-intermediate1-out.json | 31 - ...ynthetic-checkpoint-intermediate2-out.json | 34 - ...ynthetic-checkpoint-intermediate3-out.json | 61 - ...ots-synthetic-checkpoint-noparent-out.json | 27 - .../synthetic-current-out.json | 9 + .../synthetic-intermediate1-out.json | 11 + .../synthetic-intermediate2-out.json | 11 + .../synthetic-intermediate3-out.json | 19 + .../synthetic-noparent-out.json | 11 + .../blockjob-blockdev-in.xml | 4 - 88 files changed, 2927 insertions(+), 5150 deletions(-) create mode 100644 docs/kbase/incrementalbackupinternals.rst create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json create mode 100644 tests/qemublocktestdata/bitmap/empty.json create mode 100644 tests/qemublocktestdata/bitmap/empty.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json -- 2.26.2

3 71

[libvirt PATCH] ci: Use fedora image for code style checking
by Jonathon Jongsma 22 Jun '20

22 Jun '20

Centos does not have the cppi package, so some code style checks are skipped. Switch to a fedora image to do the code style checks. Signed-off-by: Jonathon Jongsma <jjongsma(a)redhat.com> --- .gitlab-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index bfb66a652d..b8d0645e99 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -496,7 +496,7 @@ website: codestyle: stage: builds - image: $CI_REGISTRY_IMAGE/ci-centos-8:latest + image: $CI_REGISTRY_IMAGE/ci-fedora-32:latest before_script: - *script_variables script: -- 2.21.3

3 5

[RFC PATCH 00/41] qemu: Rewrite checkpoint code for 'block-dirty-bitmap-populate'
by Peter Krempa 22 Jun '20

22 Jun '20

Use 'block-dirty-bitmap-populate' and change how we create bitmaps corresponding to checkpoints to simplify the code and also properly integrate with backing chain images created outside of libvirt. This patchset changes how we approach checkpoints by keeping one bitmap per checkpoint and disk and not propagating the bitmaps into overlays on snapshots. This massively simplifies the code handling all the operations during blockjobs and backups. While the change isn't compatible with checkpoints created previously, we didn't yet enable the support for checkpoints/backups. Note that 'block-dirty-bitmap-populate' is _not_ in finished state in qemu yet, so I'm posting this as RFC and as reference for qemu developers of the usefulnes of it. The changes can be fetched by: git fetch https://gitlab.com/pipo.sk/libvirt.git checkpoint-bitmap-populate Note that the above branch also contains a commit enabling incremental backup for simpler testing. I've pushed the appropriate qemu patches for convenience here: git fetch https://gitlab.com/pipo.sk/qemu.git block-dirty-bitmap-populate Peter Krempa (41): virErrorPreserveLast: Return the saved error object in addition to storing it util: error: Introduce VIR_ERROR_AUTOPRESERVE_LAST macro qemuBackupDiskDataCleanup: Use VIR_ERROR_AUTO_PRESERVE_LAST qemu: backup: Split up code traversing checkpoint list looking for bitmaps qemu: backup: Fix backup of disk skipped in an intermediate checkpoint conf: backup: Store incremental backup checkpoint name per-disk qemu: backup: Move fetching of checkpoint list for incremental backup qemublocktest: Add 'empty' test case for bitmaps qemublocktest: Add 'empty' case for incremental backup test qemublocktest: Add 'empty' case for checkpoint deletion qemublocktest: Add 'empty' case for blockcopy bitmap handling test qemublocktest: Add 'empty' case for checkpoint bitmap handling qemublocktest: Disable testcases for all bitmap handling qemublocktest: Delete 'synthetic' bitmap test cases qemublocktest: Extract printing of nodename list qemu: checkpoint: Don't chain bitmaps for checkpoints qemublocktest: Replace 'basic' bitmap detection test case data qemu: snapshot: Don't propagate bitmaps to upper layers qemublocktest: Replace 'snapshots' bitmap detection test case data qemuBlockBitmapChainIsValid: Adjust to new semantics of bitmaps qemublocktest: Re-add bitmap validation for 'basic' and 'snapshots' cases qemublocktest: Add new 'synthetic' bitmap detection and validation test case qemu: checkpoint: Don't merge checkpoints during deletion qemublocktest: Rename TEST_CHECKPOINT_DELETE_MERGE to TEST_CHECKPOINT_DELETE qemublocktest: Re-introduce testing of checkpoint deletion qemucapabilitiestest: Bump qemu-5.1 caps on x86_64 for 'dirty-bitmap-populate' qemu: monitor: Add support for 'block-dirty-bitmap-populate' transaction member qemuDomainStorageSourcePrivate: Add per-source private blockjob qemu: blockjob: Introduce 'populate' blockjob qemu: domain: Introduce helper for always fetching virStorageSource private data qemu: block: Add helper to add temporary block bitmaps from allocation maps qemu: block: Add universal helper for merging dirty bitmaps for all scenarios qemu: backup: Rewrite backup bitmap handling to the new bitmap semantics qemublocktest: Add 'basic' tests for backup bitmap handling qemublocktest: Add 'snapshots' tests for backup bitmap handling qemu: Rewrite bitmap handling for block commit qemublocktest: Add 'basic' tests for commit bitmap handling qemublocktest: Add 'snapshots' tests for block commit bitmap handling qemu: blockjob: Remove 'disabledBitmapsBase' field from commit job private data qemu: Rewrite bitmap handling for block copy qemublocktest: Add test cases for handling bitmaps during block-copy src/conf/backup_conf.c | 8 + src/conf/backup_conf.h | 1 + src/qemu/qemu_backup.c | 221 +- src/qemu/qemu_backup.h | 13 +- src/qemu/qemu_block.c | 674 ++-- src/qemu/qemu_block.h | 35 +- src/qemu/qemu_blockjob.c | 182 +- src/qemu/qemu_blockjob.h | 16 +- src/qemu/qemu_checkpoint.c | 174 +- src/qemu/qemu_checkpoint.h | 1 - src/qemu/qemu_domain.c | 46 +- src/qemu/qemu_domain.h | 6 + src/qemu/qemu_driver.c | 112 +- src/qemu/qemu_monitor.c | 11 + src/qemu/qemu_monitor.h | 7 + src/qemu/qemu_monitor_json.c | 18 + src/qemu/qemu_monitor_json.h | 6 + src/util/virerror.c | 18 +- src/util/virerror.h | 12 +- tests/qemublocktest.c | 267 +- .../backupmerge/basic-deep-out.json | 35 +- .../backupmerge/basic-flat-out.json | 23 +- .../backupmerge/basic-intermediate-out.json | 23 +- .../backupmerge/empty-out.json | 3 + .../backupmerge/snapshot-deep-out.json | 38 - .../backupmerge/snapshot-flat-out.json | 6 - .../snapshot-intermediate-out.json | 14 - .../backupmerge/snapshots-deep-out.json | 46 + .../backupmerge/snapshots-flat-out.json | 25 + .../snapshots-intermediate-out.json | 31 + tests/qemublocktestdata/bitmap/basic.json | 229 +- tests/qemublocktestdata/bitmap/basic.out | 8 +- tests/qemublocktestdata/bitmap/empty.json | 70 + tests/qemublocktestdata/bitmap/empty.out | 1 + .../bitmap/snapshots-synthetic-broken.json | 837 ----- .../bitmap/snapshots-synthetic-broken.out | 14 - .../snapshots-synthetic-checkpoint.json | 827 ----- .../bitmap/snapshots-synthetic-checkpoint.out | 13 - tests/qemublocktestdata/bitmap/snapshots.json | 1254 +++----- tests/qemublocktestdata/bitmap/snapshots.out | 6 +- tests/qemublocktestdata/bitmap/synthetic.json | 606 +++- tests/qemublocktestdata/bitmap/synthetic.out | 19 +- .../bitmapblockcommit/basic-1-2 | 66 +- .../bitmapblockcommit/basic-1-3 | 66 +- .../bitmapblockcommit/basic-2-3 | 1 - .../qemublocktestdata/bitmapblockcommit/empty | 1 + .../bitmapblockcommit/snapshots-1-2 | 24 +- .../bitmapblockcommit/snapshots-1-3 | 42 +- .../bitmapblockcommit/snapshots-1-4 | 97 +- .../bitmapblockcommit/snapshots-1-5 | 99 +- .../bitmapblockcommit/snapshots-2-3 | 23 - .../bitmapblockcommit/snapshots-2-4 | 55 +- .../bitmapblockcommit/snapshots-2-5 | 59 +- .../bitmapblockcommit/snapshots-3-4 | 25 +- .../bitmapblockcommit/snapshots-3-5 | 29 +- .../bitmapblockcommit/snapshots-4-5 | 32 - .../snapshots-synthetic-broken-1-2 | 57 - .../snapshots-synthetic-broken-1-3 | 112 - .../snapshots-synthetic-broken-1-4 | 119 - .../snapshots-synthetic-broken-1-5 | 119 - .../snapshots-synthetic-broken-2-3 | 89 - .../snapshots-synthetic-broken-2-4 | 96 - .../snapshots-synthetic-broken-2-5 | 96 - .../snapshots-synthetic-broken-3-4 | 27 - .../snapshots-synthetic-broken-3-5 | 27 - .../snapshots-synthetic-broken-4-5 | 20 - .../bitmapblockcopy/basic-deep-out.json | 65 +- .../bitmapblockcopy/basic-shallow-out.json | 65 +- .../bitmapblockcopy/empty-deep-out.json | 0 .../bitmapblockcopy/empty-shallow-out.json | 0 .../bitmapblockcopy/snapshots-deep-out.json | 104 +- .../snapshots-shallow-out.json | 24 +- .../checkpointdelete/basic-current-out.json | 20 - .../basic-intermediate1-out.json | 13 - .../basic-intermediate2-out.json | 13 - .../basic-intermediate3-out.json | 13 - .../checkpointdelete/empty-out.json | 1 + .../snapshots-current-out.json | 20 - .../snapshots-intermediate1-out.json | 13 - .../snapshots-intermediate2-out.json | 51 - .../snapshots-intermediate3-out.json | 50 - .../snapshots-noparent-out.json | 16 - ...hots-synthetic-checkpoint-current-out.json | 29 - ...ynthetic-checkpoint-intermediate1-out.json | 31 - ...ynthetic-checkpoint-intermediate2-out.json | 34 - ...ynthetic-checkpoint-intermediate3-out.json | 61 - ...ots-synthetic-checkpoint-noparent-out.json | 27 - .../synthetic-current-out.json | 9 + .../synthetic-intermediate1-out.json | 11 + .../synthetic-intermediate2-out.json | 11 + .../synthetic-intermediate3-out.json | 19 + .../synthetic-noparent-out.json | 11 + .../caps_5.1.0.x86_64.replies | 2737 +++++++++-------- .../caps_5.1.0.x86_64.xml | 2 +- tests/qemumonitorjsontest.c | 3 +- .../blockjob-blockdev-in.xml | 4 - 96 files changed, 4077 insertions(+), 6717 deletions(-) create mode 100644 tests/qemublocktestdata/backupmerge/empty-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-deep-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-flat-out.json delete mode 100644 tests/qemublocktestdata/backupmerge/snapshot-intermediate-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-deep-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-flat-out.json create mode 100644 tests/qemublocktestdata/backupmerge/snapshots-intermediate-out.json create mode 100644 tests/qemublocktestdata/bitmap/empty.json create mode 100644 tests/qemublocktestdata/bitmap/empty.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-broken.out delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.json delete mode 100644 tests/qemublocktestdata/bitmap/snapshots-synthetic-checkpoint.out create mode 100644 tests/qemublocktestdata/bitmapblockcommit/empty delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-2 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-1-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-3 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-2-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-4 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-3-5 delete mode 100644 tests/qemublocktestdata/bitmapblockcommit/snapshots-synthetic-broken-4-5 create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-deep-out.json create mode 100644 tests/qemublocktestdata/bitmapblockcopy/empty-shallow-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/empty-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-current-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate1-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate2-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-intermediate3-out.json delete mode 100644 tests/qemublocktestdata/checkpointdelete/snapshots-synthetic-checkpoint-noparent-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-current-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate1-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate2-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-intermediate3-out.json create mode 100644 tests/qemublocktestdata/checkpointdelete/synthetic-noparent-out.json -- 2.26.2

2 46

[PATCH] virsh: Fix typo of metadata
by Han Han 22 Jun '20

22 Jun '20

Signed-off-by: Han Han <hhan(a)redhat.com> --- tools/virsh-domain.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index 5222949566..b7dea5e8f7 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -13559,7 +13559,7 @@ virshEventMetadataChangePrint(virConnectPtr conn G_GNUC_UNUSED, { virBuffer buf = VIR_BUFFER_INITIALIZER; - virBufferAsprintf(&buf, _("event 'metdata-change' for domain %s: %s %s\n"), + virBufferAsprintf(&buf, _("event 'metadata-change' for domain %s: %s %s\n"), virDomainGetName(dom), UNKNOWNSTR(virshEventMetadataChangeTypeTypeToString(type)), NULLSTR(nsuri)); -- 2.24.0.rc1

2 1

[PATCH v5 0/4] introduction of migration_version attribute for VFIO live migration
by Yan Zhao 22 Jun '20

22 Jun '20

This patchset introduces a migration_version attribute under sysfs of VFIO Mediated devices. This migration_version attribute is used to check migration compatibility between two mdev devices. Currently, it has two locations: (1) under mdev_type node, which can be used even before device creation, but only for mdev devices of the same mdev type. (2) under mdev device node, which can only be used after the mdev devices are created, but the src and target mdev devices are not necessarily be of the same mdev type (The second location is newly added in v5, in order to keep consistent with the migration_version node for migratable pass-though devices) Patch 1 defines migration_version attribute for the first location in Documentation/vfio-mediated-device.txt Patch 2 uses GVT as an example for patch 1 to show how to expose migration_version attribute and check migration compatibility in vendor driver. Patch 3 defines migration_version attribute for the second location in Documentation/vfio-mediated-device.txt Patch 4 uses GVT as an example for patch 3 to show how to expose migration_version attribute and check migration compatibility in vendor driver. (The previous "Reviewed-by" and "Acked-by" for patch 1 and patch 2 are kept in v5, as there are only small changes to commit messages of the two patches.) v5: added patch 2 and 4 for mdev device part of migration_version attribute. v4: 1. fixed indentation/spell errors, reworded several error messages 2. added a missing memory free for error handling in patch 2 v3: 1. renamed version to migration_version 2. let errno to be freely defined by vendor driver 3. let checking mdev_type be prerequisite of migration compatibility check 4. reworded most part of patch 1 5. print detailed error log in patch 2 and generate migration_version string at init time v2: 1. renamed patched 1 2. made definition of device version string completely private to vendor driver 3. reverted changes to sample mdev drivers 4. described intent and usage of version attribute more clearly. Yan Zhao (4): vfio/mdev: add migration_version attribute for mdev (under mdev_type node) drm/i915/gvt: export migration_version to mdev sysfs (under mdev_type node) vfio/mdev: add migration_version attribute for mdev (under mdev device node) drm/i915/gvt: export migration_version to mdev sysfs (under mdev device node) .../driver-api/vfio-mediated-device.rst | 183 ++++++++++++++++++ drivers/gpu/drm/i915/gvt/Makefile | 2 +- drivers/gpu/drm/i915/gvt/gvt.c | 39 ++++ drivers/gpu/drm/i915/gvt/gvt.h | 7 + drivers/gpu/drm/i915/gvt/kvmgt.c | 55 ++++++ drivers/gpu/drm/i915/gvt/migration_version.c | 170 ++++++++++++++++ drivers/gpu/drm/i915/gvt/vgpu.c | 13 +- 7 files changed, 466 insertions(+), 3 deletions(-) create mode 100644 drivers/gpu/drm/i915/gvt/migration_version.c -- 2.17.1

7 39

[PATCH 0/1] configure: prefer python's sphinx module
by John Snow 19 Jun '20

19 Jun '20

Hi, this is a followup to my patch which attempted to prefer 'sphinx-build' to 'sphinx-build-3' which intended to prefer a VENV installation of sphinx over Fedora's system installation. That got confusing, though, so here's something that's simpler and better: use python to invoke sphinx so we don't have to worry about python/sphinx version mismatches. This should essentially always DTRT. In cases where a specific sphinx is preferred, you would use a virtual environment with sphinx installed and configure QEMU with e.g. --python /path/to/my/venv/python35 John Snow (1): configure: prefer python's sphinx module docs/system/deprecated.rst | 19 +++++++++++++++++++ configure | 20 +++++++++----------- 2 files changed, 28 insertions(+), 11 deletions(-) -- 2.21.3

3 6

[libvirt-sandbox PATCH] gitlab: introduce CI jobs testing git master & distro libvirt
by Daniel P. Berrangé 19 Jun '20

19 Jun '20

The sandbox build needs to validate one axis - A variety of libvirt versions We test a variety of libvirt versions by running a build against the distro provided libvirt packages. All that is then missing is a build against the latest libvirt git master, which only needs to be run on a single distro, for which Ubuntu 20.04 is picked as a stable long life base. For mingw, we only test against libvirt git master, using Fedora rawhide. It is expected that any issues against older libvirt will have already been caught by the native builds. Several distros are not included because they don't provide the package pre-requisites. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- .gitlab-ci.yml | 163 ++++++++++++++++++ ci/containers/README.rst | 14 ++ ci/containers/libvirt-debian-10.Dockerfile | 64 +++++++ ci/containers/libvirt-debian-9.Dockerfile | 67 +++++++ ci/containers/libvirt-debian-sid.Dockerfile | 64 +++++++ ci/containers/libvirt-fedora-31.Dockerfile | 64 +++++++ ci/containers/libvirt-fedora-32.Dockerfile | 64 +++++++ .../libvirt-fedora-rawhide.Dockerfile | 65 +++++++ ci/containers/libvirt-ubuntu-1804.Dockerfile | 67 +++++++ ci/containers/libvirt-ubuntu-2004.Dockerfile | 69 ++++++++ ci/containers/refresh | 27 +++ 11 files changed, 728 insertions(+) create mode 100644 ci/containers/README.rst create mode 100644 ci/containers/libvirt-debian-10.Dockerfile create mode 100644 ci/containers/libvirt-debian-9.Dockerfile create mode 100644 ci/containers/libvirt-debian-sid.Dockerfile create mode 100644 ci/containers/libvirt-fedora-31.Dockerfile create mode 100644 ci/containers/libvirt-fedora-32.Dockerfile create mode 100644 ci/containers/libvirt-fedora-rawhide.Dockerfile create mode 100644 ci/containers/libvirt-ubuntu-1804.Dockerfile create mode 100644 ci/containers/libvirt-ubuntu-2004.Dockerfile create mode 100755 ci/containers/refresh diff --git a/.gitlab-ci.yml b/.gitlab-ci.yml index 50dae92..c12a262 100644 --- a/.gitlab-ci.yml +++ b/.gitlab-ci.yml @@ -1,6 +1,87 @@ stages: - prebuild + - containers + - builds + +.container_job_template: &container_job_definition + image: docker:stable + stage: containers + services: + - docker:dind + before_script: + - export TAG="$CI_REGISTRY_IMAGE/ci-$NAME:latest" + - export COMMON_TAG="$CI_REGISTRY/libvirt/libvirt-sandbox/ci-$NAME:latest" + - docker info + - docker login registry.gitlab.com -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" + script: + - docker pull "$TAG" || docker pull "$COMMON_TAG" || true + - docker build --cache-from "$TAG" --cache-from "$COMMON_TAG" --tag "$TAG" -f "ci/containers/libvirt-$NAME.Dockerfile" ci + - docker push "$TAG" + after_script: + - docker logout + +.script_variables: &script_variables | + export MAKEFLAGS="-j$(getconf _NPROCESSORS_ONLN)" + export SCRATCH_DIR="/tmp/scratch" + export VROOT="$SCRATCH_DIR/vroot" + export CCACHE_DIR="$PWD/ccache" + export CCACHE_MAXSIZE="500M" + export PATH="$CCACHE_WRAPPERSDIR:$VROOT/bin:$PATH" + export PKG_CONFIG_PATH="$VROOT/lib/pkgconfig" + export XDG_DATA_DIRS="$VROOT/share:/usr/share" + export GI_TYPELIB_PATH="$VROOT/lib/girepository-1.0" + +.git_native_build_job_template: &git_native_build_job_definition + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds + cache: + paths: + - ccache/ + key: "$CI_JOB_NAME" + before_script: + - *script_variables + - export LD_LIBRARY_PATH="$VROOT/lib" + script: + - pushd "$PWD" + - mkdir -p "$SCRATCH_DIR" + - cd "$SCRATCH_DIR" + - git clone --depth 1 https://gitlab.com/libvirt/libvirt.git + - git clone --depth 1 https://gitlab.com/libvirt/libvirt-glib.git + - mkdir libvirt/build + - cd libvirt/build + - ../autogen.sh --prefix="$VROOT" --without-libvirtd + - $MAKE install + - cd ../.. + - mkdir libvirt-glib/build + - cd libvirt-glib/build + - ../autogen.sh --prefix="$VROOT" --without-libvirtd + - $MAKE install + - popd + - mkdir build + - cd build + - ../autogen.sh --prefix="$VROOT" --enable-gtk-doc + - $MAKE install + - $MAKE dist + - if test -x /usr/bin/rpmbuild && test "$RPM" != "skip" ; then rpmbuild --nodeps -ta libvirt-sandbox*.tar.gz ; fi + +.dist_native_build_job_template: &dist_native_build_job_definition + image: $CI_REGISTRY_IMAGE/ci-$NAME:latest + stage: builds + cache: + paths: + - ccache/ + key: "$CI_JOB_NAME" + before_script: + - *script_variables + script: + - mkdir build + - cd build + - ../autogen.sh --prefix="$VROOT" --enable-gtk-doc + - $MAKE install + - $MAKE dist + - if test -x /usr/bin/rpmbuild && test "$RPM" != "skip" ; then rpmbuild -ta libvirt-sandbox*.tar.gz ; fi + # Check that all commits are signed-off for the DCO. # Skip on "libvirt" namespace, since we only need to run @@ -14,3 +95,85 @@ check-dco: except: variables: - $CI_PROJECT_NAMESPACE == 'libvirt' + +x64-debian-9-container: + <<: *container_job_definition + variables: + NAME: debian-9 + +x64-debian-10-container: + <<: *container_job_definition + variables: + NAME: debian-10 + +x64-debian-sid-container: + <<: *container_job_definition + variables: + NAME: debian-sid + +x64-fedora-31-container: + <<: *container_job_definition + variables: + NAME: fedora-31 + +x64-fedora-32-container: + <<: *container_job_definition + variables: + NAME: fedora-32 + +x64-fedora-rawhide-container: + <<: *container_job_definition + variables: + NAME: fedora-rawhide + +x64-ubuntu-1804-container: + <<: *container_job_definition + variables: + NAME: ubuntu-1804 + +x64-ubuntu-2004-container: + <<: *container_job_definition + variables: + NAME: ubuntu-2004 + + + +x64-debian-9-dist-build: + <<: *dist_native_build_job_definition + variables: + NAME: debian-9 + +x64-debian-10-dist-build: + <<: *dist_native_build_job_definition + variables: + NAME: debian-10 + +x64-debian-sid-dist-build: + <<: *dist_native_build_job_definition + variables: + NAME: debian-sid + +x64-fedora-31-dist-build: + <<: *dist_native_build_job_definition + variables: + NAME: fedora-31 + +x64-fedora-32-dist-build: + <<: *dist_native_build_job_definition + variables: + NAME: fedora-32 + +x64-fedora-rawhide-dist-build: + <<: *dist_native_build_job_definition + variables: + NAME: fedora-rawhide + +x64-ubuntu-1804-dist-build: + <<: *dist_native_build_job_definition + variables: + NAME: ubuntu-1804 + +x64-ubuntu-2004-git-build: + <<: *git_native_build_job_definition + variables: + NAME: ubuntu-2004 diff --git a/ci/containers/README.rst b/ci/containers/README.rst new file mode 100644 index 0000000..530897e --- /dev/null +++ b/ci/containers/README.rst @@ -0,0 +1,14 @@ +CI job assets +============= + +This directory contains assets used in the automated CI jobs, most +notably the Dockerfiles used to build container images in which the +CI jobs then run. + +The ``refresh`` script is used to re-create the Dockerfiles using the +``lcitool`` command that is provided by repo +https://gitlab.com/libvirt/libvirt-ci + +The containers are built during the CI process and cached in the GitLab +container registry of the project doing the build. The cached containers +can be deleted at any time and will be correctly rebuilt. diff --git a/ci/containers/libvirt-debian-10.Dockerfile b/ci/containers/libvirt-debian-10.Dockerfile new file mode 100644 index 0000000..51f79e2 --- /dev/null +++ b/ci/containers/libvirt-debian-10.Dockerfile @@ -0,0 +1,64 @@ +FROM debian:10 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + gtk-doc-tools \ + libc-dev-bin \ + libc6-dev \ + libcap-ng-dev \ + libgirepository1.0-dev \ + libglib2.0-dev \ + liblzma-dev \ + libselinux1-dev \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + libvirt-glib-1.0-dev \ + libxml2-dev \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim \ + zlib1g-dev && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/libvirt-debian-9.Dockerfile b/ci/containers/libvirt-debian-9.Dockerfile new file mode 100644 index 0000000..cf94a67 --- /dev/null +++ b/ci/containers/libvirt-debian-9.Dockerfile @@ -0,0 +1,67 @@ +FROM debian:9 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + gtk-doc-tools \ + libc-dev-bin \ + libc6-dev \ + libcap-ng-dev \ + libgirepository1.0-dev \ + libglib2.0-dev \ + liblzma-dev \ + libselinux1-dev \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + libvirt-glib-1.0-dev \ + libxml2-dev \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim \ + zlib1g-dev && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/libvirt-debian-sid.Dockerfile b/ci/containers/libvirt-debian-sid.Dockerfile new file mode 100644 index 0000000..6a34306 --- /dev/null +++ b/ci/containers/libvirt-debian-sid.Dockerfile @@ -0,0 +1,64 @@ +FROM debian:sid + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + gtk-doc-tools \ + libc-dev-bin \ + libc6-dev \ + libcap-ng-dev \ + libgirepository1.0-dev \ + libglib2.0-dev \ + liblzma-dev \ + libselinux1-dev \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + libvirt-glib-1.0-dev \ + libxml2-dev \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim \ + zlib1g-dev && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/libvirt-fedora-31.Dockerfile b/ci/containers/libvirt-fedora-31.Dockerfile new file mode 100644 index 0000000..6e0da18 --- /dev/null +++ b/ci/containers/libvirt-fedora-31.Dockerfile @@ -0,0 +1,64 @@ +FROM fedora:31 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + glibc-static \ + gobject-introspection-devel \ + gtk-doc \ + libcap-ng-devel \ + libselinux-devel \ + libtirpc-devel \ + libtool \ + libvirt-devel \ + libvirt-gobject-devel \ + libxml2-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim \ + xz-devel \ + xz-static \ + zlib-devel \ + zlib-static && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/libvirt-fedora-32.Dockerfile b/ci/containers/libvirt-fedora-32.Dockerfile new file mode 100644 index 0000000..5d86af2 --- /dev/null +++ b/ci/containers/libvirt-fedora-32.Dockerfile @@ -0,0 +1,64 @@ +FROM fedora:32 + +RUN dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + glibc-static \ + gobject-introspection-devel \ + gtk-doc \ + libcap-ng-devel \ + libselinux-devel \ + libtirpc-devel \ + libtool \ + libvirt-devel \ + libvirt-gobject-devel \ + libxml2-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim \ + xz-devel \ + xz-static \ + zlib-devel \ + zlib-static && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/libvirt-fedora-rawhide.Dockerfile b/ci/containers/libvirt-fedora-rawhide.Dockerfile new file mode 100644 index 0000000..8a299fd --- /dev/null +++ b/ci/containers/libvirt-fedora-rawhide.Dockerfile @@ -0,0 +1,65 @@ +FROM fedora:rawhide + +RUN dnf update -y --nogpgcheck fedora-gpg-keys && \ + dnf update -y && \ + dnf install -y \ + autoconf \ + automake \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + cppi \ + gcc \ + gdb \ + gettext \ + gettext-devel \ + git \ + glib2-devel \ + glibc-devel \ + glibc-langpack-en \ + glibc-static \ + gobject-introspection-devel \ + gtk-doc \ + libcap-ng-devel \ + libselinux-devel \ + libtirpc-devel \ + libtool \ + libvirt-devel \ + libvirt-gobject-devel \ + libxml2-devel \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconfig \ + python3 \ + python3-setuptools \ + python3-wheel \ + rpcgen \ + rpm-build \ + screen \ + strace \ + sudo \ + vim \ + xz-devel \ + xz-static \ + zlib-devel \ + zlib-static && \ + dnf autoremove -y && \ + dnf clean all -y && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/libvirt-ubuntu-1804.Dockerfile b/ci/containers/libvirt-ubuntu-1804.Dockerfile new file mode 100644 index 0000000..bea1fb4 --- /dev/null +++ b/ci/containers/libvirt-ubuntu-1804.Dockerfile @@ -0,0 +1,67 @@ +FROM ubuntu:18.04 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + gtk-doc-tools \ + libc-dev-bin \ + libc6-dev \ + libcap-ng-dev \ + libgirepository1.0-dev \ + libglib2.0-dev \ + liblzma-dev \ + libselinux1-dev \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libvirt-dev \ + libvirt-glib-1.0-dev \ + libxml2-dev \ + locales \ + lsof \ + make \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-pip \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + vim \ + zlib1g-dev && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +RUN pip3 install \ + meson==0.49.0 + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/libvirt-ubuntu-2004.Dockerfile b/ci/containers/libvirt-ubuntu-2004.Dockerfile new file mode 100644 index 0000000..0c2c4d6 --- /dev/null +++ b/ci/containers/libvirt-ubuntu-2004.Dockerfile @@ -0,0 +1,69 @@ +FROM ubuntu:20.04 + +RUN export DEBIAN_FRONTEND=noninteractive && \ + apt-get update && \ + apt-get dist-upgrade -y && \ + apt-get install --no-install-recommends -y \ + autoconf \ + automake \ + autopoint \ + bash \ + bash-completion \ + ca-certificates \ + ccache \ + chrony \ + gcc \ + gdb \ + gettext \ + git \ + gtk-doc-tools \ + libc-dev-bin \ + libc6-dev \ + libcap-ng-dev \ + libgirepository1.0-dev \ + libglib2.0-dev \ + libgnutls28-dev \ + liblzma-dev \ + libnl-3-dev \ + libnl-route-3-dev \ + libselinux1-dev \ + libtirpc-dev \ + libtool \ + libtool-bin \ + libxml2-dev \ + libxml2-utils \ + locales \ + lsof \ + make \ + meson \ + net-tools \ + ninja-build \ + patch \ + perl \ + pkgconf \ + python3 \ + python3-docutils \ + python3-setuptools \ + python3-wheel \ + screen \ + strace \ + sudo \ + valac \ + vim \ + xsltproc \ + zlib1g-dev && \ + apt-get autoremove -y && \ + apt-get autoclean -y && \ + sed -Ei 's,^# (en_US\.UTF-8 .*)$,\1,' /etc/locale.gen && \ + dpkg-reconfigure locales && \ + mkdir -p /usr/libexec/ccache-wrappers && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/cc && \ + ln -s /usr/bin/ccache /usr/libexec/ccache-wrappers/$(basename /usr/bin/gcc) + +ENV LANG "en_US.UTF-8" + +ENV MAKE "/usr/bin/make" +ENV NINJA "/usr/bin/ninja" +ENV PYTHON "/usr/bin/python3" + +ENV CCACHE_WRAPPERSDIR "/usr/libexec/ccache-wrappers" diff --git a/ci/containers/refresh b/ci/containers/refresh new file mode 100755 index 0000000..ce65700 --- /dev/null +++ b/ci/containers/refresh @@ -0,0 +1,27 @@ +#!/bin/sh + +if test -z "$1" +then + echo "syntax: $0 PATH-TO-LCITOOL" + exit 1 +fi + +LCITOOL=$1 + +if ! test -x "$LCITOOL" +then + echo "$LCITOOL is not executable" + exit 1 +fi + +HOSTS=$($LCITOOL hosts | grep -v -E '(freebsd|centos|opensuse)') + +for host in $HOSTS +do + if test "$host" = "libvirt-ubuntu-2004" + then + $LCITOOL dockerfile $host libvirt+minimal,libvirt-glib,libvirt-sandbox > $host.Dockerfile + else + $LCITOOL dockerfile $host libvirt+dist,libvirt-glib+dist,libvirt-sandbox > $host.Dockerfile + fi +done -- 2.26.2

2 3

[libvirt PATCH v4 00/12] Add ability to create mediated devices in libvirt
by Jonathon Jongsma 19 Jun '20

19 Jun '20

This is the first portion of an effort to support persistent mediated devices with libvirt. This first series simply enables creating and destroying non-persistent mediated devices via the virNodeDeviceCreateXML() and virNodeDeviceDestroy() functions. The 'mdevctl' utility[1] provides the backend implementation. Hopefully these are the last changes and version can simply be pushed upstream. Changes in v4: - coding style / spacing fixes - remove 'persist' arg from start command - fixed distcheck failure by including test data dir in EXTRA_DIST - Add an item to NEWS.rst [1] https://github.com/mdevctl/mdevctl Jonathon Jongsma (12): nodedev: make iommuGroup optional for mdevs nodedev: factor out nodeDeviceHasCapability() nodedev: add support for mdev attributes nodedev: refactor nodeDeviceFindNewDevice() nodedev: store mdev UUID in mdev caps nodedev: add mdev support to virNodeDeviceCreateXML() nodedev: Build a non-loadable driver lib nodedev: Add testing for 'mdevctl start' nodedev: add mdev support to virNodeDeviceDestroy() nodedev: Add testing for 'mdevctl stop' docs: note node device fields that are read-only news: mediated devices can be created NEWS.rst | 7 + build-aux/syntax-check.mk | 2 +- docs/formatnode.html.in | 21 +- docs/schemas/nodedev.rng | 18 +- libvirt.spec.in | 2 + m4/virt-external-programs.m4 | 3 + src/conf/node_device_conf.c | 68 +++- src/conf/node_device_conf.h | 3 + src/conf/virnodedeviceobj.c | 34 ++ src/conf/virnodedeviceobj.h | 3 + src/libvirt_private.syms | 3 + src/node_device/Makefile.inc.am | 23 +- src/node_device/node_device_driver.c | 360 ++++++++++++++++-- src/node_device/node_device_driver.h | 7 + src/node_device/node_device_udev.c | 5 +- src/util/virmdev.c | 12 + src/util/virmdev.h | 11 + tests/Makefile.am | 15 + ...019_36ea_4111_8f0a_8c9a70e21366-start.argv | 1 + ...019_36ea_4111_8f0a_8c9a70e21366-start.json | 1 + ...d39_495e_4243_ad9f_beb3f14c23d9-start.argv | 1 + ...d39_495e_4243_ad9f_beb3f14c23d9-start.json | 1 + ...916_1ca8_49ac_b176_871d16c13076-start.argv | 1 + ...916_1ca8_49ac_b176_871d16c13076-start.json | 1 + tests/nodedevmdevctldata/mdevctl-stop.argv | 1 + tests/nodedevmdevctltest.c | 303 +++++++++++++++ ...v_d069d019_36ea_4111_8f0a_8c9a70e21366.xml | 7 + ...v_d2441d39_495e_4243_ad9f_beb3f14c23d9.xml | 9 + ...v_fedc4916_1ca8_49ac_b176_871d16c13076.xml | 8 + 29 files changed, 863 insertions(+), 68 deletions(-) create mode 100644 tests/nodedevmdevctldata/mdev_d069d019_36ea_4111_8f0a_8c9a70e21366-start.argv create mode 100644 tests/nodedevmdevctldata/mdev_d069d019_36ea_4111_8f0a_8c9a70e21366-start.json create mode 100644 tests/nodedevmdevctldata/mdev_d2441d39_495e_4243_ad9f_beb3f14c23d9-start.argv create mode 100644 tests/nodedevmdevctldata/mdev_d2441d39_495e_4243_ad9f_beb3f14c23d9-start.json create mode 100644 tests/nodedevmdevctldata/mdev_fedc4916_1ca8_49ac_b176_871d16c13076-start.argv create mode 100644 tests/nodedevmdevctldata/mdev_fedc4916_1ca8_49ac_b176_871d16c13076-start.json create mode 100644 tests/nodedevmdevctldata/mdevctl-stop.argv create mode 100644 tests/nodedevmdevctltest.c create mode 100644 tests/nodedevschemadata/mdev_d069d019_36ea_4111_8f0a_8c9a70e21366.xml create mode 100644 tests/nodedevschemadata/mdev_d2441d39_495e_4243_ad9f_beb3f14c23d9.xml create mode 100644 tests/nodedevschemadata/mdev_fedc4916_1ca8_49ac_b176_871d16c13076.xml -- 2.21.3

4 18

[libvirt PATCH] nodedev: Makefile: Use AM_LDFLAGS_MOD_NOUNDEF with the installed library
by Erik Skultety 19 Jun '20

19 Jun '20

Rather than using the AM_LDFLAGS_MOD_NOUNDEF options with the noinstall library that will come out of libtool from libvirt_driver_nodedev_impl_la, use it with the installed version libvirt_driver_nodedev_la. Broken-by-commit: c44bffb9 Fixes: https://ci.centos.org/job/libvirt-rpm/systems=libvirt-fedora-32/1155/ Signed-off-by: Erik Skultety <eskultet(a)redhat.com> --- Technically a build fix, but I'd like one more pair of eyes to look at this, especially if they understand libtool. src/node_device/Makefile.inc.am | 1 + 1 file changed, 1 insertion(+) diff --git a/src/node_device/Makefile.inc.am b/src/node_device/Makefile.inc.am index 5993165b56..abe26b23bf 100644 --- a/src/node_device/Makefile.inc.am +++ b/src/node_device/Makefile.inc.am @@ -37,6 +37,7 @@ if WITH_NODE_DEVICES noinst_LTLIBRARIES += libvirt_driver_nodedev_impl.la libvirt_driver_nodedev_la_SOURCES = libvirt_driver_nodedev_la_LIBADD = libvirt_driver_nodedev_impl.la +libvirt_driver_nodedev_la_LDFLAGS = $(AM_LDFLAGS_MOD_NOUNDEF) mod_LTLIBRARIES += libvirt_driver_nodedev.la libvirt_driver_nodedev_impl_la_SOURCES = $(NODE_DEVICE_DRIVER_SOURCES) -- 2.26.2

2 3

[PATCH] qemu: auto-assign hostdev interface devices to PCIe
by Laine Stump 18 Jun '20

18 Jun '20

Until recently, an <interface type='network'> would automatically be assigned model "rtl8139" (the default on x86 machinetypes), which in turn would lead to the device being assigned a PCI address on a conventional PCI controller (i.e. a pcie-to-pci-bridge). If the network was a typical Linux host bridge-based network that used an emulated device, this would be appropriate, since the guest actually would get an emulated rtl8139 NIC, and that device is a conventional PCI device. However, if the network being used was a pool of hostdev devices, the guest would get an actual PCIe network device assigned from the host via VFIO; while the interface model in that case is irrelevant for the QEMU commandline to assign the device, the PCI address would have already been assigned prior to runtime, so the address assignment would be done based on the model='rtl8139' - a conventional PCI device. VFIO assignment of a PCIe device to a conventional PCI slot works, but we would rather have these devices in a PCIe slot. Since commit bdb8f2e4186, if <interface type='network'> points to a etwork that is a pool of hostdev devices, the interface model will be _unset_ by default. This patch uses that information when deciding what type of slot to assign to the device: since all hostdev network interfaces are SR-IOV VFs, and *all* SR-IOV network cards are PCIe, it is safe to assume that the VFs are PCIe and we should assign then to a PCIe slot in the guest. Signed-off-by: Laine Stump <laine(a)redhat.com> --- (NB: all talk of PCIe vs. conventional PCI is irrelevant for machinetypes that don't support PCIe - any requirement for PCIe is "squashed" down to just "PCI" in those cases) (Note to Paulo - I realize this doesn't describe exactly what happens on s390, since the default interface model in that case is "virtio" rather than "rtl8139", and this patch should actually cause no behavior change on S390. I'm Cc'ing you since you're the author of the other patch I mention in the commit message, and also so you can try running your same tests with this patch added, and verify that it really doesn't break anything for S390.) src/qemu/qemu_domain_address.c | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c index 07431343ed..05cf251cd5 100644 --- a/src/qemu/qemu_domain_address.c +++ b/src/qemu/qemu_domain_address.c @@ -734,6 +734,14 @@ qemuDomainDeviceCalculatePCIConnectFlags(virDomainDeviceDefPtr dev, if (net->model == VIR_DOMAIN_NET_MODEL_E1000E) return pcieFlags; + /* the only time model can be "unknown" is for type='hostdev' + * or for type='network' where the network is a pool of + * hostdev devices. These will always be pcie on the host, and + * should be pcie in the guest if it supports pcie. + */ + if (net->model == VIR_DOMAIN_NET_MODEL_UNKNOWN) + return pcieFlags; + return pciFlags; } -- 2.25.4

3 2