July 2019 - Devel - libvirt List Archives

[libvirt] [PATCH 00/11] qemu: Add formatter for 'blockdev-create' JSON (blockdev-add saga)
by Peter Krempa 08 Jul '19

08 Jul '19

To allow using blockdev-create we need to be able to generate the properties for it, this series adds the necessary bits and also converts to use 'auto-read-only' instead of 'read-only'. Kevin, please look into the test data to see if there's something wrong with the semantic usage. The data is validated against the QMP schema. Format in 2/11 will be used as the 'backing-store' string via the json:// pseudo protocol for recording into new files for network use. 5/11 is the conversion to use auto-read-only. The test data in the patch covers stadard use with -blockdev/blockdev-add 11/11 is the format to be used with 'blockdev-create' Thanks. Peter Krempa (11): qemu: block: Allow skipping non-target related data when formating disk JSON tests: qemublock: Add testing of pure disk source specification JSON generator util: json: Introduce helper to prepend string into a virJSONValueObject qemu: block: Extract formating of 'driver' attribute from child formatters qemu: block: Use 'auto-read-only' instead of 'read-only' for backing chain qemu: block: Add generator for image format creation properties qemu: block: Add generator for creating storage with blockdev-create qemu: domain: Export qemuDomainPrepareStorageSourceBlockdev conf: snapshot: Export virDomainSnapshotDiskDefParseXML conf: snapshot: Provide a function to free virDomainSnapshotDiskDefPtr tests: qemublock: Add testing of 'blockdev-create' generators src/conf/snapshot_conf.c | 13 +- src/conf/snapshot_conf.h | 6 + src/libvirt_private.syms | 1 + src/qemu/qemu_block.c | 584 ++++++++++++++++-- src/qemu/qemu_block.h | 18 +- src/qemu/qemu_command.c | 4 +- src/qemu/qemu_domain.c | 9 +- src/qemu/qemu_domain.h | 5 + src/qemu/qemu_migration.c | 4 +- src/util/virjson.c | 50 +- src/util/virjson.h | 1 + tests/qemublocktest.c | 244 +++++++- .../imagecreate/luks-encopts.json | 19 + .../imagecreate/luks-encopts.xml | 16 + .../imagecreate/luks-noopts.json | 14 + .../imagecreate/luks-noopts.xml | 14 + .../imagecreate/network-gluster-qcow2.json | 28 + .../imagecreate/network-gluster-qcow2.xml | 13 + .../imagecreate/network-rbd-qcow2.json | 26 + .../imagecreate/network-rbd-qcow2.xml | 13 + .../imagecreate/network-sheepdog-qcow2.json | 20 + .../imagecreate/network-sheepdog-qcow2.xml | 12 + .../imagecreate/network-ssh-qcow2.json | 19 + .../imagecreate/network-ssh-qcow2.xml | 12 + .../imagecreate/qcow2-backing-luks.json | 15 + .../imagecreate/qcow2-backing-luks.xml | 1 + .../imagecreate/qcow2-backing-raw-nbd.json | 15 + .../imagecreate/qcow2-backing-raw-nbd.xml | 1 + .../imagecreate/qcow2-backing-raw.json | 15 + .../imagecreate/qcow2-backing-raw.xml | 1 + .../qcow2-luks-encopts-backing.json | 24 + .../qcow2-luks-encopts-backing.xml | 1 + .../imagecreate/qcow2-luks-encopts.json | 22 + .../imagecreate/qcow2-luks-encopts.xml | 16 + .../imagecreate/qcow2-luks-noopts.json | 17 + .../imagecreate/qcow2-luks-noopts.xml | 14 + .../qemublocktestdata/imagecreate/qcow2.json | 13 + tests/qemublocktestdata/imagecreate/qcow2.xml | 11 + .../imagecreate/raw-nbd.json | 4 + .../qemublocktestdata/imagecreate/raw-nbd.xml | 12 + tests/qemublocktestdata/imagecreate/raw.json | 9 + tests/qemublocktestdata/imagecreate/raw.xml | 11 + .../xml2json/block-raw-noopts-srconly.json | 4 + .../xml2json/block-raw-noopts.json | 2 +- .../block-raw-reservations-srconly.json | 4 + .../xml2json/block-raw-reservations.json | 2 +- .../xml2json/dir-fat-cache-srconly.json | 5 + .../xml2json/dir-fat-cache.json | 2 +- .../xml2json/dir-fat-floppy-srconly.json | 5 + .../xml2json/dir-fat-floppy.json | 2 +- .../xml2json/dir-fat-readonly-srconly.json | 5 + .../xml2json/dir-fat-readonly.json | 2 +- ...ile-backing_basic-aio_threads-srconly.json | 24 + .../file-backing_basic-aio_threads.json | 8 +- ...acking_basic-cache-directsync-srconly.json | 24 + .../file-backing_basic-cache-directsync.json | 8 +- ...file-backing_basic-cache-none-srconly.json | 24 + .../file-backing_basic-cache-none.json | 8 +- ...le-backing_basic-cache-unsafe-srconly.json | 24 + .../file-backing_basic-cache-unsafe.json | 8 +- ...backing_basic-cache-writeback-srconly.json | 24 + .../file-backing_basic-cache-writeback.json | 8 +- ...king_basic-cache-writethrough-srconly.json | 24 + ...file-backing_basic-cache-writethrough.json | 8 +- .../file-backing_basic-detect-srconly.json | 24 + .../xml2json/file-backing_basic-detect.json | 8 +- .../file-backing_basic-noopts-srconly.json | 16 + .../xml2json/file-backing_basic-noopts.json | 8 +- ...le-backing_basic-unmap-detect-srconly.json | 24 + .../file-backing_basic-unmap-detect.json | 8 +- ...le-backing_basic-unmap-ignore-srconly.json | 24 + .../file-backing_basic-unmap-ignore.json | 8 +- .../file-backing_basic-unmap-srconly.json | 24 + .../xml2json/file-backing_basic-unmap.json | 8 +- .../xml2json/file-bochs-noopts-srconly.json | 4 + .../xml2json/file-bochs-noopts.json | 2 +- .../xml2json/file-cloop-noopts-srconly.json | 4 + .../xml2json/file-cloop-noopts.json | 2 +- .../xml2json/file-dmg-noopts-srconly.json | 4 + .../xml2json/file-dmg-noopts.json | 2 +- .../xml2json/file-ploop-noopts-srconly.json | 4 + .../xml2json/file-ploop-noopts.json | 2 +- ...cow2-backing-chain-encryption-srconly.json | 8 + .../file-qcow2-backing-chain-encryption.json | 4 +- ...le-qcow2-backing-chain-noopts-srconly.json | 40 ++ .../file-qcow2-backing-chain-noopts.json | 20 +- ...w2-backing-chain-unterminated-srconly.json | 8 + ...file-qcow2-backing-chain-unterminated.json | 4 +- .../xml2json/file-raw-aio_native-srconly.json | 4 + .../xml2json/file-raw-aio_native.json | 2 +- .../xml2json/file-raw-luks-srconly.json | 4 + .../xml2json/file-raw-luks.json | 2 +- .../xml2json/file-raw-noopts-srconly.json | 4 + .../xml2json/file-raw-noopts.json | 2 +- .../xml2json/file-vdi-noopts-srconly.json | 4 + .../xml2json/file-vdi-noopts.json | 2 +- .../xml2json/file-vhd-noopts-srconly.json | 4 + .../xml2json/file-vhd-noopts.json | 2 +- .../xml2json/file-vpc-noopts-srconly.json | 4 + .../xml2json/file-vpc-noopts.json | 2 +- .../xml2json/network-nbd-tls-srconly.json | 8 + .../xml2json/network-nbd-tls.json | 2 +- ...w2-backing-chain-cache-unsafe-srconly.json | 22 + ...work-qcow2-backing-chain-cache-unsafe.json | 4 +- ...backing-chain-encryption_auth-srconly.json | 22 + ...k-qcow2-backing-chain-encryption_auth.json | 4 +- .../qemu-ns.x86_64-4.0.0.args | 2 +- .../qemu-ns.x86_64-latest.args | 2 +- 108 files changed, 1764 insertions(+), 170 deletions(-) create mode 100644 tests/qemublocktestdata/imagecreate/luks-encopts.json create mode 100644 tests/qemublocktestdata/imagecreate/luks-encopts.xml create mode 100644 tests/qemublocktestdata/imagecreate/luks-noopts.json create mode 100644 tests/qemublocktestdata/imagecreate/luks-noopts.xml create mode 100644 tests/qemublocktestdata/imagecreate/network-gluster-qcow2.json create mode 100644 tests/qemublocktestdata/imagecreate/network-gluster-qcow2.xml create mode 100644 tests/qemublocktestdata/imagecreate/network-rbd-qcow2.json create mode 100644 tests/qemublocktestdata/imagecreate/network-rbd-qcow2.xml create mode 100644 tests/qemublocktestdata/imagecreate/network-sheepdog-qcow2.json create mode 100644 tests/qemublocktestdata/imagecreate/network-sheepdog-qcow2.xml create mode 100644 tests/qemublocktestdata/imagecreate/network-ssh-qcow2.json create mode 100644 tests/qemublocktestdata/imagecreate/network-ssh-qcow2.xml create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-backing-luks.json create mode 120000 tests/qemublocktestdata/imagecreate/qcow2-backing-luks.xml create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-backing-raw-nbd.json create mode 120000 tests/qemublocktestdata/imagecreate/qcow2-backing-raw-nbd.xml create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-backing-raw.json create mode 120000 tests/qemublocktestdata/imagecreate/qcow2-backing-raw.xml create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-luks-encopts-backing.json create mode 120000 tests/qemublocktestdata/imagecreate/qcow2-luks-encopts-backing.xml create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-luks-encopts.json create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-luks-encopts.xml create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-luks-noopts.json create mode 100644 tests/qemublocktestdata/imagecreate/qcow2-luks-noopts.xml create mode 100644 tests/qemublocktestdata/imagecreate/qcow2.json create mode 100644 tests/qemublocktestdata/imagecreate/qcow2.xml create mode 100644 tests/qemublocktestdata/imagecreate/raw-nbd.json create mode 100644 tests/qemublocktestdata/imagecreate/raw-nbd.xml create mode 100644 tests/qemublocktestdata/imagecreate/raw.json create mode 100644 tests/qemublocktestdata/imagecreate/raw.xml create mode 100644 tests/qemublocktestdata/xml2json/block-raw-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/block-raw-reservations-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/dir-fat-cache-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/dir-fat-floppy-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/dir-fat-readonly-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-aio_threads-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-cache-directsync-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-cache-none-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-cache-unsafe-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-cache-writeback-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-cache-writethrough-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-detect-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-unmap-detect-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-unmap-ignore-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-backing_basic-unmap-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-bochs-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-cloop-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-dmg-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-ploop-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-qcow2-backing-chain-encryption-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-qcow2-backing-chain-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-qcow2-backing-chain-unterminated-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-raw-aio_native-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-raw-luks-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-raw-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-vdi-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-vhd-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/file-vpc-noopts-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/network-nbd-tls-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/network-qcow2-backing-chain-cache-unsafe-srconly.json create mode 100644 tests/qemublocktestdata/xml2json/network-qcow2-backing-chain-encryption_auth-srconly.json -- 2.21.0

2 22

[libvirt] [PATCH] m4: Fix check for yajl.pc
by Andrea Bolognani 08 Jul '19

08 Jul '19

Commit 44b8df4cb4b3 introduced a check for yajl.pc that is extremely similar to the one we already had in place for readline.pc - so similar, in fact, that it's still looking for readline.pc instead of yajl.pc :) Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- m4/virt-yajl.m4 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/m4/virt-yajl.m4 b/m4/virt-yajl.m4 index 70cfb983e3..b602abd0a7 100644 --- a/m4/virt-yajl.m4 +++ b/m4/virt-yajl.m4 @@ -24,7 +24,7 @@ AC_DEFUN([LIBVIRT_ARG_YAJL],[ AC_DEFUN([LIBVIRT_CHECK_YAJL],[ dnl YAJL JSON library http://lloyd.github.com/yajl/ - PKG_CHECK_EXISTS([readline], [use_pkgconfig=1], [use_pkgconfig=0]) + PKG_CHECK_EXISTS([yajl], [use_pkgconfig=1], [use_pkgconfig=0]) if test $use_pkgconfig = 1; then dnl 2.0.3 was the version where the pkg-config file was first added -- 2.21.0

2 1

[libvirt] [PATCH 00/23] Use a slirp helper process
by marcandre.lureau＠redhat.com 08 Jul '19

08 Jul '19

From: Marc-André Lureau <marcandre.lureau(a)redhat.com> Hi, SLIRP networking can be running in a separate process. This allows for stricter security policies for QEMU & SLIRP, as SLIRP is notoriously not very safe (discussed on ML, various CVEs, and even the code says so explicitly in the comments), yet people rely on it for various reasons. With this series, for a network interface "user", libvirt will: - check the slirp-helper presence and capabilites (see [1]) - setup a socket pair between qemu and the helper - use -net socket - setup migration thanks to dbus-vmstate There are no changes required to domain configuration to benefit it. "guestfwd" isn't supported at this point, but it is known to be in a broken state with libvirt+qemu anyway. The dbus-vmstate is being proposed to QEMU. The libslirp-rs slirp-helper hasn't yet received a release, but I plan to make one soon, depending on the feedback. fwiw, I have been working on an alternative rust-only implementation of a slirp-helper that will also follow [1]. [1] https://gitlab.freedesktop.org/slirp/libslirp-rs/blob/master/src/bin/README… Marc-André Lureau (23): Add .editorconfig dbus: correctly build reply message qemu: replace logCtxt with qemuDomainLogAppendMessage() qemu: reset VM id after external devices stop qemu: add socket datagram capability qemu: add dbus-vmstate capability qemu-conf: add configurable dbus-daemon location qemu-conf: add dbusStateDir qemu-security: add qemuSecurityCommandRun() qemu: add a DBus daemon helper unit qemu: add dbus-vmstate helper migration support domain-conf: add network def private data qemu: add qemuDomainNetworkPrivate qemu-conf: add configurable slirp-helper location qemu: add slirp helper unit qemu-domain: save and restore slirp state qemu: add a flag to the cookie to prevent slirp-helper setup qemu-migration: prevent migration if slirp cannot be migrated qemu-extdevice: prepare, start and stop slirp-helper qemu-command: use -net socket,fd= with slirp-helper qemu-process: prepare slirp-helper qemu-hotplug: handle hotplugging of slirp-helper tests: add slirp-helper qemuxml2argv test .editorconfig | 21 + m4/virt-driver-qemu.m4 | 10 + src/conf/domain_conf.c | 21 +- src/conf/domain_conf.h | 6 + src/qemu/Makefile.inc.am | 6 + src/qemu/libvirtd_qemu.aug | 2 + src/qemu/qemu.conf | 6 + src/qemu/qemu_alias.c | 7 + src/qemu/qemu_alias.h | 2 + src/qemu/qemu_capabilities.c | 10 + src/qemu/qemu_capabilities.h | 4 + src/qemu/qemu_command.c | 100 ++++- src/qemu/qemu_command.h | 6 +- src/qemu/qemu_conf.c | 20 +- src/qemu/qemu_conf.h | 3 + src/qemu/qemu_dbus.c | 384 ++++++++++++++++ src/qemu/qemu_dbus.h | 44 ++ src/qemu/qemu_domain.c | 206 ++++++++- src/qemu/qemu_domain.h | 27 ++ src/qemu/qemu_driver.c | 11 +- src/qemu/qemu_extdevice.c | 82 ++-- src/qemu/qemu_extdevice.h | 10 +- src/qemu/qemu_hotplug.c | 118 ++++- src/qemu/qemu_hotplug.h | 8 + src/qemu/qemu_interface.c | 27 ++ src/qemu/qemu_interface.h | 4 + src/qemu/qemu_migration.c | 62 +++ src/qemu/qemu_monitor.c | 35 +- src/qemu/qemu_monitor.h | 6 +- src/qemu/qemu_monitor_json.c | 15 + src/qemu/qemu_monitor_json.h | 5 + src/qemu/qemu_process.c | 37 +- src/qemu/qemu_process.h | 1 + src/qemu/qemu_security.c | 22 + src/qemu/qemu_security.h | 6 + src/qemu/qemu_slirp.c | 412 ++++++++++++++++++ src/qemu/qemu_slirp.h | 78 ++++ src/qemu/qemu_tpm.c | 10 +- src/qemu/qemu_tpm.h | 5 +- src/qemu/test_libvirtd_qemu.aug.in | 2 + src/util/virdbus.c | 18 +- src/util/virdbus.h | 6 +- tests/Makefile.am | 1 + .../caps_4.0.0.aarch64.xml | 1 + .../qemucapabilitiesdata/caps_4.0.0.ppc64.xml | 1 + .../caps_4.0.0.riscv32.xml | 1 + .../caps_4.0.0.riscv64.xml | 1 + .../qemucapabilitiesdata/caps_4.0.0.s390x.xml | 1 + .../caps_4.0.0.x86_64.xml | 1 + .../caps_4.1.0.x86_64.xml | 1 + .../net-user.x86_64-4.0.0.args | 34 ++ tests/qemuxml2argvtest.c | 16 + tests/testutilsqemu.h | 1 + tests/virfirewalltest.c | 9 +- tests/virpolkittest.c | 3 +- 55 files changed, 1849 insertions(+), 87 deletions(-) create mode 100644 .editorconfig create mode 100644 src/qemu/qemu_dbus.c create mode 100644 src/qemu/qemu_dbus.h create mode 100644 src/qemu/qemu_slirp.c create mode 100644 src/qemu/qemu_slirp.h create mode 100644 tests/qemuxml2argvdata/net-user.x86_64-4.0.0.args -- 2.22.0.214.g8dca754b1e

1 23

[libvirt] [PATCH] libvirt: remove link to outdated mailing list post
by Daniel P. Berrangé 08 Jul '19

08 Jul '19

The virConnectGetType() method has an unfortunate signature, returning a static string that must not be freed by the caller. The remote driver, however, gets this string dynamically over an RPC call, which raised a design discussion on the mailing list. Eventually the problem was resolved by having the remote driver cache the returned string internally and free it when the connection was closed. The link to the mailing list is thus talking about a problem that does not actually exist in the final implementation, and at best serves to confuse the reader into thinking there might be a memory leak. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- src/libvirt-host.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/src/libvirt-host.c b/src/libvirt-host.c index 2978825d22..e5c4e5f72a 100644 --- a/src/libvirt-host.c +++ b/src/libvirt-host.c @@ -98,9 +98,6 @@ virConnectSupportsFeature(virConnectPtr conn, int feature) * hypervisor, use virConnectGetCapabilities(). * * Returns NULL in case of error, a static zero terminated string otherwise. - * - * See also: - * http://www.redhat.com/archives/libvir-list/2007-February/msg00096.html */ const char * virConnectGetType(virConnectPtr conn) -- 2.21.0

2 1

[libvirt] [PATCH v8 00/21] Incremental backups
by Eric Blake 06 Jul '19

06 Jul '19

The API is fairly stable, but there are still probably code cleanups worth adding to the qemu driver implementation. I've also seen a desire on the list to support all APIs in the test driver, so I'll try and whip that together for my next round of patches. If we are confident that the API is worth having in 5.3 (especially since the earlier part of this series already has positive reviews), then it's probably worth committing that part of the series now even while I still address review comments on the later patches. I've pushed a tag backup-v8 to both my libvirt.git and libvirt-python.git repos to match: https://repo.or.cz/libvirt/ericb.git/shortlog/refs/tags/backup-v8 https://repo.or.cz/libvirt-python/ericb.git/shortlog/refs/tags/backup-v8 Here's hoping we're happy enough with the API, including the fact that rudimentary operation of pull mode backups work with qemu 4.0-rc1, for this to make it into the 5.2 release. Notable diffs from v7: - address several review comments (although I didn't get to every comment later in the series) - actually support job id of 0 (it was documented but not working in v7) - add support for Unix socket pull mode backups - improve checkpoint XML handling (more tests, separate postparse actions out from initial parse) - rebase to master 001/21:[0044] [FC] 'backup: Document new XML for checkpoints' 002/21:[----] [--] 'backup: Document new XML for backups' 003/21:[0006] [FC] 'backup: Introduce virDomainCheckpoint APIs' 004/21:[0002] [FC] 'backup: Introduce virDomainBackup APIs' 005/21:[----] [--] 'backup: Document nuances between different state capture APIs' 006/21:[0304] [FC] 'backup: Parse and output checkpoint XML' 007/21:[0007] [FC] 'backup: Allow for lists of checkpoint objects' 008/21:[0004] [FC] 'backup: Add new domain:checkpoint access control' 009/21:[0043] [FC] 'backup: Implement backup APIs for remote driver' 010/21:[down] 'conf: Add parameter to virDomainDiskSourceFormat' 011/21:[0019] [FC] 'backup: Parse and output backup XML' 012/21:[0006] [FC] 'backup: Implement virsh support for checkpoints' 013/21:[0002] [FC] 'backup: Implement virsh support for backup' 014/21:[down] 'backup: Prepare for Unix sockets in QMP nbd-server-start' 015/21:[0008] [FC] 'backup: Add new qemu monitor interactions' 016/21:[0022] [FC] 'backup: qemu: Implement metadata tracking for checkpoint APIs' 017/21:[0004] [FC] 'backup: Wire up qemu checkpoint commands over QMP' 018/21:[0019] [FC] 'backup: qemu: Implement framework for backup job APIs' 019/21:[0042] [FC] 'backup: Wire up qemu full pull backup commands over QMP' 020/21:[----] [-C] 'backup: qemu: Wire up qemu full push backup commands over QMP' 021/21:[0003] [FC] 'backup: implement qemu incremental pull backup' Eric Blake (21): backup: Document new XML for checkpoints backup: Document new XML for backups backup: Introduce virDomainCheckpoint APIs backup: Introduce virDomainBackup APIs backup: Document nuances between different state capture APIs backup: Parse and output checkpoint XML backup: Allow for lists of checkpoint objects backup: Add new domain:checkpoint access control backup: Implement backup APIs for remote driver conf: Add parameter to virDomainDiskSourceFormat backup: Parse and output backup XML backup: Implement virsh support for checkpoints backup: Implement virsh support for backup backup: Prepare for Unix sockets in QMP nbd-server-start backup: Add new qemu monitor interactions backup: qemu: Implement metadata tracking for checkpoint APIs backup: Wire up qemu checkpoint commands over QMP backup: qemu: Implement framework for backup job APIs backup: Wire up qemu full pull backup commands over QMP backup: qemu: Wire up qemu full push backup commands over QMP backup: implement qemu incremental pull backup include/libvirt/libvirt-domain-checkpoint.h | 161 ++ include/libvirt/libvirt-domain-snapshot.h | 8 +- include/libvirt/libvirt-domain.h | 47 +- include/libvirt/libvirt.h | 5 +- src/access/viraccessperm.h | 6 + src/conf/backup_conf.h | 97 + src/conf/checkpoint_conf.h | 103 ++ src/conf/domain_conf.h | 3 + src/conf/virconftypes.h | 12 + src/conf/virdomaincheckpointobjlist.h | 74 + src/conf/virdomainmomentobjlist.h | 7 +- src/conf/virdomainobjlist.h | 7 +- src/driver-hypervisor.h | 79 + src/qemu/qemu_block.h | 3 + src/qemu/qemu_blockjob.h | 1 + src/qemu/qemu_capabilities.h | 2 + src/qemu/qemu_conf.h | 2 + src/qemu/qemu_domain.h | 19 + src/qemu/qemu_monitor.h | 27 +- src/qemu/qemu_monitor_json.h | 22 +- tools/virsh-checkpoint.h | 29 + tools/virsh-completer.h | 4 + tools/virsh-util.h | 3 + tools/virsh.h | 1 + docs/Makefile.am | 3 + docs/apibuild.py | 2 + docs/docs.html.in | 10 +- docs/domainstatecapture.html.in | 315 ++++ docs/format.html.in | 2 + docs/formatbackup.html.in | 184 ++ docs/formatcheckpoint.html.in | 204 +++ docs/formatsnapshot.html.in | 2 + docs/index.html.in | 4 +- docs/schemas/domainbackup.rng | 219 +++ docs/schemas/domaincheckpoint.rng | 87 + examples/object-events/event-test.c | 3 + libvirt.spec.in | 3 + mingw-libvirt.spec.in | 6 + po/POTFILES | 3 + src/Makefile.am | 2 + src/access/viraccessperm.c | 3 +- src/conf/Makefile.inc.am | 6 + src/conf/backup_conf.c | 538 ++++++ src/conf/checkpoint_conf.c | 636 +++++++ src/conf/domain_conf.c | 22 +- src/conf/snapshot_conf.c | 3 +- src/conf/virdomaincheckpointobjlist.c | 223 +++ src/conf/virdomainmomentobjlist.c | 40 +- src/conf/virdomainobjlist.c | 11 + src/conf/virdomainsnapshotobjlist.c | 2 +- src/libvirt-domain-checkpoint.c | 750 ++++++++ src/libvirt-domain-snapshot.c | 89 + src/libvirt-domain.c | 237 ++- src/libvirt_private.syms | 32 + src/libvirt_public.syms | 24 + src/qemu/qemu_block.c | 12 + src/qemu/qemu_capabilities.c | 4 + src/qemu/qemu_conf.c | 5 + src/qemu/qemu_domain.c | 221 ++- src/qemu/qemu_driver.c | 1577 +++++++++++++++++ src/qemu/qemu_migration.c | 9 +- src/qemu/qemu_monitor.c | 72 +- src/qemu/qemu_monitor_json.c | 222 ++- src/qemu/qemu_process.c | 9 + src/remote/remote_daemon_dispatch.c | 20 + src/remote/remote_driver.c | 32 +- src/remote/remote_protocol.x | 258 ++- src/remote_protocol-structs | 139 ++ src/rpc/gendispatch.pl | 32 +- tests/Makefile.am | 13 +- tests/domainbackupxml2xmlin/backup-pull.xml | 9 + tests/domainbackupxml2xmlin/backup-push.xml | 9 + tests/domainbackupxml2xmlin/empty.xml | 1 + tests/domainbackupxml2xmlout/backup-pull.xml | 9 + tests/domainbackupxml2xmlout/backup-push.xml | 9 + tests/domainbackupxml2xmlout/empty.xml | 7 + tests/domaincheckpointxml2xmlin/empty.xml | 1 + tests/domaincheckpointxml2xmlin/sample.xml | 7 + tests/domaincheckpointxml2xmlin/size.xml | 4 + tests/domaincheckpointxml2xmlout/empty.xml | 7 + .../internal-active-invalid.xml | 53 + .../internal-inactive-invalid.xml | 53 + tests/domaincheckpointxml2xmlout/redefine.xml | 63 + tests/domaincheckpointxml2xmlout/sample.xml | 12 + tests/domaincheckpointxml2xmlout/size.xml | 11 + tests/domaincheckpointxml2xmltest.c | 223 +++ tests/qemublocktest.c | 3 +- .../caps_4.0.0.riscv32.xml | 2 + .../caps_4.0.0.riscv64.xml | 2 + .../caps_4.0.0.x86_64.xml | 2 + tests/qemumonitorjsontest.c | 32 +- tests/virschematest.c | 4 + tests/virstoragetest.c | 2 +- tools/Makefile.am | 1 + tools/virsh-checkpoint.c | 1370 ++++++++++++++ tools/virsh-completer.c | 51 + tools/virsh-domain-monitor.c | 23 + tools/virsh-domain.c | 268 ++- tools/virsh-snapshot.c | 37 +- tools/virsh-util.c | 11 + tools/virsh.c | 2 + tools/virsh.pod | 302 +++- 102 files changed, 9502 insertions(+), 100 deletions(-) create mode 100644 include/libvirt/libvirt-domain-checkpoint.h create mode 100644 src/conf/backup_conf.h create mode 100644 src/conf/checkpoint_conf.h create mode 100644 src/conf/virdomaincheckpointobjlist.h create mode 100644 tools/virsh-checkpoint.h create mode 100644 docs/domainstatecapture.html.in create mode 100644 docs/formatbackup.html.in create mode 100644 docs/formatcheckpoint.html.in create mode 100644 docs/schemas/domainbackup.rng create mode 100644 docs/schemas/domaincheckpoint.rng create mode 100644 src/conf/backup_conf.c create mode 100644 src/conf/checkpoint_conf.c create mode 100644 src/conf/virdomaincheckpointobjlist.c create mode 100644 src/libvirt-domain-checkpoint.c create mode 100644 tests/domainbackupxml2xmlin/backup-pull.xml create mode 100644 tests/domainbackupxml2xmlin/backup-push.xml create mode 100644 tests/domainbackupxml2xmlin/empty.xml create mode 100644 tests/domainbackupxml2xmlout/backup-pull.xml create mode 100644 tests/domainbackupxml2xmlout/backup-push.xml create mode 100644 tests/domainbackupxml2xmlout/empty.xml create mode 100644 tests/domaincheckpointxml2xmlin/empty.xml create mode 100644 tests/domaincheckpointxml2xmlin/sample.xml create mode 100644 tests/domaincheckpointxml2xmlin/size.xml create mode 100644 tests/domaincheckpointxml2xmlout/empty.xml create mode 100644 tests/domaincheckpointxml2xmlout/internal-active-invalid.xml create mode 100644 tests/domaincheckpointxml2xmlout/internal-inactive-invalid.xml create mode 100644 tests/domaincheckpointxml2xmlout/redefine.xml create mode 100644 tests/domaincheckpointxml2xmlout/sample.xml create mode 100644 tests/domaincheckpointxml2xmlout/size.xml create mode 100644 tests/domaincheckpointxml2xmltest.c create mode 100644 tools/virsh-checkpoint.c -- 2.20.1

2 46

[libvirt] [PATCH] docs: minor updates to to new API doc
by Jonathon Jongsma 05 Jul '19

05 Jul '19

Fix a filename and add a couple missing words. Signed-off-by: Jonathon Jongsma <jjongsma(a)redhat.com> --- docs/api_extension.html.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/api_extension.html.in b/docs/api_extension.html.in index d7fbbd6e90..3bbaf4893c 100644 --- a/docs/api_extension.html.in +++ b/docs/api_extension.html.in @@ -229,7 +229,7 @@ <h3><a name='rpcclient'>Implement the RPC client</a></h3> <p> - Implementing the uses the rpcgen generated .h files. The remote + Implementing the RPC client uses the rpcgen generated .h files. The remote method calls go in: </p> @@ -256,7 +256,7 @@ The server side dispatchers are implemented in: </p> - <p><code>src/remote/daemon_dispatch.c</code></p> + <p><code>src/remote/remote_daemon_dispatch.c</code></p> <p>Again, this step uses the .h files generated by make rpcgen.</p> -- 2.20.1

2 1

[libvirt] [PATCH] tests: fix mocking of virFileGetXAttrQuiet on FreeBSD
by Daniel P. Berrangé 05 Jul '19

05 Jul '19

The qemusecuritytest is failing on FreeBSD 11/12, reporting that files are not correctly restored. Debugging code printfs show that the virFileGetXAttrQuiet mock is returning 0, but the virFileGetXAttr function is seeing -1 as the return value. Essentially there appears to be some kind of optimization between the real virFileGetXAttrQuiet and the real virFileGetXAttr, which breaks when we mock virFileGetXAttrQuiet. Rather than trying to figure out how to avoid this, it is simpler to just mock virFileGetXAttr too since it is very short code. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- Pushed as a CI build fix tests/qemusecuritymock.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/tests/qemusecuritymock.c b/tests/qemusecuritymock.c index 2a9095e1bf..a15eef29c9 100644 --- a/tests/qemusecuritymock.c +++ b/tests/qemusecuritymock.c @@ -156,6 +156,34 @@ virFileGetXAttrQuiet(const char *path, } +/* + * This may look redundant but is needed to work around an + * compiler quirk. The call from the real virFileGetXAttr + * to the real virFileGetXAttrQuiet has a quirk where the + * return value from virFileGetXAttrQuiet gets scrambled + * if we mock virFileGetXAttrQuiet, returning -1 instead + * of 0 despite succeeding. This happens on FreeBSD 11/12 + * hosts with CLang, and is suspected to be some kind of + * compiler optimization. By mocking this function too we + * can workaround it. + */ +int +virFileGetXAttr(const char *path, + const char *name, + char **value) +{ + int ret; + + if ((ret = virFileGetXAttrQuiet(path, name, value)) < 0) { + virReportSystemError(errno, + "Unable to get XATTR %s on %s", + name, path); + } + + return ret; +} + + int virFileSetXAttr(const char *path, const char *name, const char *value) -- 2.21.0

1 0

Re: [libvirt] [libvirt-users] UDP broadcasts vs. nat Masquerading issue
by Nikolai Zhubr 05 Jul '19

05 Jul '19

Hi Daniel and Laine, [...] >> -A POSTROUTING -o br0 -j MASQUERADE >> -A POSTROUTING -o enp0s25 -j MASQUERADE >> -A POSTROUTING -o virbr2_nic -j MASQUERADE >> -A POSTROUTING -o vnet0 -j MASQUERADE > > *None* of those rules were added by libvirt (unless your build of [...] > You can verify my "counter-claim" by running "virsh net-destroy" for all > of your libvirt networks, and seeing that the offending rules haven't > been removed. > > In short, you need to look elsewhere for the culprit. Yes, found it. You were both right, essentially. The offending rules were added by a firewall in response to new interfaces created by libvirt dynamically, due to some dubious relict settings left in the firewall. (Silly me.) So this it not an issue of libvirt indeed! Tons of thanks for the quick and precise hit! Regards, Nikolai >> >> Here, virbr2_nic and vnet0 are used by libvirt for arranging network >> configurations for VMs, ok. However, br0 is a main interface of this >> host with primary ip address, with enp0s25 being a physical nic of >> this host, and it is used for all sorts of regular (unrelated to >> virtualization) communications. Also, br0 is used for attaching >> bridged (as opposed to NATed) VMs managed by libvirt. >> >> Clearly, libvirt somehow chooses to set up masquerading for literally >> all existing network interfaces here (except lo), > > It's clear that the rules are there. It's not clear that they were added > by libvirt. > >> but I can't see a real reason for the first two rules in the list >> above. Furthermore, they corrupt UDP broadcats coming from outside and >> reaching this host (through enp0s25/br0) such that source address gets >> replaced by this hosts primary address (as per masquerading). I've >> verified this by arranging a hand-crafted UDP listener and printing >> the respective source addresses as seen by normal userspace. >> >> Now I've discovered that I can "eliminate" the problem by either: >> >> 1. Removing "-A POSTROUTING -o br0 -j MASQUERADE" (manually) >> 2. Inserting "-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.255/32 -j >> ACCEPT" >> (Of course correcting rules by hand is not a solution, just a test) >> >> So question is, how the correct rules should ideally look like? And, >> is this issue known/fixed in most current libvirt? > > Except for putting the libvirt-added rules in their own private chains > (appearing in libvirt 5.1.0, released on Feb 1, 2019), the iptables > rules added by libvirt to support its virtual networks didn't materially > change in > 10 years. Your email is the first time I've ever seen such > rules attributed to libvirt so, as I said above, I think you need to > take a deeper dive into your host system's config. > > > Good luck! >

2 1

[libvirt] [PATCH 0/3] remote: misc cleanups to client driver
by Daniel P. Berrangé 05 Jul '19

05 Jul '19

A few bits of cleanup identified during work on the daemon splitting patches sent now since they are independent. Daniel P. Berrangé (3): remote: stop declaring variables in the middle of a function remote: use autofree for many string variables remote: conditionally declare tty variable for non-Win32 platforms src/remote/remote_driver.c | 61 ++++++++++++++------------------------ 1 file changed, 23 insertions(+), 38 deletions(-) -- 2.21.0

2 4

[libvirt] [PATCH rust 0/5] ci related plus some small updates
by Sahid Orentino Ferdjaoui 05 Jul '19

05 Jul '19

Hello Daniel, When you have a moment, can you please merge this serie. It fixes CI issue, switch to bionic, and updates the tested versions. Results: https://travis-ci.org/sahid/libvirt-rust/builds/554170698 Thanks, s. Sahid Orentino Ferdjaoui (5): fix bug integration test with rust multithreading make lookup_by_id() test more robust update tested versions from 2.5.0 to 5.5.0 switch to the last ubuntu lts bionic fix code formating in README .travis.yml | 10 +++++----- README.md | 6 +++++- tests/domain.rs | 19 +++++++++---------- tests/integration_qemu.rs | 15 +++++++++++---- tests/libvirtd.sasl | 2 ++ 5 files changed, 32 insertions(+), 20 deletions(-) create mode 100644 tests/libvirtd.sasl -- 2.20.1

2 14