August 2024 - Devel - libvirt List Archives

[RFC PATCH v4 0/5] Added virtio-net RSS with eBPF support.
by Andrew Melnychenko 14 Oct '25

14 Oct '25

This series of rfc patches adds support for loading the RSS eBPF program and passing it to the QEMU. Comments and suggestions would be useful. QEMU with vhost may work with RSS through eBPF. To load eBPF, the capabilities required that Libvirt may provide. eBPF program and maps may be unique for particular QEMU and Libvirt retrieves eBPF through qapi. For now, there is only "RSS" eBPF object in QEMU, in the future, there may be another one(g.e. network filters). That's why in Libvirt added logic to load and store any eBPF object that QEMU provides using qapi schema. One of the reasons why this series of patches is in RFC are tests. To this series of patches, the tests were added. For now, the tests are synthetic, the proper "reply" file should be generated with a new "caps" file. Currently, there are changes in caps-9.0.0* and caps-9.1.0 files. There was added support for ebpf_rss_fds feature, and request-ebpf command. So, overall, the tests are required for review, comment, and discussion how we want them to be implemented in the future. For virtio-net RSS, the document has not changed. ``` <interface type="network"> <model type="virtio"/> <driver queues="4" rss="on" rss_hash_report="off"/> <interface type="network"> ``` Simplified routine for RSS: * Libvirt retrieves eBPF "RSS" and load it. * Libvirt passes file descriptors to virtio-net with property "ebpf_rss_fds" ("rss" property should be "on" too). * if fds was provided - QEMU using eBPF RSS implementation. * if fds was not provided - QEMU tries to load eBPF RSS in own context and use it. * if eBPF RSS was not loaded - QEMU uses "in-qemu" RSS(vhost not supported). Changes since RFC v3: * changed tests a bit * refactored and rebased * removed "allowEBPF" from qemu config(now env is used for tests) Changes since RFC v2: * refactored and rebased. * applied changes according to the Qemu. * added basic test. Changes since RFC v1: * changed eBPF format saved in the XML cache. * refactored and checked with syntax test. * refactored patch hunks. Andrew Melnychenko (5): qemu_monitor: Added QEMU's "request-ebpf" support. qemu_capabilities: Added logic for retrieving eBPF objects from QEMU. qemu_interface: Added routine for loading the eBPF objects. qemu_command: Added "ebpf_rss_fds" support for virtio-net. tests: Added tests for eBPF blob loading. libvirt.spec.in | 3 + meson.build | 7 + meson_options.txt | 1 + src/qemu/meson.build | 1 + src/qemu/qemu_capabilities.c | 132 ++++++++++++ src/qemu/qemu_capabilities.h | 5 + src/qemu/qemu_command.c | 60 ++++++ src/qemu/qemu_domain.c | 4 + src/qemu/qemu_domain.h | 3 + src/qemu/qemu_interface.c | 87 ++++++++ src/qemu/qemu_interface.h | 7 + src/qemu/qemu_monitor.c | 9 + src/qemu/qemu_monitor.h | 4 + src/qemu/qemu_monitor_json.c | 27 +++ src/qemu/qemu_monitor_json.h | 4 + .../caps_9.0.0_sparc.replies | 95 +++++---- .../qemucapabilitiesdata/caps_9.0.0_sparc.xml | 3 + .../caps_9.0.0_x86_64.replies | 199 ++++++++++-------- .../caps_9.0.0_x86_64.xml | 4 + .../caps_9.1.0_x86_64.replies | 199 ++++++++++-------- .../caps_9.1.0_x86_64.xml | 4 + tests/qemuxml2argvmock.c | 24 +++ .../net-virtio-rss-bpf.x86_64-latest.args | 37 ++++ .../net-virtio-rss-bpf.x86_64-latest.xml | 46 ++++ tests/qemuxmlconfdata/net-virtio-rss-bpf.xml | 46 ++++ tests/qemuxmlconftest.c | 5 + 26 files changed, 792 insertions(+), 224 deletions(-) create mode 100644 tests/qemuxmlconfdata/net-virtio-rss-bpf.x86_64-latest.args create mode 100644 tests/qemuxmlconfdata/net-virtio-rss-bpf.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/net-virtio-rss-bpf.xml -- 2.45.2

3 7

[PATCH 0/3] hw/sd/sdcard: Deprecate support for spec v1.10
by Philippe Mathieu-Daudé 02 Sep '25

02 Sep '25

Deprecate SD spec v1.10, use v3.01 as new default. Philippe Mathieu-Daudé (3): hw/sd/sdcard: Deprecate support for spec v1.10 hw/sd/sdcard: Use spec v3.01 by default hw/sd/sdcard: Remove support for spec v1.10 docs/about/removed-features.rst | 5 +++++ include/hw/sd/sd.h | 1 - hw/sd/sd.c | 14 +++----------- 3 files changed, 8 insertions(+), 12 deletions(-) -- 2.41.0

2 10

[PATCH rfcv4 00/13] LIBVIRT: X86: TDX support
by Zhenzhong Duan 02 Apr '25

02 Apr '25

Hi, This series brings libvirt the x86 TDX support. * What's TDX? TDX stands for Trust Domain Extensions which isolates VMs from the virtual-machine manager (VMM)/hypervisor and any other software on the platform. To support TDX, multiple software components, not only KVM but also QEMU, guest Linux and virtual bios, need to be updated. For more details, please check link[1]. This patchset is another software component to extend libvirt to support TDX, with which one can start a TDX guest from high level rather than running qemu directly. * Misc As QEMU use a software emulated way to reset guest which isn't supported by TDX guest for security reason. We simulate reboot for TDX guest by kill and create a new one in FakeReboot framework. Complete code can be found at [2], matching qemu code can be found at [3]. There is a 'debug' property for tdx-guest object which isn't in matching qemu[3] yet. I keep them intentionally as they will be implemented in qemu as extention series of [3]. * Test start/stop/reboot with virsh stop/reboot trigger in guest stop with on_poweroff=destroy/restart reboot with on_reboot=destroy/restart * Patch organization - patch 1-4: Support query of TDX capabilities. - patch 5-8: Add TDX type to launchsecurity framework. - patch 9-11: Add reboot support to TDX guest - patch 12-13: Add test and docs TODO: - update QEMU capabilities data in tests, depending on qemu TDX merged beforehand - add reconnect logic in virsh command [1] https://lore.kernel.org/kvm/cover.1708933498.git.isaku.yamahata@intel.com [2] https://github.com/intel/libvirt-tdx/commits/tdx_for_upstream_rfcv4 [3] https://github.com/intel/qemu-tdx/tree/tdx-qemu-upstream-v5 Thanks Zhenzhong Changelog: rfcv4: - add a check to tools/virt-host-validate-qemu.c (Daniel) - remove check of q35 (Daniel) - model 'SocktetAddress' QAPI in xml schema (Daniel) - s/Quote-Generation-Service/quoteGenerationService/ (Daniel) - define bits in tdx->policy and add validating logic (Daniel) - presume QEMU choose split kernel irqchip for TDX guest by default (Daniel) - utilize existing FakeReboot framework to do reboot for TDX guest (Daniel) - drop patch11 'conf: Add support to keep same domid for hard reboot' (Daniel) - add test in tests/ to validate parsing and formatting logic (Daniel) - add doc in docs/formatdomain.rst (Daniel) - add R-B rfcv3: - Change to generate qemu cmdline with -bios - drop firmware auto match as -bios is used - add a hard reboot method to reboot TDX guest rfcv3: https://www.mail-archive.com/devel@lists.libvirt.org/msg00385.html rfcv2: - give up using qmp cmd and check TDX directly on host for TDX capabilities. - use launchsecurity framework to support TDX - use <os>.<loader> for general loader - add auto firmware match feature for TDX A example TDVF fimware description file 70-edk2-x86_64-tdx.json: { "description": "UEFI firmware for x86_64, supporting Intel TDX", "interface-types": [ "uefi" ], "mapping": { "device": "generic", "filename": "/usr/share/OVMF/OVMF_CODE-tdx.fd" }, "targets": [ { "architecture": "x86_64", "machines": [ "pc-q35-*" ] } ], "features": [ "intel-tdx", "verbose-dynamic" ], "tags": [ ] } rfcv2: https://www.mail-archive.com/libvir-list@redhat.com/msg219378.html Zhenzhong Duan (13): tools: Secure guest check for Intel in virt-host-validate qemu: Check if INTEL Trust Domain Extention support is enabled qemu: Add TDX capability conf: expose TDX feature in domain capabilities conf: add tdx as launch security type qemu: Add command line and validation for TDX type qemu: force special parameters enabled for TDX guest Add Intel TDX Quote Generation Service(QGS) support qemu: add FakeReboot support for TDX guest qemu: Support reboot command in guest qemu: Avoid duplicate FakeReboot for secure guest Add test cases for Intel TDX docs: domain: Add documentation for Intel TDX guest docs/formatdomain.rst | 68 ++++ docs/formatdomaincaps.rst | 1 + src/conf/domain_capabilities.c | 1 + src/conf/domain_capabilities.h | 1 + src/conf/domain_conf.c | 312 ++++++++++++++++++ src/conf/domain_conf.h | 75 +++++ src/conf/schemas/domaincaps.rng | 9 + src/conf/schemas/domaincommon.rng | 135 ++++++++ src/conf/virconftypes.h | 2 + src/qemu/qemu_capabilities.c | 36 +- src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 139 ++++++++ src/qemu/qemu_firmware.c | 1 + src/qemu/qemu_monitor.c | 28 +- src/qemu/qemu_monitor.h | 2 +- src/qemu/qemu_monitor_json.c | 6 +- src/qemu/qemu_namespace.c | 1 + src/qemu/qemu_process.c | 75 +++++ src/qemu/qemu_validate.c | 44 +++ ...unch-security-tdx-qgs-fd.x86_64-latest.xml | 77 +++++ .../launch-security-tdx-qgs-fd.xml | 30 ++ ...ch-security-tdx-qgs-inet.x86_64-latest.xml | 77 +++++ .../launch-security-tdx-qgs-inet.xml | 30 ++ ...ch-security-tdx-qgs-unix.x86_64-latest.xml | 77 +++++ .../launch-security-tdx-qgs-unix.xml | 30 ++ ...h-security-tdx-qgs-vsock.x86_64-latest.xml | 77 +++++ .../launch-security-tdx-qgs-vsock.xml | 30 ++ tests/qemuxmlconftest.c | 24 ++ tools/virt-host-validate-common.c | 22 +- tools/virt-host-validate-common.h | 1 + 30 files changed, 1407 insertions(+), 5 deletions(-) create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-fd.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-fd.xml create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-inet.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-inet.xml create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-unix.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-unix.xml create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-vsock.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/launch-security-tdx-qgs-vsock.xml -- 2.34.1

6 45

[PATCH 0/4] hw/s390x: Alias @dump-skeys -> @dump-s390-skey and deprecate
by Philippe Mathieu-Daudé 10 Mar '25

10 Mar '25

We are trying to unify all qemu-system-FOO to a single binary. In order to do that we need to remove QAPI target specific code. @dump-skeys is only available on qemu-system-s390x. This series rename it as @dump-s390-skey, making it available on other binaries. We take care of backward compatibility via deprecation. Philippe Mathieu-Daudé (4): hw/s390x: Introduce the @dump-s390-skeys QMP command hw/s390x: Introduce the 'dump_s390_skeys' HMP command hw/s390x: Deprecate the HMP 'dump_skeys' command hw/s390x: Deprecate the QMP @dump-skeys command docs/about/deprecated.rst | 5 +++++ qapi/misc-target.json | 5 +++++ qapi/misc.json | 18 ++++++++++++++++++ include/monitor/hmp.h | 1 + hw/s390x/s390-skeys-stub.c | 24 ++++++++++++++++++++++++ hw/s390x/s390-skeys.c | 19 +++++++++++++++++-- hmp-commands.hx | 17 +++++++++++++++-- hw/s390x/meson.build | 5 +++++ 8 files changed, 90 insertions(+), 4 deletions(-) create mode 100644 hw/s390x/s390-skeys-stub.c -- 2.41.0

7 24

[libvirt] [PATCH] Fix python error reporting for some storage operations
by Cole Robinson 07 Mar '25

07 Mar '25

In the python bindings, all vir* classes expect to be passed a virConnect object when instantiated. Before the storage stuff, these classes were only instantiated in virConnect methods, so the generator is hardcoded to pass 'self' as the connection instance to these classes. Problem is there are some methods that return pool or vol instances which aren't called from virConnect: you can lookup a storage volume's associated pool, and can lookup volumes from a pool. In these cases passing 'self' doesn't give the vir* instance a connection, so when it comes time to raise an exception crap hits the fan. Rather than rework the generator to accomodate this edge case, I just fixed the init functions for virStorage* to pull the associated connection out of the passed value if it's not a virConnect instance. Thanks, Cole diff --git a/python/generator.py b/python/generator.py index 01a17da..c706b19 100755 --- a/python/generator.py +++ b/python/generator.py @@ -962,8 +962,12 @@ def buildWrappers(): list = reference_keepers[classname] for ref in list: classes.write(" self.%s = None\n" % ref[1]) - if classname in [ "virDomain", "virNetwork", "virStoragePool", "virStorageVol" ]: + if classname in [ "virDomain", "virNetwork" ]: classes.write(" self._conn = conn\n") + elif classname in [ "virStorageVol", "virStoragePool" ]: + classes.write(" self._conn = conn\n" + \ + " if not isinstance(conn, virConnect):\n" + \ + " self._conn = conn._conn\n") classes.write(" if _obj != None:self._o = _obj;return\n") classes.write(" self._o = None\n\n"); destruct=None

4 3

[RFC 0/4] meson: Enable -Wundef
by Andrea Bolognani 27 Feb '25

27 Feb '25

A few days ago I have posted a patch[1] that addresses an issue introduced when a meson check was dropped but some uses of the corresponding WITH_ macro were not removed at the same time. That got me thinking about what we can do to prevent such scenarios from happening again in the future. I have come up with something that I think would be effective, but since applying the approach throughout the entire codebase would require a non-trivial amount of work, I figured I'd ask for feedback before embarking on it. The idea is that there are two types of macros we can use for conditional compilation: external ones, coming from the OS or other libraries, and internal ones, which are the result of meson tests. The external ones (e.g. SIOCSIFFLAGS, __APPLE__) are usually only defined if they apply, so it is correct to check for their presence with #ifdef. Using #if will also work, as undefined macros evaluate to zero, but it's not good practice to use them that way. If -Wundef has been passed to the compiler, those incorrect uses will be reported (only on platforms where they are not defined, of course). The internal ones (e.g. WITH_QEMU, WITH_STRUCT_IFREQ) are similar, but in this case we control their definition. This means that using means that the feature is not available on the machine we're building on, but it could also mean that we've removed the meson check and forgot to update all users of the macro. In this case, -Wundef would work 100% reliably to detect the issue: if the meson check doesn't exist, neither will the macro, regardless of what platform we're building on. So the approach I'm suggesting is to use a syntax-check rule to ensure that internal macros are only ever checked with #if instead of Of course this requires a full sweep to fix all cases in which we're not already doing things according to the proposal. Should be fairly easy, if annoying. A couple of examples are included here for demonstration purposes. The bigger impact is going to be on the build system. Right now we generally only define WITH_ macros if the check passed, but that will have to change and the result is going to be quite a bit of additional meson code I'm afraid. Thoughts? [1] https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/SKJ… Andrea Bolognani (4): configmake: Check for WIN32 correctly meson: Always define WITH_*_DECL macros syntax-check: Ensure WITH_ macros are used correctly meson: Enable -Wundef build-aux/syntax-check.mk | 5 +++++ configmake.h.in | 2 +- meson.build | 3 +++ tests/virmockstathelpers.c | 28 ++++++++++++++-------------- 4 files changed, 23 insertions(+), 15 deletions(-) -- 2.43.2

4 8

Re: [PATCH] Сheck snapshot disk is not NULL when searching it in the VM config
by Peter Krempa 29 Jan '25

29 Jan '25

On Mon, May 20, 2024 at 14:48:47 +0000, Efim Shevrin via Devel wrote: > Hello, > > > If vmdisk is NULL, shouldn't this function (qemuSnapshotDeleteValidate()) return an error? > > I think this qemuSnapshotDeleteValidate should not return an error. > > It seems to me that when vmdisk is NULL, this does not invalidate > the snapshot itself, but indicates that the config has changed since > the snapshot was done. And if the VM config has changed, this adds evidence that the snapshot should be deleted, > because the snapshot does not reflect the real vm config. > > Since we do not have an analogue of the --force option for deleting a snapshot, in the case when qemuSnapshotDeleteValidate returns > an error when vmdisk is NULL, we will never delete a snapshot which has invalid disk. Snapshot deletion does have something that can be considered force and that is the '--metadata' option that removes just the snapshot definition (metadata) and doesn't touch the disk images. > > Similarly, disk can be NULL too > Thank you for the comment regarding the disk variable. I`ve reworked patch. > > When creating a snapshot of a VM with multiple hard disks, > the snapshot takes into account the presence of all disks > in the system. If, over time, one of the disks is deleted, > the snapshot will continue to store knowledge of the deleted disk. > This results in the fact that at the moment of deleting the snapshot, > at the validation stage, a disk from the snapshot will be searched which > is not in the VM configuration. As a result, vmdisk variable will > be equal to NULL. Dereferencing a null pointer at the time of calling > virStorageSourceIsSameLocation(vmdisk->src, disk->src) > will result in SIGSEGV. Crashing is obviously not okay ... > Also, the disk variable can also be equal to NULL and this > requires to check that disk != NULL before calling the > virStorageSourceIsSameLocation function to avoid SIGSEGV. .. but going ahead with the snapshot deletion isn't always okay either. The disk isn't referenced by the VM so the disk state can't be merged, while the state would be merged for any other disk. When reverting back to a previous snapshot, which is still referencing the older state of the disk which was removed from the VM, the VM would see that the image state of disks that were present at deletion would contain the merged state, but only a partial state for the disk which was later removed.

3 3

[libvirt PATCH] qemu_snapshot: allow reverting to external disk only snapshot
by Pavel Hrdina 29 Jan '25

29 Jan '25

When snapshot is created with disk-only flag it is always external snapshot without memory state. Historically when there was not support to revert external snapshots this produced error message. error: Failed to revert snapshot s1 error: internal error: Invalid target domain state 'disk-snapshot'. Refusing snapshot reversion Now we can simply consider this as reverting to offline snapshot as the possible damage to file system is already done at the point of snapshot creation. Resolves: https://issues.redhat.com/browse/RHEL-21549 Signed-off-by: Pavel Hrdina <phrdina(a)redhat.com> --- src/qemu/qemu_snapshot.c | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/qemu/qemu_snapshot.c b/src/qemu/qemu_snapshot.c index 0cac0c4146..7964f70553 100644 --- a/src/qemu/qemu_snapshot.c +++ b/src/qemu/qemu_snapshot.c @@ -2606,6 +2606,7 @@ qemuSnapshotRevert(virDomainObj *vm, case VIR_DOMAIN_SNAPSHOT_SHUTDOWN: case VIR_DOMAIN_SNAPSHOT_SHUTOFF: case VIR_DOMAIN_SNAPSHOT_CRASHED: + case VIR_DOMAIN_SNAPSHOT_DISK_SNAPSHOT: ret = qemuSnapshotRevertInactive(vm, snapshot, snap, driver, cfg, &inactiveConfig, @@ -2617,8 +2618,6 @@ qemuSnapshotRevert(virDomainObj *vm, _("qemu doesn't support reversion of snapshot taken in PMSUSPENDED state")); goto endjob; - case VIR_DOMAIN_SNAPSHOT_DISK_SNAPSHOT: - /* Rejected earlier as an external snapshot */ case VIR_DOMAIN_SNAPSHOT_NOSTATE: case VIR_DOMAIN_SNAPSHOT_BLOCKED: case VIR_DOMAIN_SNAPSHOT_LAST: -- 2.43.0

2 1

[PATCH 00/12] Introduce SEV-SNP support
by Michal Privoznik 22 Jan '25

22 Jan '25

SEV-SNP support just landed in QEMU. Here is the first round of patches to incorporate support into libvirt. TODOs (aka problems of future me): - Teach tools/virt-qemu-sev-validate how to deal with SEV-SNP - Try to find a SEV-SNP machine a test these patches in real worl - Write a kbase article on attestation with SEV-SNP Michal Prívozník (12): qemu_monitor_json: Report error in error paths in SEV related code conf: Move some members of virDomainSEVDef into virDomainSEVCommonDef conf: Separate SEV formatting into a function Drop needless typecast to virDomainLaunchSecurity src: Convert some _virDomainSecDef::sectype checks to switch() qemu_monitor: Allow querying SEV-SNP state in 'query-sev' qemu: Report snp-policy in virDomainGetLaunchSecurityInfo() qemu_capabilities: Introduce QEMU_CAPS_SEV_SNP_GUEST conf: Introduce SEV-SNP support qemu: Build cmd line for SEV-SNP qemu: Allow setting launch security for SEV-SNP qemu_firmware: Pick the right firmware for SEV-SNP guests docs/formatdomain.rst | 108 ++++++++++++ include/libvirt/libvirt-domain.h | 10 ++ src/conf/domain_conf.c | 156 ++++++++++++++---- src/conf/domain_conf.h | 28 +++- src/conf/domain_validate.c | 44 +++++ src/conf/schemas/domaincommon.rng | 73 ++++++-- src/conf/virconftypes.h | 4 + src/qemu/qemu_capabilities.c | 4 + src/qemu/qemu_capabilities.h | 3 + src/qemu/qemu_cgroup.c | 19 ++- src/qemu/qemu_command.c | 56 ++++++- src/qemu/qemu_driver.c | 60 +++++-- src/qemu/qemu_firmware.c | 20 ++- src/qemu/qemu_monitor.c | 7 +- src/qemu/qemu_monitor.h | 41 ++++- src/qemu/qemu_monitor_json.c | 67 ++++++-- src/qemu/qemu_monitor_json.h | 8 +- src/qemu/qemu_namespace.c | 3 +- src/qemu/qemu_process.c | 34 ++-- src/qemu/qemu_validate.c | 13 +- src/security/security_dac.c | 34 +++- .../caps_9.1.0_x86_64.xml | 1 + .../firmware/60-edk2-ovmf-x64-amdsev.json | 1 + tests/qemumonitorjsontest.c | 65 +++++++- ...launch-security-sev-snp.x86_64-latest.args | 35 ++++ .../launch-security-sev-snp.x86_64-latest.xml | 1 + .../launch-security-sev-snp.xml | 47 ++++++ tests/qemuxmlconftest.c | 2 + 28 files changed, 817 insertions(+), 127 deletions(-) create mode 100644 tests/qemuxmlconfdata/launch-security-sev-snp.x86_64-latest.args create mode 120000 tests/qemuxmlconfdata/launch-security-sev-snp.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/launch-security-sev-snp.xml -- 2.44.2

5 31

[PATCH 00/20] qemu: support mapped-ram+directio+mulitfd
by Jim Fehlig 15 Jan '25

15 Jan '25

This series is essentially V1 of a prior RFC [1] to support QEMU's mapped-ram stream format [2] and migration capability. Along with supporting mapped-ram, it implements a design approach we discussed for supporting parallel save/restore [3]. In summary, the approach is 1. Add mapped-ram migration capability 2. Steal an element from save header 'unused' for a 'features' variable and bump save version to 3. 3. Add /etc/libvirt/qemu.conf knob for the save format version, defaulting to latest v3 4. Use v3 (aka mapped-ram) by default 5. Use mapped-ram with BYPASS_CACHE for v3, old approach for v2 6. include: Define constants for parallel save/restore 7. qemu: Add support for parallel save. Implies mapped-ram, reject if v2 8. qemu: Add support for parallel restore. Implies mapped-ram. Reject if v2 9. tools: add parallel parameter to virsh save command 10. tools: add parallel parameter to virsh restore command With this series, saving and restoring using mapped-ram is enabled by default if the underlying QEMU advertises the mapped-ram migration capability. It can be disabled by changing the 'save_image_version' setting in qemu.conf. To use mapped-ram with QEMU: - The 'mapped-ram' migration capability must be set to true - The 'multifd' migration capability must be set to true and the 'multifd-channels' migration parameter must set to a value >= 1 - QEMU must be provided an fdset containing the migration fd(s) - The 'migrate' qmp command is invoked with a URI referencing the fdset and an offset where to start reading or writing the data stream, e.g. {"execute":"migrate", "arguments":{"detach":true,"resume":false, "uri":"file:/dev/fdset/0,offset=0x11921"}} The mapped-ram stream, in conjunction with direct IO and multifd, can significantly improve the time required to save VM memory state. The following tables compare mapped-ram with the existing, sequential save stream. In all cases, the save and restore operations are to/from a block device comprised of two NVMe disks in RAID0 configuration with xfs (~8600MiB/s). The values in the 'save time' and 'restore time' columns were scraped from the 'real' time reported by time(1). The 'Size' and 'Blocks' columns were provided by the corresponding outputs of stat(1). VM: 32G RAM, 1 vcpu, idle (shortly after boot) | save | restore | | time | time | Size | Blocks -----------------------+---------+---------+--------------+-------- legacy | 6.193s | 4.399s | 985744812 | 1925288 -----------------------+---------+---------+--------------+-------- mapped-ram | 5.109s | 1.176s | 34368554354 | 1774472 -----------------------+---------+---------+--------------+-------- legacy + direct IO | 5.725s | 4.512s | 985765251 | 1925328 -----------------------+---------+---------+--------------+-------- mapped-ram + direct IO | 4.627s | 1.490s | 34368554354 | 1774304 -----------------------+---------+---------+--------------+-------- mapped-ram + direct IO | | | | + multifd-channels=8 | 4.421s | 0.845s | 34368554318 | 1774312 ------------------------------------------------------------------- VM: 32G RAM, 30G dirty, 1 vcpu in tight loop dirtying memory | save | restore | | time | time | Size | Blocks -----------------------+---------+---------+--------------+--------- legacy | 25.800s | 14.332s | 33154309983 | 64754512 -----------------------+---------+---------+--------------+--------- mapped-ram | 18.742s | 15.027s | 34368559228 | 64617160 -----------------------+---------+---------+--------------+--------- legacy + direct IO | 13.115s | 18.050s | 33154310496 | 64754520 -----------------------+---------+---------+--------------+--------- mapped-ram + direct IO | 13.623s | 15.959s | 34368557392 | 64662040 -----------------------+-------- +---------+--------------+--------- mapped-ram + direct IO | | | | + multifd-channels=8 | 6.994s | 6.470s | 34368554980 | 64665776 -------------------------------------------------------------------- As can be seen from the tables, one caveat of mapped-ram is the logical file size of a saved image is basically equivalent to the VM memory size. Note however that mapped-ram typically uses fewer blocks on disk. Support for mapped-ram+direct-io only recently landed in upstream QEMU and will first appear in the 9.1 release, which may complicate merging support in libvirt. Specifically, I'm not sure how to detect if the combination is supported by QEMU. Suggestions welcomed. Similar to the RFC, V1 ignores compression. libvirt currently supports compression by connecting the output of QEMU's save stream to the specified compression program via a pipe. This approach is incompatible with mapped-ram since the fd provided to QEMU must be seekable. In general, we can consider mapped-ram and compression incompatible and document they cannot be used together. [1] https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/EF6… [2] https://gitlab.com/qemu-project/qemu/-/blob/master/docs/devel/migration/map… [3] https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/K4B… Claudio Fontana (2): include: Define constants for parallel save/restore tools: add parallel parameter to virsh restore command Jim Fehlig (17): lib: virDomainSaveParams: Ensure absolute save path qemu_fd: Add function to retrieve fdset ID qemu: Add function to check capability in migration params qemu: Add function to get bool value from migration params qemu: Add mapped-ram migration capability qemu: Add function to get migration params for save qemu: QEMU_SAVE_VERSION: Bump to version 3 qemu: conf: Add setting for save image version qemu: Add helper function for creating save image fd qemu: Add support for mapped-ram on save qemu: Decompose qemuSaveImageOpen qemu: Move creation of qemuProcessIncomingDef struct qemu: Apply migration parameters in qemuMigrationDstRun qemu: Add support for mapped-ram on restore qemu: Support O_DIRECT with mapped-ram on save qemu: Support O_DIRECT with mapped-ram on restore qemu: Add support for parallel save and restore Li Zhang (1): tools: add parallel parameter to virsh save command docs/manpages/virsh.rst | 9 +- include/libvirt/libvirt-domain.h | 13 ++ src/libvirt-domain.c | 52 +++++-- src/qemu/libvirtd_qemu.aug | 1 + src/qemu/qemu.conf.in | 6 + src/qemu/qemu_conf.c | 16 +++ src/qemu/qemu_conf.h | 5 + src/qemu/qemu_driver.c | 104 +++++++++----- src/qemu/qemu_fd.c | 18 +++ src/qemu/qemu_fd.h | 3 + src/qemu/qemu_migration.c | 192 +++++++++++++++++-------- src/qemu/qemu_migration.h | 9 +- src/qemu/qemu_migration_params.c | 86 ++++++++++++ src/qemu/qemu_migration_params.h | 17 +++ src/qemu/qemu_monitor.c | 39 ++++++ src/qemu/qemu_monitor.h | 5 + src/qemu/qemu_process.c | 120 +++++++++++----- src/qemu/qemu_process.h | 19 ++- src/qemu/qemu_saveimage.c | 216 ++++++++++++++++++++--------- src/qemu/qemu_saveimage.h | 35 +++-- src/qemu/qemu_snapshot.c | 26 ++-- src/qemu/test_libvirtd_qemu.aug.in | 1 + tools/virsh-domain.c | 79 +++++++++-- 23 files changed, 827 insertions(+), 244 deletions(-) -- 2.35.3

4 47