- Devel - libvirt List Archives

[RFC PATCH v3 0/6] Added virtio-net RSS with eBPF support.
by Andrew Melnychenko 27 Jun '24

27 Jun '24

This series of rfc patches adds support for loading the RSS eBPF program and passing it to the QEMU. Comments and suggestions would be useful. QEMU with vhost may work with RSS through eBPF. To load eBPF, the capabilities required that Libvirt may provide. eBPF program and maps may be unique for particular QEMU and Libvirt retrieves eBPF through qapi. For now, there is only "RSS" eBPF object in QEMU, in the future, there may be another one(g.e. network filters). That's why in Libvirt added logic to load and store any eBPF object that QEMU provides using qapi schema. One of the reasons why this series of patches is in RFC are tests. To this series of patches, the tests were added. For now, the tests are synthetic, the proper "reply" file should be generated with a new "caps" file. Currently, there are changes in caps-9.0.0* files. There was added support for ebpf_rss_fds feature, and request-ebpf command. Also, there was added new config for qemuConfig - allowEBPF. This config allows to enable/disable eBPF blob loading explicitly. This is required for qemuxmlconf tests - where some test expects that RSS would not support eBPF. So, overall, the tests are required for review, comment, and discussion how we want them to be implemented in the future. For virtio-net RSS, the document has not changed. ``` <interface type="network"> <model type="virtio"/> <driver queues="4" rss="on" rss_hash_report="off"/> <interface type="network"> ``` Simplified routine for RSS: * Libvirt retrieves eBPF "RSS" and load it. * Libvirt passes file descriptors to virtio-net with property "ebpf_rss_fds" ("rss" property should be "on" too). * if fds was provided - QEMU using eBPF RSS implementation. * if fds was not provided - QEMU tries to load eBPF RSS in own context and use it. * if eBPF RSS was not loaded - QEMU uses "in-qemu" RSS(vhost not supported). Changes since RFC v2: * refactored and rebased. * applied changes according to the Qemu. * added basic test. Changes since RFC v1: * changed eBPF format saved in the XML cache. * refactored and checked with syntax test. * refactored patch hunks. Andrew Melnychenko (6): qemu_monitor: Added QEMU's "request-ebpf" support. qemu_capabilities: Added logic for retrieving eBPF objects from QEMU. qemu_interface: Added routine for loading the eBPF objects. qemu_command: Added "ebpf_rss_fds" support for virtio-net. qemu_conf: Added configuration to optionally disable eBPF loading. tests: Added tests for eBPF blob loading. meson.build | 7 + meson_options.txt | 1 + src/qemu/meson.build | 1 + src/qemu/qemu_capabilities.c | 126 +++++++++++ src/qemu/qemu_capabilities.h | 6 + src/qemu/qemu_command.c | 63 ++++++ src/qemu/qemu_conf.c | 2 + src/qemu/qemu_conf.h | 2 + src/qemu/qemu_domain.c | 4 + src/qemu/qemu_domain.h | 3 + src/qemu/qemu_interface.c | 83 ++++++++ src/qemu/qemu_interface.h | 4 + src/qemu/qemu_monitor.c | 13 ++ src/qemu/qemu_monitor.h | 3 + src/qemu/qemu_monitor_json.c | 26 +++ src/qemu/qemu_monitor_json.h | 3 + .../caps_9.0.0_x86_64.replies | 199 ++++++++++-------- .../caps_9.0.0_x86_64.xml | 4 + tests/qemuxml2argvmock.c | 21 ++ .../net-virtio-rss-bpf.x86_64-latest.args | 37 ++++ .../net-virtio-rss-bpf.x86_64-latest.xml | 46 ++++ tests/qemuxmlconfdata/net-virtio-rss-bpf.xml | 46 ++++ tests/qemuxmlconftest.c | 4 + 23 files changed, 612 insertions(+), 92 deletions(-) create mode 100644 tests/qemuxmlconfdata/net-virtio-rss-bpf.x86_64-latest.args create mode 100644 tests/qemuxmlconfdata/net-virtio-rss-bpf.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/net-virtio-rss-bpf.xml -- 2.44.0

3 19

[PATCH v2] security_manager: Ensure top lock is acquired before nested locks
by hongmianquan 27 Jun '24

27 Jun '24

Fix libvirtd hang since fork() was called while another thread had security manager locked. We have the stack security driver, which internally manages other security drivers, just call them "top" and "nested". We call virSecurityStackPreFork() to lock the top one, and it also locks and then unlocks the nested drivers prior to fork. Then in qemuSecurityPostFork(), it unlocks the top one, but not the nested ones. Thus, if one of the nested drivers ("dac" or "selinux") is still locked, it will cause a deadlock. We discovered this case: the nested list obtained through the qemuSecurityGetNested() will be locked for subsequent use, such as in virQEMUDriverCreateCapabilities(), where the nested list is locked using qemuSecurityGetDOI, but the top one is not locked beforehand. The problem stack is as follows: libvirtd thread1 libvirtd thread2 child libvirtd | | | | | | virsh capabilities qemuProcessLanuch | | | | | lock top | | | | lock nested | | | | | | fork------------------->|(nested lock held by thread1) | | | | | | unlock nested unlock top unlock top | | qemuSecuritySetSocketLabel | | lock nested (deadlock) In this commit, we ensure that the top lock is acquired before the nested lock, so during fork, it's not possible for another task to acquire the nested lock. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1303031 Signed-off-by: hongmianquan <hongmianquan(a)bytedance.com> --- src/libvirt_private.syms | 3 ++- src/qemu/qemu_conf.c | 9 ++++++++- src/qemu/qemu_driver.c | 16 +++++++++------- src/qemu/qemu_security.h | 2 ++ src/security/security_manager.c | 22 ++++++++++++++++++++++ src/security/security_manager.h | 2 ++ 6 files changed, 45 insertions(+), 9 deletions(-) diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index bac4a8a366..39cdb90772 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -1806,7 +1806,8 @@ virSecurityManagerTransactionAbort; virSecurityManagerTransactionCommit; virSecurityManagerTransactionStart; virSecurityManagerVerify; - +virSecurityManagerStackLock; +virSecurityManagerStackUnlock; # security/security_util.h virSecurityXATTRNamespaceDefined; diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index 4050a82341..21f0739fd5 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -1380,6 +1380,9 @@ virCaps *virQEMUDriverCreateCapabilities(virQEMUDriver *driver) return NULL; } + /* Ensure top lock is acquired before nested locks */ + qemuSecurityStackLock(driver->securityManager); + /* access sec drivers and create a sec model for each one */ if (!(sec_managers = qemuSecurityGetNested(driver->securityManager))) return NULL; @@ -1402,8 +1405,10 @@ virCaps *virQEMUDriverCreateCapabilities(virQEMUDriver *driver) lbl = qemuSecurityGetBaseLabel(sec_managers[i], virtTypes[j]); type = virDomainVirtTypeToString(virtTypes[j]); if (lbl && - virCapabilitiesHostSecModelAddBaseLabel(sm, type, lbl) < 0) + virCapabilitiesHostSecModelAddBaseLabel(sm, type, lbl) < 0) { + qemuSecurityStackUnlock(driver->securityManager); return NULL; + } } VIR_DEBUG("Initialized caps for security driver \"%s\" with " @@ -1412,6 +1417,8 @@ virCaps *virQEMUDriverCreateCapabilities(virQEMUDriver *driver) caps->host.numa = virCapabilitiesHostNUMANewHost(); caps->host.cpu = virQEMUDriverGetHostCPU(driver); + + qemuSecurityStackUnlock(driver->securityManager); return g_steal_pointer(&caps); } diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index fc1704f4fc..c980a0990f 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -560,7 +560,6 @@ qemuStateInitialize(bool privileged, bool autostart = true; size_t i; const char *defsecmodel = NULL; - g_autofree virSecurityManager **sec_managers = NULL; g_autoptr(virIdentity) identity = virIdentityGetCurrent(); qemu_driver = g_new0(virQEMUDriver, 1); @@ -835,11 +834,8 @@ qemuStateInitialize(bool privileged, if (!qemu_driver->qemuCapsCache) goto error; - if (!(sec_managers = qemuSecurityGetNested(qemu_driver->securityManager))) - goto error; - - if (sec_managers[0] != NULL) - defsecmodel = qemuSecurityGetModel(sec_managers[0]); + if (qemu_driver->securityManager != NULL) + defsecmodel = qemuSecurityGetModel(qemu_driver->securityManager); if (!(qemu_driver->xmlopt = virQEMUDriverCreateXMLConf(qemu_driver, defsecmodel))) @@ -5663,7 +5659,12 @@ static int qemuDomainGetSecurityLabelList(virDomainPtr dom, ret = 0; } else { int len = 0; - virSecurityManager ** mgrs = qemuSecurityGetNested(driver->securityManager); + virSecurityManager ** mgrs = NULL; + + /* Ensure top lock is acquired before nested locks */ + qemuSecurityStackLock(driver->securityManager); + + mgrs = qemuSecurityGetNested(driver->securityManager); if (!mgrs) goto cleanup; @@ -5688,6 +5689,7 @@ static int qemuDomainGetSecurityLabelList(virDomainPtr dom, } cleanup: + qemuSecurityStackUnlock(driver->securityManager); virDomainObjEndAPI(&vm); return ret; } diff --git a/src/qemu/qemu_security.h b/src/qemu/qemu_security.h index 41da33debc..19fcb3c939 100644 --- a/src/qemu/qemu_security.h +++ b/src/qemu/qemu_security.h @@ -151,3 +151,5 @@ int qemuSecurityCommandRun(virQEMUDriver *driver, #define qemuSecuritySetTapFDLabel virSecurityManagerSetTapFDLabel #define qemuSecurityStackAddNested virSecurityManagerStackAddNested #define qemuSecurityVerify virSecurityManagerVerify +#define qemuSecurityStackLock virSecurityManagerStackLock +#define qemuSecurityStackUnlock virSecurityManagerStackUnlock \ No newline at end of file diff --git a/src/security/security_manager.c b/src/security/security_manager.c index 24f2f3d3dc..c49c4f708f 100644 --- a/src/security/security_manager.c +++ b/src/security/security_manager.c @@ -989,6 +989,28 @@ virSecurityManagerGetNested(virSecurityManager *mgr) return list; } +/* + * Usually called before virSecurityManagerGetNested(). + * We need to ensure locking the stack security manager before + * locking the nested security manager to maintain the correct + * synchronization state. + * It must be followed by a call virSecurityManagerStackUnlock(). + */ +void +virSecurityManagerStackLock(virSecurityManager *mgr) +{ + if (STREQ("stack", mgr->drv->name)) + virObjectLock(mgr); +} + + +void +virSecurityManagerStackUnlock(virSecurityManager *mgr) +{ + if (STREQ("stack", mgr->drv->name)) + virObjectUnlock(mgr); +} + /** * virSecurityManagerDomainSetPathLabel: diff --git a/src/security/security_manager.h b/src/security/security_manager.h index a416af3215..bb6d22bc31 100644 --- a/src/security/security_manager.h +++ b/src/security/security_manager.h @@ -158,6 +158,8 @@ int virSecurityManagerSetTapFDLabel(virSecurityManager *mgr, char *virSecurityManagerGetMountOptions(virSecurityManager *mgr, virDomainDef *vm); virSecurityManager ** virSecurityManagerGetNested(virSecurityManager *mgr); +void virSecurityManagerStackLock(virSecurityManager *mgr); +void virSecurityManagerStackUnlock(virSecurityManager *mgr); typedef enum { VIR_SECURITY_DOMAIN_IMAGE_LABEL_BACKING_CHAIN = 1 << 0, -- 2.11.0

1 0

[PATCH] qemuDomainDiskChangeSupported: Fill in missing check
by Adam Julis 26 Jun '24

26 Jun '24

Any modification of iothreads attribute is prohibit, the original setup must be identical with updated version of xml. Resolves: https://issues.redhat.com/browse/RHEL-23607 Signed-off-by: Adam Julis <ajulis(a)redhat.com> --- src/qemu/qemu_domain.c | 47 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 1a90311ca5..6d7b939a31 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -8484,6 +8484,53 @@ qemuDomainDiskChangeSupported(virDomainDiskDef *disk, CHECK_EQ(discard, "discard", true); CHECK_EQ(iothread, "iothread", true); + /* check if new or original disk contains iothreads information + * if yes, they must be identical, any changes are prohibit*/ + + if (!!disk->iothreads != !!orig_disk->iothreads) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, + _("cannot modify field '%1$s' of the disk"), + "iothreads"); + return false; + } + + if (disk->iothreads && orig_disk->iothreads) { + GSList *n, *m; + + m = disk->iothreads; + + for (n = orig_disk->iothreads; n; n = n->next) { + size_t i; + virDomainDiskIothreadDef *n_data = (virDomainDiskIothreadDef *) n->data; + virDomainDiskIothreadDef *m_data = (virDomainDiskIothreadDef *) m->data; + + if (n_data->id != m_data->id || n_data->nqueues != m_data->nqueues) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, + _("cannot modify field '%1$s' of the disk"), + "iothreads"); + return false; + } + + for (i = 0; i < n_data->nqueues; i++) { + if (n_data->queues[i] != m_data->queues[i]) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, + _("cannot modify field '%1$s' of the disk"), + "iothreads"); + return false; + } + } + + m = m->next; + + if (!m && n->next) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, + _("cannot modify field '%1$s' of the disk"), + "iothreads"); + return false; + } + } + } + CHECK_STREQ_NULLABLE(domain_name, "backenddomain"); -- 2.45.2

1 0

[libvirt RFC PATCH 0/2] qemu: do not use deprecated options for new virtiofsd
by Ján Tomko 25 Jun '24

25 Jun '24

Depends on https://gitlab.com/virtio-fs/virtiofsd/-/merge_requests/231 Ján Tomko (2): qemu: fill capabilities for virtiofsd qemu: do not use deprecated options for new virtiofsd src/conf/domain_conf.h | 1 + src/qemu/qemu_vhost_user.c | 59 +++++++++++++++++++++++++ src/qemu/qemu_vhost_user.h | 12 ++++++ src/qemu/qemu_virtiofs.c | 88 ++++++++++++++++++++++++++------------ 4 files changed, 132 insertions(+), 28 deletions(-) -- 2.45.2

1 2

[PATCH 0/8] Expose SEV-SNP in domcaps and virt-host-validate
by Michal Privoznik 25 Jun '24

25 Jun '24

This is a promised follow up to: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/message/7QQ… Michal Prívozník (8): libvirt_private.syms: Export virDomainLaunchSecurity enum handlers qemuxmlconftest; Explicitly enable QEMU_CAPS_SEV_SNP_GUEST for "launch-security-sev-snp" qemu_capabilities: Probe SEV capabilities even for QEMU_CAPS_SEV_SNP_GUEST domcaps: Report launchSecurity qemu: Fill launchSecurity in domaincaps qemu_validate: Use domaincaps to validate supported launchSecurity type virt-host-validate: Move AMD SEV into a separate func virt-host-validate: Detect SEV-ES and SEV-SNP docs/formatdomaincaps.rst | 10 +++ src/conf/domain_capabilities.c | 14 ++++ src/conf/domain_capabilities.h | 9 ++ src/conf/schemas/domaincaps.rng | 10 +++ src/libvirt_private.syms | 2 + src/qemu/qemu_capabilities.c | 24 +++++- src/qemu/qemu_capabilities.h | 3 + src/qemu/qemu_validate.c | 29 +++---- .../domaincapsdata/qemu_4.2.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_4.2.0-tcg.x86_64.xml | 3 + .../qemu_4.2.0-virt.aarch64.xml | 3 + tests/domaincapsdata/qemu_4.2.0.aarch64.xml | 3 + tests/domaincapsdata/qemu_4.2.0.ppc64.xml | 3 + tests/domaincapsdata/qemu_4.2.0.s390x.xml | 3 + tests/domaincapsdata/qemu_4.2.0.x86_64.xml | 3 + .../domaincapsdata/qemu_5.0.0-q35.x86_64.xml | 3 + .../qemu_5.0.0-tcg-virt.riscv64.xml | 3 + .../domaincapsdata/qemu_5.0.0-tcg.x86_64.xml | 3 + .../qemu_5.0.0-virt.aarch64.xml | 3 + .../qemu_5.0.0-virt.riscv64.xml | 3 + tests/domaincapsdata/qemu_5.0.0.aarch64.xml | 3 + tests/domaincapsdata/qemu_5.0.0.ppc64.xml | 3 + tests/domaincapsdata/qemu_5.0.0.x86_64.xml | 3 + .../domaincapsdata/qemu_5.1.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_5.1.0-tcg.x86_64.xml | 3 + tests/domaincapsdata/qemu_5.1.0.sparc.xml | 3 + tests/domaincapsdata/qemu_5.1.0.x86_64.xml | 3 + .../domaincapsdata/qemu_5.2.0-q35.x86_64.xml | 3 + .../qemu_5.2.0-tcg-virt.riscv64.xml | 3 + .../domaincapsdata/qemu_5.2.0-tcg.x86_64.xml | 3 + .../qemu_5.2.0-virt.aarch64.xml | 3 + .../qemu_5.2.0-virt.riscv64.xml | 3 + tests/domaincapsdata/qemu_5.2.0.aarch64.xml | 3 + tests/domaincapsdata/qemu_5.2.0.ppc64.xml | 3 + tests/domaincapsdata/qemu_5.2.0.s390x.xml | 3 + tests/domaincapsdata/qemu_5.2.0.x86_64.xml | 3 + .../domaincapsdata/qemu_6.0.0-q35.x86_64.xml | 5 ++ .../domaincapsdata/qemu_6.0.0-tcg.x86_64.xml | 5 ++ .../qemu_6.0.0-virt.aarch64.xml | 3 + tests/domaincapsdata/qemu_6.0.0.aarch64.xml | 3 + tests/domaincapsdata/qemu_6.0.0.s390x.xml | 5 ++ tests/domaincapsdata/qemu_6.0.0.x86_64.xml | 5 ++ .../domaincapsdata/qemu_6.1.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_6.1.0-tcg.x86_64.xml | 3 + tests/domaincapsdata/qemu_6.1.0.x86_64.xml | 3 + .../domaincapsdata/qemu_6.2.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_6.2.0-tcg.x86_64.xml | 3 + .../qemu_6.2.0-virt.aarch64.xml | 3 + tests/domaincapsdata/qemu_6.2.0.aarch64.xml | 3 + tests/domaincapsdata/qemu_6.2.0.ppc64.xml | 3 + tests/domaincapsdata/qemu_6.2.0.x86_64.xml | 3 + .../qemu_7.0.0-hvf.aarch64+hvf.xml | 3 + .../domaincapsdata/qemu_7.0.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_7.0.0-tcg.x86_64.xml | 3 + .../qemu_7.0.0-virt.aarch64.xml | 3 + tests/domaincapsdata/qemu_7.0.0.aarch64.xml | 3 + tests/domaincapsdata/qemu_7.0.0.ppc64.xml | 3 + tests/domaincapsdata/qemu_7.0.0.x86_64.xml | 3 + .../domaincapsdata/qemu_7.1.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_7.1.0-tcg.x86_64.xml | 3 + tests/domaincapsdata/qemu_7.1.0.ppc64.xml | 3 + tests/domaincapsdata/qemu_7.1.0.x86_64.xml | 3 + .../qemu_7.2.0-hvf.x86_64+hvf.xml | 3 + .../domaincapsdata/qemu_7.2.0-q35.x86_64.xml | 3 + .../qemu_7.2.0-tcg.x86_64+hvf.xml | 3 + .../domaincapsdata/qemu_7.2.0-tcg.x86_64.xml | 3 + tests/domaincapsdata/qemu_7.2.0.ppc.xml | 3 + tests/domaincapsdata/qemu_7.2.0.x86_64.xml | 3 + .../domaincapsdata/qemu_8.0.0-q35.x86_64.xml | 3 + .../qemu_8.0.0-tcg-virt.riscv64.xml | 3 + .../domaincapsdata/qemu_8.0.0-tcg.x86_64.xml | 3 + .../qemu_8.0.0-virt.riscv64.xml | 3 + tests/domaincapsdata/qemu_8.0.0.x86_64.xml | 3 + .../domaincapsdata/qemu_8.1.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_8.1.0-tcg.x86_64.xml | 3 + tests/domaincapsdata/qemu_8.1.0.s390x.xml | 5 ++ tests/domaincapsdata/qemu_8.1.0.x86_64.xml | 3 + .../domaincapsdata/qemu_8.2.0-q35.x86_64.xml | 3 + .../qemu_8.2.0-tcg-virt.loongarch64.xml | 3 + .../domaincapsdata/qemu_8.2.0-tcg.x86_64.xml | 3 + .../qemu_8.2.0-virt.aarch64.xml | 3 + .../qemu_8.2.0-virt.loongarch64.xml | 3 + tests/domaincapsdata/qemu_8.2.0.aarch64.xml | 3 + tests/domaincapsdata/qemu_8.2.0.armv7l.xml | 3 + tests/domaincapsdata/qemu_8.2.0.s390x.xml | 5 ++ tests/domaincapsdata/qemu_8.2.0.x86_64.xml | 3 + .../domaincapsdata/qemu_9.0.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_9.0.0-tcg.x86_64.xml | 3 + tests/domaincapsdata/qemu_9.0.0.x86_64.xml | 3 + .../domaincapsdata/qemu_9.1.0-q35.x86_64.xml | 3 + .../domaincapsdata/qemu_9.1.0-tcg.x86_64.xml | 3 + tests/domaincapsdata/qemu_9.1.0.x86_64.xml | 3 + .../caps_9.1.0_x86_64.xml | 1 - tests/qemuxmlconftest.c | 6 +- tools/virt-host-validate-common.c | 83 ++++++++++++++----- 95 files changed, 413 insertions(+), 42 deletions(-) -- 2.44.2

3 12

[PATCH 0/2] qemu_command: Enable sev-guest.legacy-vm-type when possible
by Michal Privoznik 25 Jun '24

25 Jun '24

*** IMPORTANT *** There's a competing patch sent to qemu-devel which when merged renders these patches unnecessary: https://mail.gnu.org/archive/html/qemu-devel/2024-06/msg02776.html But I accumulated this change whilst working on previous patches that enabled SEV SNP and figured might as well send them if QEMU patch doesn't make it in time for our release. Michal Prívozník (2): qemu_capabilities: Introduce QEMU_CAPS_SEV_GUEST_LEGACY_VM_TYPE qemu_command: Enable sev-guest.legacy-vm-type when possible src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 9 +++++++++ tests/qemucapabilitiesdata/caps_9.1.0_x86_64.xml | 1 + .../launch-security-sev-direct.x86_64-latest.args | 2 +- 5 files changed, 14 insertions(+), 1 deletion(-) -- 2.44.2

1 2

[PATCH v2] qemu_block: Validate number of hosts for iSCSI disk device
by Rayhan Faizel 25 Jun '24

25 Jun '24

An iSCSI device with zero hosts will result in a segmentation fault. This patch adds a check for the number of hosts, which must be one in the case of iSCSI. Minimal reproducing XML: <domain type='qemu'> <name>MyGuest</name> <uuid>4dea22b3-1d52-d8f3-2516-782e98ab3fa0</uuid> <os> <type arch='x86_64'>hvm</type> </os> <memory>4096</memory> <devices> <disk type='network'> <source name='dummy' protocol='iscsi'/> <target dev='vda'/> </disk> </devices> </domain> Signed-off-by: Rayhan Faizel <rayhan.faizel(a)gmail.com> --- This crashing XML was detected by the WIP fuzzer which is being developed as part of Google Summer of Code 2024. [Changes in v2] - Added testcase for iSCSI disk definitions with zero hosts src/qemu/qemu_block.c | 6 ++++ ...iscsi-zero-hosts-invalid.x86_64-latest.err | 1 + ...iscsi-zero-hosts-invalid.x86_64-latest.xml | 35 +++++++++++++++++++ .../disk-network-iscsi-zero-hosts-invalid.xml | 27 ++++++++++++++ tests/qemuxmlconftest.c | 2 ++ 5 files changed, 71 insertions(+) create mode 100644 tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.err create mode 100644 tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.xml create mode 100644 tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.xml diff --git a/src/qemu/qemu_block.c b/src/qemu/qemu_block.c index 738b72d7ea..d6cdf521c4 100644 --- a/src/qemu/qemu_block.c +++ b/src/qemu/qemu_block.c @@ -602,6 +602,12 @@ qemuBlockStorageSourceGetISCSIProps(virStorageSource *src, * } */ + if (src->nhosts != 1) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("iSCSI protocol accepts only one host")); + return NULL; + } + target = g_strdup(src->path); /* Separate the target and lun */ diff --git a/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.err b/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.err new file mode 100644 index 0000000000..ec66bebf22 --- /dev/null +++ b/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.err @@ -0,0 +1 @@ +internal error: iSCSI protocol accepts only one host diff --git a/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.xml b/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.xml new file mode 100644 index 0000000000..ad556180ab --- /dev/null +++ b/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.x86_64-latest.xml @@ -0,0 +1,35 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='x86_64' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <cpu mode='custom' match='exact' check='none'> + <model fallback='forbid'>qemu64</model> + </cpu> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + <disk type='network' device='disk'> + <driver name='qemu' type='raw'/> + <source protocol='iscsi' name='iqn.1992-01.com.example:storage/1'/> + <target dev='vda' bus='virtio'/> + <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> + </disk> + <controller type='usb' index='0' model='piix3-uhci'> + <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> + </controller> + <controller type='pci' index='0' model='pci-root'/> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <audio id='1' type='none'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.xml b/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.xml new file mode 100644 index 0000000000..6369f01644 --- /dev/null +++ b/tests/qemuxmlconfdata/disk-network-iscsi-zero-hosts-invalid.xml @@ -0,0 +1,27 @@ +<domain type='qemu'> + <name>QEMUGuest1</name> + <uuid>c7a5fdbd-edaf-9455-926a-d65c16db1809</uuid> + <memory unit='KiB'>219136</memory> + <currentMemory unit='KiB'>219136</currentMemory> + <vcpu placement='static'>1</vcpu> + <os> + <type arch='x86_64' machine='pc'>hvm</type> + <boot dev='hd'/> + </os> + <clock offset='utc'/> + <on_poweroff>destroy</on_poweroff> + <on_reboot>restart</on_reboot> + <on_crash>destroy</on_crash> + <devices> + <emulator>/usr/bin/qemu-system-x86_64</emulator> + <disk type='network' device='disk'> + <source protocol='iscsi' name='iqn.1992-01.com.example:storage/1'/> + <target dev='vda' bus='virtio'/> + </disk> + <controller type='usb' index='0'/> + <controller type='pci' index='0' model='pci-root'/> + <input type='mouse' bus='ps2'/> + <input type='keyboard' bus='ps2'/> + <memballoon model='none'/> + </devices> +</domain> diff --git a/tests/qemuxmlconftest.c b/tests/qemuxmlconftest.c index 2842b44b3e..0c71db1f22 100644 --- a/tests/qemuxmlconftest.c +++ b/tests/qemuxmlconftest.c @@ -2987,6 +2987,8 @@ mymain(void) DO_TEST_CAPS_LATEST("net-usb") DO_TEST_CAPS_LATEST("sound-device-virtio") + DO_TEST_CAPS_LATEST_FAILURE("disk-network-iscsi-zero-hosts-invalid") + /* check that all input files were actually used here */ if (testConfXMLCheck(existingTestCases) < 0) ret = -1; -- 2.34.1

2 3

[PATCH v2] qemu: add support for qemu switchover-ack
by Jon Kohler 25 Jun '24

25 Jun '24

Add plumbing for QEMU's switchover-ack migration capability, which helps lower the downtime during VFIO migrations. This capability is enabled by default as long as both the source and destination support it. Note: switchover-ack depends on the return path capability, so this may not be used when VIR_MIGRATE_TUNNELLED flag is set. Extensive details about the qemu switchover-ack implementation are available in the qemu series v6 cover letter [1] where the highlight is the extreme reduction in guest visible downtime. In addition to the original test results below, I saw a roughly ~20% reduction in downtime for VFIO VGPU devices at minimum. === Test results === The below table shows the downtime of two identical migrations. In the first migration swithcover ack is disabled and in the second it is enabled. The migrated VM is assigned with a mlx5 VFIO device which has 300MB of device data to be migrated. +----------------------+-----------------------+----------+ | Switchover ack | VFIO device data size | Downtime | +----------------------+-----------------------+----------+ | Disabled | 300MB | 1900ms | | Enabled | 300MB | 420ms | +----------------------+-----------------------+----------+ Switchover ack gives a roughly 4.5 times improvement in downtime. The 1480ms difference is time that is used for resource allocation for the VFIO device in the destination. Without switchover ack, this time is spent when the source VM is stopped and thus the downtime is much higher. With switchover ack, the time is spent when the source VM is still running. [1] https://patchwork.kernel.org/project/qemu-devel/cover/20230621111201.29729-… Signed-off-by: Jon Kohler <jon(a)nutanix.com> Cc: Alex Williamson <alex.williamson(a)redhat.com> Cc: Avihai Horon <avihaih(a)nvidia.com> Cc: Markus Armbruster <armbru(a)redhat.com> Cc: Peter Xu <peterx(a)redhat.com> Cc: YangHang Liu <yanghliu(a)redhat.com> --- v1 - https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/2XCW… v1 -> v2: - Addressed comments to simplify approach (Daniel, Jiri) --- src/qemu/qemu_migration.h | 1 + src/qemu/qemu_migration_params.c | 8 +++++++- src/qemu/qemu_migration_params.h | 1 + 3 files changed, 9 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index ed62fd4a91..cd89e100e1 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -62,6 +62,7 @@ VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES | \ VIR_MIGRATE_POSTCOPY_RESUME | \ VIR_MIGRATE_ZEROCOPY | \ + VIR_MIGRATE_SWITCHOVER_ACK | \ 0) /* All supported migration parameters and their types. */ diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c index 48f8657f71..9593b6ba65 100644 --- a/src/qemu/qemu_migration_params.c +++ b/src/qemu/qemu_migration_params.c @@ -105,6 +105,7 @@ VIR_ENUM_IMPL(qemuMigrationCapability, "return-path", "zero-copy-send", "postcopy-preempt", + "switchover-ack", ); @@ -138,7 +139,7 @@ struct _qemuMigrationParamsAlwaysOnItem { typedef struct _qemuMigrationParamsFlagMapItem qemuMigrationParamsFlagMapItem; struct _qemuMigrationParamsFlagMapItem { /* Describes what to do with the capability if @flag is found. - * When se to QEMU_MIGRATION_FLAG_REQUIRED, the capability will be + * When set to QEMU_MIGRATION_FLAG_REQUIRED, the capability will be * enabled iff the specified migration flag is enabled. On the other hand * QEMU_MIGRATION_FLAG_FORBIDDEN will enable the capability as long as * the specified migration flag is not enabled. */ @@ -215,6 +216,11 @@ static const qemuMigrationParamsFlagMapItem qemuMigrationParamsFlagMap[] = { .flag = VIR_MIGRATE_ZEROCOPY, .cap = QEMU_MIGRATION_CAP_ZERO_COPY_SEND, .party = QEMU_MIGRATION_SOURCE}, + + {.match = QEMU_MIGRATION_FLAG_FORBIDDEN, + .flag = VIR_MIGRATE_TUNNELLED, + .cap = QEMU_MIGRATION_CAP_SWITCHOVER_ACK, + .party = QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION}, }; /* Translation from VIR_MIGRATE_PARAM_* typed parameters to diff --git a/src/qemu/qemu_migration_params.h b/src/qemu/qemu_migration_params.h index 91bc6792cd..df67f1fb92 100644 --- a/src/qemu/qemu_migration_params.h +++ b/src/qemu/qemu_migration_params.h @@ -41,6 +41,7 @@ typedef enum { QEMU_MIGRATION_CAP_RETURN_PATH, QEMU_MIGRATION_CAP_ZERO_COPY_SEND, QEMU_MIGRATION_CAP_POSTCOPY_PREEMPT, + QEMU_MIGRATION_CAP_SWITCHOVER_ACK, QEMU_MIGRATION_CAP_LAST } qemuMigrationCapability; -- 2.43.0

2 1

[PATCH] qemu: Fix migration with disabled vmx-* CPU features
by Jiri Denemark 25 Jun '24

25 Jun '24

When starting a domain on a host which lacks a vmx-* CPU feature which is expected to be enabled by the CPU model specified in the domain XML, libvirt properly marks such feature as disabled in the active domain XML. But migrating the domain to a similar host which lacks the same vmx-* feature will fail with libvirt reporting the feature as missing. This is because of a bug in the hack ensuring backward compatibility libvirt running on the destination thinks the missing feature is expected to be enabled. https://issues.redhat.com/browse/RHEL-40899 Fixes: v10.1.0-85-g5fbfa5ab8a Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com> --- src/cpu/cpu_x86.c | 44 +++++++++++++++++++++++++++----------------- 1 file changed, 27 insertions(+), 17 deletions(-) diff --git a/src/cpu/cpu_x86.c b/src/cpu/cpu_x86.c index 7a70eed9e7..fcbce0ec46 100644 --- a/src/cpu/cpu_x86.c +++ b/src/cpu/cpu_x86.c @@ -3021,10 +3021,7 @@ virCPUx86UpdateLive(virCPUDef *cpu, if (!(map = virCPUx86GetMap())) return -1; - if (!(model = x86ModelFromCPU(cpu, map, -1))) - return -1; - - if (hostPassthrough && + if (!(model = x86ModelFromCPU(cpu, map, -1)) || !(modelDisabled = x86ModelFromCPU(cpu, map, VIR_CPU_FEATURE_DISABLE))) return -1; @@ -3037,12 +3034,19 @@ virCPUx86UpdateLive(virCPUDef *cpu, for (i = 0; i < map->nfeatures; i++) { virCPUx86Feature *feature = map->features[i]; virCPUFeaturePolicy expected = VIR_CPU_FEATURE_LAST; + bool explicit = false; + bool ignore = false; - if (x86DataIsSubset(&model->data, &feature->data)) + if (x86DataIsSubset(&model->data, &feature->data)) { + explicit = true; expected = VIR_CPU_FEATURE_REQUIRE; - else if (!hostPassthrough || - x86DataIsSubset(&modelDisabled->data, &feature->data)) + } else if (x86DataIsSubset(&modelDisabled->data, &feature->data)) { + explicit = true; + expected = VIR_CPU_FEATURE_DISABLE; + } else if (!hostPassthrough) { + /* implicitly disabled */ expected = VIR_CPU_FEATURE_DISABLE; + } if (x86DataIsSubset(&enabled, &feature->data) && x86DataIsSubset(&disabled, &feature->data)) { @@ -3052,30 +3056,36 @@ virCPUx86UpdateLive(virCPUDef *cpu, return -1; } + /* Features enabled or disabled by the hypervisor are ignored by + * check='full' in case they were added to the model later and not + * explicitly mentioned in the CPU definition. This matches how libvirt + * behaved before the features were added. + */ + if (!explicit && + g_strv_contains((const char **) model->addedFeatures, feature->name)) + ignore = true; + if (expected == VIR_CPU_FEATURE_DISABLE && x86DataIsSubset(&enabled, &feature->data)) { VIR_DEBUG("Feature '%s' enabled by the hypervisor", feature->name); - /* Extra features enabled by the hypervisor are ignored by - * check='full' in case they were added to the model later for - * backward compatibility with the older definition of the model. - */ - if (cpu->check == VIR_CPU_CHECK_FULL && - !g_strv_contains((const char **) model->addedFeatures, feature->name)) { + if (cpu->check == VIR_CPU_CHECK_FULL && !ignore) virBufferAsprintf(&bufAdded, "%s,", feature->name); - } else { + else virCPUDefUpdateFeature(cpu, feature->name, VIR_CPU_FEATURE_REQUIRE); - } } if (x86DataIsSubset(&disabled, &feature->data) || (expected == VIR_CPU_FEATURE_REQUIRE && !x86DataIsSubset(&enabled, &feature->data))) { VIR_DEBUG("Feature '%s' disabled by the hypervisor", feature->name); - if (cpu->check == VIR_CPU_CHECK_FULL) + + if (cpu->check == VIR_CPU_CHECK_FULL && !ignore) { virBufferAsprintf(&bufRemoved, "%s,", feature->name); - else + } else { virCPUDefUpdateFeature(cpu, feature->name, VIR_CPU_FEATURE_DISABLE); + x86DataSubtract(&disabled, &feature->data); + } } } -- 2.45.1

2 1

[PATCH] qemu: add support for qemu switchover-ack
by Jon Kohler 24 Jun '24

24 Jun '24

Add plumbing for QEMU's switchover-ack migration capability, which helps lower the downtime during VFIO migrations. This capability is enabled by default as long as both the source and destination support it. Note: switchover-ack depends on the return path capability, so this may not be used when VIR_MIGRATE_TUNNELLED flag is set. Extensive details about the qemu switchover-ack implementation are available in the qemu series v6 cover letter [1] where the highlight is the extreme reduction in guest visible downtime. In addition to the original test results below, I saw a roughly ~20% reduction in downtime for VFIO VGPU devices at minimum. === Test results === The below table shows the downtime of two identical migrations. In the first migration swithcover ack is disabled and in the second it is enabled. The migrated VM is assigned with a mlx5 VFIO device which has 300MB of device data to be migrated. +----------------------+-----------------------+----------+ | Switchover ack | VFIO device data size | Downtime | +----------------------+-----------------------+----------+ | Disabled | 300MB | 1900ms | | Enabled | 300MB | 420ms | +----------------------+-----------------------+----------+ Switchover ack gives a roughly 4.5 times improvement in downtime. The 1480ms difference is time that is used for resource allocation for the VFIO device in the destination. Without switchover ack, this time is spent when the source VM is stopped and thus the downtime is much higher. With switchover ack, the time is spent when the source VM is still running. [1] https://patchwork.kernel.org/project/qemu-devel/cover/20230621111201.29729-… Signed-off-by: Jon Kohler <jon(a)nutanix.com> Cc: Alex Williamson <alex.williamson(a)redhat.com> Cc: Avihai Horon <avihaih(a)nvidia.com> Cc: Markus Armbruster <armbru(a)redhat.com> Cc: Peter Xu <peterx(a)redhat.com> Cc: YangHang Liu <yanghliu(a)redhat.com> --- include/libvirt/libvirt-domain.h | 11 +++++++++++ src/libvirt-domain.c | 20 ++++++++++++++++++++ src/qemu/qemu_migration.h | 1 + src/qemu/qemu_migration_params.c | 8 +++++++- src/qemu/qemu_migration_params.h | 1 + 5 files changed, 40 insertions(+), 1 deletion(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 2f5b01bbfe..9543629f30 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1100,6 +1100,17 @@ typedef enum { * Since: 8.5.0 */ VIR_MIGRATE_ZEROCOPY = (1 << 20), + + /* Use switchover ack migration capability to reduce downtime on VFIO + * device migration. This prevents the source from stopping the VM and + * completing the migration until an ACK is received from the destination + * that it's OK to do so. Thus, a VFIO device can make sure that its + * initial bytes were sent and loaded in the destination before the + * source VM is stopped. + * + * Since: 10.5.0 + */ + VIR_MIGRATE_SWITCHOVER_ACK = (1 << 21), } virDomainMigrateFlags; diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 7c6b93963c..786fef317d 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -3822,6 +3822,10 @@ virDomainMigrate(virDomainPtr domain, VIR_MIGRATE_PARALLEL, error); + VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_TUNNELLED, + VIR_MIGRATE_SWITCHOVER_ACK, + error); + VIR_REQUIRE_FLAG_GOTO(VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES, VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC, error); @@ -4021,6 +4025,10 @@ virDomainMigrate2(virDomainPtr domain, VIR_MIGRATE_PARALLEL, error); + VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_TUNNELLED, + VIR_MIGRATE_SWITCHOVER_ACK, + error); + VIR_REQUIRE_FLAG_GOTO(VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES, VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC, error); @@ -4497,6 +4505,10 @@ virDomainMigrateToURI(virDomainPtr domain, VIR_MIGRATE_PARALLEL, error); + VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_TUNNELLED, + VIR_MIGRATE_SWITCHOVER_ACK, + error); + if (virDomainMigrateUnmanagedCheckCompat(domain, flags) < 0) goto error; @@ -4577,6 +4589,10 @@ virDomainMigrateToURI2(virDomainPtr domain, VIR_MIGRATE_PARALLEL, error); + VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_TUNNELLED, + VIR_MIGRATE_SWITCHOVER_ACK, + error); + if (virDomainMigrateUnmanagedCheckCompat(domain, flags) < 0) goto error; @@ -4656,6 +4672,10 @@ virDomainMigrateToURI3(virDomainPtr domain, VIR_MIGRATE_PARALLEL, error); + VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_TUNNELLED, + VIR_MIGRATE_SWITCHOVER_ACK, + error); + if (virDomainMigrateUnmanagedCheckCompat(domain, flags) < 0) goto error; diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index ed62fd4a91..cd89e100e1 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -62,6 +62,7 @@ VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES | \ VIR_MIGRATE_POSTCOPY_RESUME | \ VIR_MIGRATE_ZEROCOPY | \ + VIR_MIGRATE_SWITCHOVER_ACK | \ 0) /* All supported migration parameters and their types. */ diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c index 48f8657f71..9593b6ba65 100644 --- a/src/qemu/qemu_migration_params.c +++ b/src/qemu/qemu_migration_params.c @@ -105,6 +105,7 @@ VIR_ENUM_IMPL(qemuMigrationCapability, "return-path", "zero-copy-send", "postcopy-preempt", + "switchover-ack", ); @@ -138,7 +139,7 @@ struct _qemuMigrationParamsAlwaysOnItem { typedef struct _qemuMigrationParamsFlagMapItem qemuMigrationParamsFlagMapItem; struct _qemuMigrationParamsFlagMapItem { /* Describes what to do with the capability if @flag is found. - * When se to QEMU_MIGRATION_FLAG_REQUIRED, the capability will be + * When set to QEMU_MIGRATION_FLAG_REQUIRED, the capability will be * enabled iff the specified migration flag is enabled. On the other hand * QEMU_MIGRATION_FLAG_FORBIDDEN will enable the capability as long as * the specified migration flag is not enabled. */ @@ -215,6 +216,11 @@ static const qemuMigrationParamsFlagMapItem qemuMigrationParamsFlagMap[] = { .flag = VIR_MIGRATE_ZEROCOPY, .cap = QEMU_MIGRATION_CAP_ZERO_COPY_SEND, .party = QEMU_MIGRATION_SOURCE}, + + {.match = QEMU_MIGRATION_FLAG_FORBIDDEN, + .flag = VIR_MIGRATE_TUNNELLED, + .cap = QEMU_MIGRATION_CAP_SWITCHOVER_ACK, + .party = QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION}, }; /* Translation from VIR_MIGRATE_PARAM_* typed parameters to diff --git a/src/qemu/qemu_migration_params.h b/src/qemu/qemu_migration_params.h index 91bc6792cd..df67f1fb92 100644 --- a/src/qemu/qemu_migration_params.h +++ b/src/qemu/qemu_migration_params.h @@ -41,6 +41,7 @@ typedef enum { QEMU_MIGRATION_CAP_RETURN_PATH, QEMU_MIGRATION_CAP_ZERO_COPY_SEND, QEMU_MIGRATION_CAP_POSTCOPY_PREEMPT, + QEMU_MIGRATION_CAP_SWITCHOVER_ACK, QEMU_MIGRATION_CAP_LAST } qemuMigrationCapability; -- 2.43.0

4 8