April 2022 - Devel - libvirt List Archives

[libvirt RFC v3 00/19] multifd save restore prototype
by Claudio Fontana 28 Apr '22

28 Apr '22

This is the multifd save prototype in its first semi-functional state, now with both save and restore minimally functional. Still as mentioned before, likely there are quite a few rough edges, let me know what you think about this possible option. changes from v2 are many, mainly: * added ability to restore the VM from disk using multifd * fixed the multifd-helper to work in both directions, assuming the need to listen for save, and connect for restore. * fixed a large number of bugs, and probably introduced some :-) KNOWN ISSUES: 1) this applies only to virsh save and virsh restore for now (no managed save etc). 2) the .pl scripts to generate the headers for the new APIs do not reliably work for me, for the Restore case. I get: src/remote/remote_daemon_dispatch_stubs.h:10080:9: error: too few arguments to function ‘virDomainRestoreParametersFlags’ if (virDomainRestoreParametersFlags(params, nparams, args->flags) < 0) To work around this I had to fixup the header manually to look like: ...(conn, params, nparams, args->flags) < 0) Thanks for your thoughts, Claudio Claudio Fontana (19): iohelper: introduce new struct to carry copy operation parameters iohelper: refactor copy operation as a separate function libvirt: introduce virDomainSaveParametersFlags public API libvirt: introduce virDomainRestoreParametersFlags public API remote: Add RPC support for the virDomainSaveParametersFlags API remote: Add RPC support for the virDomainRestoreParametersFlags API qemu: add a stub for virDomainSaveParametersFlags API qemu: add a stub for virDomainRestoreParametersFlags API qemu: saveimage: introduce virQEMUSaveFd iohelper: move runIO function to a separate module runio: add arguments to extend use beyond just stdin and stdout multifd-helper: new helper for parallel save/restore qemu: wire up saveimage code with the multifd helper qemu: implement qemuMigrationSrcToFilesMultiFd qemu: add parameter to qemuMigrationDstRun to skip waiting qemu: implement qemuSaveImageLoadMultiFd tools: add parallel parameter to virsh save command tools: add parallel parameter to virsh restore command qemu: add migration parameter multifd-compression docs/manpages/virsh.rst | 34 ++- include/libvirt/libvirt-domain.h | 13 + src/driver-hypervisor.h | 14 + src/libvirt-domain.c | 99 +++++++ src/libvirt_private.syms | 1 + src/libvirt_public.syms | 6 + src/qemu/qemu_capabilities.c | 5 + src/qemu/qemu_capabilities.h | 3 + src/qemu/qemu_driver.c | 233 +++++++++++---- src/qemu/qemu_migration.c | 155 ++++++---- src/qemu/qemu_migration.h | 16 +- src/qemu/qemu_migration_params.c | 71 +++-- src/qemu/qemu_migration_params.h | 15 + src/qemu/qemu_process.c | 3 +- src/qemu/qemu_process.h | 5 +- src/qemu/qemu_saveimage.c | 480 ++++++++++++++++++++++++------- src/qemu/qemu_saveimage.h | 49 +++- src/qemu/qemu_snapshot.c | 6 +- src/remote/remote_driver.c | 2 + src/remote/remote_protocol.x | 29 +- src/remote_protocol-structs | 17 ++ src/util/iohelper.c | 150 +--------- src/util/meson.build | 15 + src/util/multifd-helper.c | 250 ++++++++++++++++ src/util/runio.c | 214 ++++++++++++++ src/util/runio.h | 38 +++ src/util/virthread.c | 5 + src/util/virthread.h | 1 + tools/virsh-domain.c | 96 +++++-- 29 files changed, 1599 insertions(+), 426 deletions(-) create mode 100644 src/util/multifd-helper.c create mode 100644 src/util/runio.c create mode 100644 src/util/runio.h -- 2.34.1

3 34

[PATCH] build-aux: remove duplicated syntax check filter for 'select'
by Daniel P. Berrangé 28 Apr '22

28 Apr '22

Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- build-aux/syntax-check.mk | 3 --- 1 file changed, 3 deletions(-) diff --git a/build-aux/syntax-check.mk b/build-aux/syntax-check.mk index 66ebc3e066..6664763faf 100644 --- a/build-aux/syntax-check.mk +++ b/build-aux/syntax-check.mk @@ -1621,9 +1621,6 @@ exclude_file_name_regexp--sc_prohibit_newline_at_end_of_diagnostic = \ exclude_file_name_regexp--sc_prohibit_nonreentrant = \ ^((po|tests|examples)/|docs/.*(py|js|html\.in|.rst)|run.in$$|tools/wireshark/util/genxdrstub\.pl|tools/virt-login-shell\.c$$) -exclude_file_name_regexp--sc_prohibit_select = \ - ^build-aux/syntax-check\.mk$$ - exclude_file_name_regexp--sc_prohibit_canonicalize_file_name = \ ^(build-aux/syntax-check\.mk|tests/virfilemock\.c)$$ -- 2.35.1

2 1

[libvirt RFC] add API for parallel Saves (not for committing)
by Claudio Fontana 28 Apr '22

28 Apr '22

RFC, starting point for discussion. Sketch API changes to allow parallel Saves, and open up and implementation for QEMU to leverage multifd migration to files, with optional multifd compression. This allows to improve save times for huge VMs. The idea is to issue commands like: virsh save domain /path/savevm --parallel --parallel-connections 2 and have libvirt start a multifd migration to: /path/savevm : main migration connection /path/savevm.1 : multifd channel 1 /path/savevm.2 : multifd channel 2 Signed-off-by: Claudio Fontana <cfontana(a)suse.de> --- include/libvirt/libvirt-domain.h | 5 +++++ src/driver-hypervisor.h | 7 +++++++ src/libvirt_public.syms | 5 +++++ src/qemu/qemu_driver.c | 1 + tools/virsh-domain.c | 8 ++++++++ 5 files changed, 26 insertions(+) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 2d5718301e..a7b9c4132d 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1270,6 +1270,7 @@ typedef enum { VIR_DOMAIN_SAVE_RUNNING = 1 << 1, /* Favor running over paused */ VIR_DOMAIN_SAVE_PAUSED = 1 << 2, /* Favor paused over running */ VIR_DOMAIN_SAVE_RESET_NVRAM = 1 << 3, /* Re-initialize NVRAM from template */ + VIR_DOMAIN_SAVE_PARALLEL = 1 << 4, /* Parallel Save/Restore to multiple files */ } virDomainSaveRestoreFlags; int virDomainSave (virDomainPtr domain, @@ -1278,6 +1279,10 @@ int virDomainSaveFlags (virDomainPtr domain, const char *to, const char *dxml, unsigned int flags); +int virDomainSaveParametersFlags (virDomainPtr domain, + virTypedParameterPtr params, + int nparams, + unsigned int flags); int virDomainRestore (virConnectPtr conn, const char *from); int virDomainRestoreFlags (virConnectPtr conn, diff --git a/src/driver-hypervisor.h b/src/driver-hypervisor.h index 4423eb0885..a4e1d21e76 100644 --- a/src/driver-hypervisor.h +++ b/src/driver-hypervisor.h @@ -240,6 +240,12 @@ typedef int const char *dxml, unsigned int flags); +typedef int +(*virDrvDomainSaveParametersFlags)(virDomainPtr domain, + virTypedParameterPtr params, + int nparams, + unsigned int flags); + typedef int (*virDrvDomainRestore)(virConnectPtr conn, const char *from); @@ -1489,6 +1495,7 @@ struct _virHypervisorDriver { virDrvDomainGetControlInfo domainGetControlInfo; virDrvDomainSave domainSave; virDrvDomainSaveFlags domainSaveFlags; + virDrvDomainSaveParametersFlags domainSaveParametersFlags; virDrvDomainRestore domainRestore; virDrvDomainRestoreFlags domainRestoreFlags; virDrvDomainSaveImageGetXMLDesc domainSaveImageGetXMLDesc; diff --git a/src/libvirt_public.syms b/src/libvirt_public.syms index f93692c427..eb3a7afb75 100644 --- a/src/libvirt_public.syms +++ b/src/libvirt_public.syms @@ -916,4 +916,9 @@ LIBVIRT_8.0.0 { virDomainSetLaunchSecurityState; } LIBVIRT_7.8.0; +LIBVIRT_8.3.0 { + global: + virDomainSaveParametersFlags; +} LIBVIRT_8.0.0; + # .... define new API here using predicted next version number .... diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 77012eb527..249105356c 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -20826,6 +20826,7 @@ static virHypervisorDriver qemuHypervisorDriver = { .domainGetControlInfo = qemuDomainGetControlInfo, /* 0.9.3 */ .domainSave = qemuDomainSave, /* 0.2.0 */ .domainSaveFlags = qemuDomainSaveFlags, /* 0.9.4 */ + .domainSaveParametersFlags = qemuDomainSaveParametersFlags, /* 8.3.0 */ .domainRestore = qemuDomainRestore, /* 0.2.0 */ .domainRestoreFlags = qemuDomainRestoreFlags, /* 0.9.4 */ .domainSaveImageGetXMLDesc = qemuDomainSaveImageGetXMLDesc, /* 0.9.4 */ diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index d5fd8be7c3..ccded6d265 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -4164,6 +4164,14 @@ static const vshCmdOptDef opts_save[] = { .type = VSH_OT_BOOL, .help = N_("avoid file system cache when saving") }, + {.name = "parallel", + .type = VSH_OT_BOOL, + .help = N_("enable parallel save to files") + }, + {.name = "parallel-connections", + .type = VSH_OT_INT, + .help = N_("number of connections/files for parallel save") + }, {.name = "xml", .type = VSH_OT_STRING, .completer = virshCompletePathLocalExisting, -- 2.34.1

4 12

[PATCH] NEWS: Mention bump of minimum qemu version to qemu-3.1
by Peter Krempa 27 Apr '22

27 Apr '22

Signed-off-by: Peter Krempa <pkrempa(a)redhat.com> --- NEWS.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/NEWS.rst b/NEWS.rst index 7ad8b9ec0b..c9b0ad42ed 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -15,6 +15,11 @@ v8.3.0 (unreleased) * **Removed features** + * qemu: Remove support for QEMU < 3.1 + + In accordance with our platform support policy, the oldest supported QEMU + version is now bumped from 2.11 to 3.1. + * **New features** * qemu: Introduce support for virtio-iommu -- 2.35.1

2 1

[PATCH] cpu: Remove pointless check
by Martin Kletzander 27 Apr '22

27 Apr '22

These two pointers can never be NULL since they are initialised to a reference of a struct. This became apparent when commit 210a19539447 added a VIR_DEBUG which used both pointers because due to the concise condition the compiler saw that if the "and" part of the condition did short-circuit (and it assumed that can happen) the second variable would not be initialised, but it is used in the debugging message, so the build failed with: In file included from ../src/cpu/cpu_x86.c:27: ../src/cpu/cpu_x86.c: In function ‘virCPUx86DataIsIdentical’: ../src/util/virlog.h:79:5: error: ‘bdata’ may be used uninitialized in this function [-Werror=maybe-uninitialized] Fix this by just assigning the helper pointers and remove the condition altogether. Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- Pushed under the build-breaker rule, but feel free to review/object. src/cpu/cpu_x86.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/cpu/cpu_x86.c b/src/cpu/cpu_x86.c index 7e9d1cea47d1..a5eac7601cad 100644 --- a/src/cpu/cpu_x86.c +++ b/src/cpu/cpu_x86.c @@ -3314,10 +3314,8 @@ virCPUx86DataIsIdentical(const virCPUData *a, return VIR_CPU_COMPARE_INCOMPATIBLE; } - if (!((adata = &a->data.x86) && (bdata = &b->data.x86))) { - VIR_DEBUG("missing x86 data: a:%p b:%p", adata, bdata); - return VIR_CPU_COMPARE_ERROR; - } + adata = &a->data.x86; + bdata = &b->data.x86; if (adata->len != bdata->len) { VIR_DEBUG("unequal length a:%zu b:%zu", adata->len, bdata->len); -- 2.35.1

2 2

[libvirt PATCH] util: fix double destroy / missing unref of GSource
by Daniel P. Berrangé 27 Apr '22

27 Apr '22

A leak of the GSource was introduced in commit 87a43a907f0ad4897a28ad7c216bc70f37270b93 Author: Michal Prívozník <mprivozn(a)redhat.com> Date: Fri Jan 28 18:42:45 2022 +0100 lib: Use g_clear_pointer() more As it mistakenly replaced the g_vir_source_unref call with a second call to g_source_destroy. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- src/util/vireventglib.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/util/vireventglib.c b/src/util/vireventglib.c index fc04d8f712..3cc42597fd 100644 --- a/src/util/vireventglib.c +++ b/src/util/vireventglib.c @@ -227,7 +227,7 @@ virEventGLibHandleUpdate(int watch, goto cleanup; VIR_DEBUG("Removed old handle source=%p", data->source); - g_source_destroy(data->source); + vir_g_source_unref(data->source, NULL); g_clear_pointer(&data->source, g_source_destroy); data->events = 0; } -- 2.35.1

2 1

[PATCH] qemu: do not use domain virt type to get default version
by Liang Yan 26 Apr '22

26 Apr '22

We do not need VIR_DOMAIN_VIRT_QEMU to get qemu default version. With the 'os_type' and 'arch'in capabilities, we could identify 'emulator' which is enough to get the version. Actually VIR_DOMAIN_VIRT_QEMU is not the only domain virt type for qemu driver, there are VIR_DOMAIN_VIRT_KVM and VIR_DOMAIN_VIRT_HVF. If TCG is disabled in qemu, it will cause the error that could not find suitable emulater when access version. Signed-off-by: Liang Yan <lyan(a)digtalocean.com> --- src/qemu/qemu_capabilities.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index b91db851bb..9a0b7ebeb4 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -1857,7 +1857,7 @@ int virQEMUCapsGetDefaultVersion(virCaps *caps, hostarch = virArchFromHost(); if (!(capsdata = virCapabilitiesDomainDataLookup(caps, - VIR_DOMAIN_OSTYPE_HVM, hostarch, VIR_DOMAIN_VIRT_QEMU, + VIR_DOMAIN_OSTYPE_HVM, hostarch, VIR_DOMAIN_VIRT_NONE, NULL, NULL))) { virReportError(VIR_ERR_INTERNAL_ERROR, _("Cannot find suitable emulator for %s"), -- 2.34.1

1 0

[PULL 0/3] Misc next patches
by Daniel P. Berrangé 26 Apr '22

26 Apr '22

The following changes since commit a1755db71e34df016ffc10aa0727360aae2c6036: Merge tag 'pull-block-2022-04-25' of https://gitlab.com/hreitz/qemu into staging (2022-04-25 13:35:41 -0700) are available in the Git repository at: https://gitlab.com/berrange/qemu tags/misc-next-pull-request for you to fetch changes up to 5cf434b5af386fadc3418df71d3738676cbb0549: github: fix config mistake preventing repo lockdown commenting (2022-04-26 16:12:26 +0100) ---------------------------------------------------------------- Misc patch queue * Removes depecated --enable-fips QEMU system emulator option * Fixes array bounds check in keycode conversion for ESCC device ---------------------------------------------------------------- Daniel P. Berrangé (3): softmmu: remove deprecated --enable-fips option hw/char: fix qcode array bounds check in ESCC impl github: fix config mistake preventing repo lockdown commenting .github/workflows/lockdown.yml | 6 +++--- docs/about/deprecated.rst | 12 ------------ docs/about/removed-features.rst | 11 +++++++++++ hw/char/escc.c | 2 +- include/qemu/osdep.h | 3 --- os-posix.c | 8 -------- qemu-options.hx | 10 ---------- ui/vnc.c | 7 ------- util/osdep.c | 28 ---------------------------- 9 files changed, 15 insertions(+), 72 deletions(-) -- 2.35.1

2 4

[PATCH] include: Move version information
by Andrea Bolognani 26 Apr '22

26 Apr '22

It belongs to the comment for VIR_DOMAIN_AFFECT_CONFIG, not to the unrelated one immediately after it. Fixes: 807cdbf75976f88e8a23b26951f263029cc9e5f4 Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- Pushed as trivial. include/libvirt/libvirt-domain.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 6fed42c47a..9386f5189c 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -323,8 +323,8 @@ typedef virDomainControlInfo *virDomainControlInfoPtr; typedef enum { VIR_DOMAIN_AFFECT_CURRENT = 0, /* Affect current domain state. (Since: v0.9.2) */ VIR_DOMAIN_AFFECT_LIVE = 1 << 0, /* Affect running domain state. (Since: v0.9.2) */ - VIR_DOMAIN_AFFECT_CONFIG = 1 << 1, /* Affect persistent domain state. */ - /* 1 << 2 is reserved for virTypedParameterFlags (Since: v0.9.2) */ + VIR_DOMAIN_AFFECT_CONFIG = 1 << 1, /* Affect persistent domain state. (Since: v0.9.2) */ + /* 1 << 2 is reserved for virTypedParameterFlags */ } virDomainModificationImpact; /** -- 2.35.1

1 0

[PATCH v2 0/8] Introduce network backed NVRAM
by Rohit Kumar 26 Apr '22

26 Apr '22

Libvirt domain XML currently allows only local filepaths that can be used to specify a NVRAM disk. Since, VMs can migrate across hypervisor hosts, it should be possible to allocate NVRAM disks on network storage for uninterrupted access. This series extends the NVRAM element to support hosting over network-backed disks, for high availability. It achieves this by embedding virStorageSource pointer for nvram into _virDomainLoaderDef. It introduces a 'type' attribute for NVRAM element to specify 'file' vs 'network' backed NVRAM. Changes v1->v2: - Split the patch into smaller patches - Added unit test - Updated the doc - Addressed Peter's comment on v1 (https://listman.redhat.com/archives/libvir-list/2022-March/229684.html) Rohit Kumar (8): Make NVRAM a virStorageSource type. Add support to parse/format virStorageSource type NVRAM Validate remote store NVRAM Cleanup diskSourceNetwork and diskSourceFile schema Update XML schema to support network backed NVRAM Update NVRAM documentation Add unit test for network backed NVRAM Add unit test to support new 'file' type NVRAM docs/formatdomain.rst | 43 +++++++-- src/conf/domain_conf.c | 88 ++++++++++++++++--- src/conf/domain_conf.h | 2 +- src/conf/schemas/domaincommon.rng | 80 +++++++++++------ src/qemu/qemu_cgroup.c | 3 +- src/qemu/qemu_command.c | 2 +- src/qemu/qemu_domain.c | 14 +-- src/qemu/qemu_driver.c | 5 +- src/qemu/qemu_firmware.c | 23 +++-- src/qemu/qemu_namespace.c | 5 +- src/qemu/qemu_process.c | 5 +- src/qemu/qemu_validate.c | 22 +++++ src/security/security_dac.c | 6 +- src/security/security_selinux.c | 6 +- src/security/virt-aa-helper.c | 5 +- src/vbox/vbox_common.c | 2 +- .../bios-nvram-file.x86_64-latest.args | 37 ++++++++ tests/qemuxml2argvdata/bios-nvram-file.xml | 23 +++++ .../bios-nvram-network.x86_64-latest.args | 37 ++++++++ tests/qemuxml2argvdata/bios-nvram-network.xml | 25 ++++++ tests/qemuxml2argvtest.c | 2 + 21 files changed, 360 insertions(+), 75 deletions(-) create mode 100644 tests/qemuxml2argvdata/bios-nvram-file.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/bios-nvram-file.xml create mode 100644 tests/qemuxml2argvdata/bios-nvram-network.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/bios-nvram-network.xml -- 2.25.1

3 34