July 2023 - Devel - libvirt List Archives

[PULL 00/25] Migration 20230726 patches
by Juan Quintela 27 Jul '23

27 Jul '23

The following changes since commit 6cb2011fedf8c4e7b66b4a3abd6b42c1bae99ce6: Update version for v8.1.0-rc1 release (2023-07-25 20:09:05 +0100) are available in the Git repository at: https://gitlab.com/juan.quintela/qemu.git tags/migration-20230726-pull-request for you to fetch changes up to 697c4c86ab515a728ffb2adc2c3c04b22fa9210f: migration/rdma: Split qemu_fopen_rdma() into input/output functions (2023-07-26 10:55:56 +0200) ---------------------------------------------------------------- Migration Pull request Hi This is the migration PULL request. It is the same than yesterday with proper PULL headers. It pass CI. It contains: - Fabiano rosas trheadinfo cleanups - Hyman Huang dirtylimit changes - Part of my changes - Peter Xu documentation - Tejus updato to migration descriptions - Wei want improvements for postocpy and multifd setup Please apply. Thanks, Juan. ---------------------------------------------------------------- Fabiano Rosas (2): migration/multifd: Rename threadinfo.c functions migration/multifd: Protect accesses to migration_threads Hyman Huang(黄勇) (8): softmmu/dirtylimit: Add parameter check for hmp "set_vcpu_dirty_limit" qapi/migration: Introduce x-vcpu-dirty-limit-period parameter qapi/migration: Introduce vcpu-dirty-limit parameters migration: Introduce dirty-limit capability migration: Refactor auto-converge capability logic migration: Put the detection logic before auto-converge checking migration: Implement dirty-limit convergence algo migration: Extend query-migrate to provide dirty page limit info Juan Quintela (11): migration-test: Be consistent for ppc migration-test: Make machine_opts regular with other options migration-test: Create arch_opts migration-test: machine_opts is really arch specific migration: skipped field is really obsolete. qemu-file: Rename qemu_file_transferred_ fast -> noflush migration: Change qemu_file_transferred to noflush qemu_file: Make qemu_file_is_writable() static qemu-file: Simplify qemu_file_shutdown() qemu-file: Make qemu_file_get_error_obj() static migration/rdma: Split qemu_fopen_rdma() into input/output functions Peter Xu (1): docs/migration: Update postcopy bits Tejus GK (1): migration: Update error description whenever migration fails Wei Wang (2): migration: enforce multifd and postcopy preempt to be set before incoming qtest/migration-tests.c: use "-incoming defer" for postcopy tests docs/about/deprecated.rst | 10 ++++ docs/devel/migration.rst | 94 ++++++++++++++++++++++---------- qapi/migration.json | 87 ++++++++++++++++++++++++++---- include/sysemu/dirtylimit.h | 2 + migration/options.h | 1 + migration/qemu-file.h | 14 ++--- migration/threadinfo.h | 7 +-- migration/migration-hmp-cmds.c | 26 +++++++++ migration/migration.c | 36 +++++++++---- migration/multifd.c | 4 +- migration/options.c | 87 +++++++++++++++++++++++++++++- migration/qemu-file.c | 24 ++------- migration/ram.c | 59 +++++++++++++++++---- migration/rdma.c | 39 +++++++------- migration/savevm.c | 6 +-- migration/threadinfo.c | 19 +++++-- migration/vmstate.c | 4 +- softmmu/dirtylimit.c | 97 ++++++++++++++++++++++++++++++---- tests/qtest/migration-test.c | 48 ++++++++--------- migration/trace-events | 1 + 20 files changed, 510 insertions(+), 155 deletions(-) -- 2.40.1

2 26

[PATCH] remote: handle partial data transmission
by Oleg Vasilev 26 Jul '23

26 Jul '23

A new bug was introduced as a part of use-after-free fix below: commit 411cbe7199ce533ae5fa78f5558dddca6f88ef1a Author: Oleg Vasilev <oleg.vasilev(a)virtuozzo.com> Date: Tue Jul 4 13:10:22 2023 +0600 remote: fix stream use-after-free When the message was processed partially, it is actually supposed to stay in the queue to be processed again. In such case, reinsert it back. Signed-off-by: Oleg Vasilev <oleg.vasilev(a)virtuozzo.com> --- src/remote/remote_daemon_stream.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/remote/remote_daemon_stream.c b/src/remote/remote_daemon_stream.c index 345c40b48c..f52af790c1 100644 --- a/src/remote/remote_daemon_stream.c +++ b/src/remote/remote_daemon_stream.c @@ -775,8 +775,12 @@ daemonStreamHandleWrite(virNetServerClient *client, ret = -1; } - if (ret > 0) - break; /* still processing data from msg */ + if (ret > 0) { + /* still processing data from msg, put it back into queue */ + msg->next = stream->rx; + stream->rx = msg; + break; + } if (ret < 0) { virNetMessageFree(msg); -- 2.41.0

2 1

[PATCH] docs: formatdomain: Mention the QEMU requirement for discard_no_unref
by Han Han 26 Jul '23

26 Jul '23

Signed-off-by: Han Han <hhan(a)redhat.com> --- docs/formatdomain.rst | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index 973de8dd4f..cd9cb02bf8 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst @@ -3296,7 +3296,8 @@ paravirtualized driver is specified via the ``disk`` element. image. When enabled, a discard request from within the guest will mark the qcow2 cluster as zero, but will keep the reference/offset of that cluster. But it will still pass the discard further to the lower layer. - This will resolve fragmentation within the qcow2 image. :since:`Since 9.5.0` + This will resolve fragmentation within the qcow2 image. :since:`Since 9.5.0 + (QEMU 8.1)` In the majority of cases the default configuration used by the hypervisor is sufficient so modifying this setting should not be necessary. For -- 2.41.0

2 2

[libvirt PATCH v2] run: add ability to set selinux context
by Jonathon Jongsma 25 Jul '23

25 Jul '23

When running libvirt from the build directory with the 'run' script, it will run as unconfined_t. This can result in unexpected behavior when selinux is enforcing due to the fact that the selinux policies are written assuming that libvirt is running with the system_u:system_r:virtd_t context. This patch adds a new --selinux option to the run script. When this option is specified, it will launch the specified binary using the 'runcon' utility to set its selinux context to the one mentioned above. Since this may require root privileges, setting the selinux context is not the default behavior and must be enabled with the command line switch. Note that this uses argparse to parse the new option, but it does so in a very limited way in order to avoid interfering with any arguments that the user might want to pass on to libvirt. For example, we do not provide a `--help` option for the run script. Signed-off-by: Jonathon Jongsma <jjongsma(a)redhat.com> --- I sent a version of this a couple months ago and Erik made a few comments and then I kind of forgot about it for a little while. Re-sending now with a couple minor changes changes in v2: - rebased to master - fixed to properly restore context of binary after execution if the systemd unit file is not currently running. - disabled abbreviated versions of options run.in | 100 +++++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 80 insertions(+), 20 deletions(-) diff --git a/run.in b/run.in index c6d3411082..869f074d71 100644 --- a/run.in +++ b/run.in @@ -40,9 +40,11 @@ # # ---------------------------------------------------------------------- +import argparse import os import os.path import random +import shutil import signal import sys import subprocess @@ -59,15 +61,20 @@ def prepend(env, varname, extradir): here = "@abs_builddir@" -if len(sys.argv) < 2: - print("syntax: %s BINARY [ARGS...]" % sys.argv[0], file=sys.stderr) +parser = argparse.ArgumentParser(add_help=False, allow_abbrev=False) +parser.add_argument('--selinux', + action='store_true', + help='Run in the appropriate selinux context') + +opts, args = parser.parse_known_args() + +if len(args) < 1: + print("syntax: %s [--selinux] BINARY [ARGS...]" % sys.argv[0], file=sys.stderr) sys.exit(1) -prog = sys.argv[1] -args = sys.argv[1:] +prog = args[0] env = os.environ - prepend(env, "LD_LIBRARY_PATH", os.path.join(here, "src")) prepend(env, "PKG_CONFIG_PATH", os.path.join(here, "src")) prepend(env, "PATH", os.path.join(here, "tools")) @@ -130,10 +137,25 @@ def change_unit(name, action): return ret == 0 +def chcon(path, user, role, type): + print("Setting file context of {} to u={}, r={}, t={}...".format(progpath, + user, + role, + type)) + ret = subprocess.call(["chcon", "-u", user, "-r", role, "-t", type, path]) + return ret == 0 + + +def restorecon(path): + print("Restoring selinux context for {}...".format(path)) + ret = subprocess.call(["restorecon", path]) + return ret == 0 + + try_stop_units = [] if is_systemd_host(): maybe_stopped_units = [] - for arg in sys.argv: + for arg in args: name = os.path.basename(arg) if is_modular_daemon(name): # Only need to stop libvirtd or this specific modular unit @@ -149,11 +171,10 @@ if is_systemd_host(): if is_unit_active(unit): try_stop_units.append(unit) -if len(try_stop_units) == 0: +if len(try_stop_units) == 0 and not opts.selinux: # Run the program directly, replacing ourselves os.execvpe(prog, args, env) else: - print("Temporarily stopping systemd units...") stopped_units = [] def sighandler(signum, frame): @@ -164,12 +185,48 @@ else: signal.signal(signal.SIGQUIT, sighandler) try: - for unit in try_stop_units: - print(" > %s" % unit) - if not change_unit(unit, "stop"): - raise Exception("Unable to stop '%s'" % unit) - - stopped_units.append(unit) + dorestorecon = False + progpath = shutil.which(prog) + if len(try_stop_units): + print("Temporarily stopping systemd units...") + + for unit in try_stop_units: + print(" > %s" % unit) + if not change_unit(unit, "stop"): + raise Exception("Unable to stop '%s'" % unit) + + stopped_units.append(unit) + + if opts.selinux: + # if using a wrapper command like 'gdb', setting the selinux + # context won't work because the wrapper command will not be a + # valid entrypoint for the virtd_t context + if os.path.basename(prog) not in ["libvirtd", *modular_daemons]: + raise Exception("'{}' is not recognized as a valid daemon. " + "Selinux process context can only be set when " + "executing a daemon directly without wrapper " + "commands".format(prog)) + + if not progpath: + raise Exception("Can't find executable {} for selinux labeling" + .format(prog)) + + if not progpath.startswith(os.path.abspath(here)): + raise Exception("Refusing to change selinux context of file " + "'{}' outside build directory" + .format(progpath)) + + # selinux won't allow us to transition to the virtd_t context from + # e.g. the user_home_t context (the likely label of the local + # executable file) + if not chcon(progpath, "system_u", "object_r", "virtd_exec_t"): + raise Exception("Failed to change selinux context of binary") + dorestorecon = True + + args = ['runcon', + '-u', 'system_u', + '-r', 'system_r', + '-t', 'virtd_t', *args] print("Running '%s'..." % str(" ".join(args))) ret = subprocess.call(args, env=env) @@ -178,9 +235,12 @@ else: except Exception as e: print("%s" % e, file=sys.stderr) finally: - print("Re-starting original systemd units...") - stopped_units.reverse() - for unit in stopped_units: - print(" > %s" % unit) - if not change_unit(unit, "start"): - print(" ! unable to restart %s" % unit, file=sys.stderr) + if len(stopped_units): + print("Re-starting original systemd units...") + stopped_units.reverse() + for unit in stopped_units: + print(" > %s" % unit) + if not change_unit(unit, "start"): + print(" ! unable to restart %s" % unit, file=sys.stderr) + if dorestorecon: + restorecon(progpath) -- 2.40.1

2 1

Re: [libvirt PATCH v2 05/24] snapshot_conf: add new argument to virDomainSnapshotAlignDisks
by Peter Krempa 25 Jul '23

25 Jul '23

Preferrably: virDomainSnapshotAlignDisks: Allow overriding user-configured snapshot default Reply-To: In-Reply-To: <322f30a38fa3a4e7a6ef9b1161a07da4b74f0d6e.1687877087.git.phrdina(a)redhat.com> On Tue, Jun 27, 2023 at 17:07:08 +0200, Pavel Hrdina wrote: > This new option will be used by external snapshot revert code. > > Signed-off-by: Pavel Hrdina <phrdina(a)redhat.com> > --- > src/conf/snapshot_conf.c | 15 ++++++++++++--- > src/conf/snapshot_conf.h | 3 ++- > src/qemu/qemu_snapshot.c | 2 +- > src/test/test_driver.c | 2 +- > 4 files changed, 16 insertions(+), 6 deletions(-) Reviewed-by: Peter Krempa <pkrempa(a)redhat.com>

1 1

[libvirt PATCH 0/5] Simplify x86 cpu feature synchronization with qemu
by Tim Wiederhake 25 Jul '23

25 Jul '23

A recurring task in libvirt is synchronizing the list of (x86) cpu features with qemu. This is an error-prone and manual task as it relies on parsing qemu c source code and therefore neccessarily lags behind qemu development. The better solution would be if qemu and libvirt both got the cpu feature information from a common source. libcpuinfo [1] wants to be that source, a comprehensive, architecture agnostic database of cpu information, together with some handy tooling, e.g. a library to access this database at run-time, quering the host for capabilities, calculate baselines for different sets of capabilities. libcpuinfo comes with c headers and contains python bindings. This patch series prepares libvirt's x86 feature list to be automatically generated from libcpuinfo data, but introduces no dependency on libcpuinfo yet. Introducing that dependency to generate the feature list on compile time or making use of libcpuinfo's more advanced features would be up to discussion at a later date. Note that libcpuinfo's feature database is not on par with libvirt yet and to generate an identical version of x86_features.xml the data from libcpuinfo's "wip" branch is required. [1] https://gitlab.com/twiederh/libcpuinfo Tim Wiederhake (5): cpu_map: Sort cpu features cpu_map: Format register values uniformly cpu_map: Format comments cpu_map: Remove unused alias information cpu_map: Generate x86 feature map from libcpuinfo src/cpu_map/x86_features.py | 119 +++++++ src/cpu_map/x86_features.xml | 594 +++++++++++++++++------------------ 2 files changed, 407 insertions(+), 306 deletions(-) create mode 100755 src/cpu_map/x86_features.py -- 2.39.2

2 6

[PATCH] qemu_domain: Partially validate memory amounts when auto-adding NUMA node
by Michal Privoznik 25 Jul '23

25 Jul '23

When automatically adding a NUMA node (qemuDomainDefNumaAutoAdd()) the memory size of the node is computed as: total_memory - sum(memory devices) And we have a nice helper for that: virDomainDefGetMemoryInitial() so it looks logical to just call it. Except, this code runs in post parse callback, i.e. memory sizes were not validated and it may happen that the sum is greater than the total memory. This would be caught by virDomainDefPostParseMemory() but that runs only after driver specific callbacks (i.e. after qemuDomainDefNumaAutoAdd()) and because the domain config was changed and memory was increased to this huge number no error is caught. So let's do what virDomainDefGetMemoryInitial() would do, but with error checking. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=2216236 Fixes: f5d4f5c8ee44e9f1939070afcc5381bdd5545e50 Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- src/qemu/qemu_domain.c | 13 +++++++++++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 6eea8a9fa5..fdda001795 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -4821,17 +4821,24 @@ qemuDomainDefNumaAutoAdd(virDomainDef *def, return 0; } - initialMem = virDomainDefGetMemoryInitial(def); + initialMem = virDomainDefGetMemoryTotal(def); if (!def->numa) def->numa = virDomainNumaNew(); virDomainNumaSetNodeCount(def->numa, 1); - virDomainNumaSetNodeMemorySize(def->numa, 0, initialMem); for (i = 0; i < def->nmems; i++) { virDomainMemoryDef *mem = def->mems[i]; + if (mem->size > initialMem) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("Total size of memory devices exceeds the total memory size")); + return -1; + } + + initialMem -= mem->size; + switch (mem->model) { case VIR_DOMAIN_MEMORY_MODEL_DIMM: case VIR_DOMAIN_MEMORY_MODEL_NVDIMM: @@ -4848,6 +4855,8 @@ qemuDomainDefNumaAutoAdd(virDomainDef *def, } } + virDomainNumaSetNodeMemorySize(def->numa, 0, initialMem); + return 0; } -- 2.41.0

2 1

[PATCH] util: file: Mark 'BeeGFS' as shared filesystem
by Peter Krempa 25 Jul '23

25 Jul '23

BeeGFS is a shared/distributed filesystem: https://doc.beegfs.io/latest/overview/overview.html Mark it as shared based on it's magic number: https://git.beegfs.io/pub/v7/-/blob/master/client_module/source/filesystem/… Closes: https://gitlab.com/libvirt/libvirt/-/issues/508 Signed-off-by: Peter Krempa <pkrempa(a)redhat.com> --- src/util/virfile.c | 6 +++++- src/util/virfile.h | 1 + 2 files changed, 6 insertions(+), 1 deletion(-) diff --git a/src/util/virfile.c b/src/util/virfile.c index 7c44a2a963..2984e2ead2 100644 --- a/src/util/virfile.c +++ b/src/util/virfile.c @@ -3382,6 +3382,8 @@ virFileRemoveLastComponent(char *path) # endif # define VIR_ACFS_MAGIC 0x61636673 +/* https://git.beegfs.io/pub/v7/-/blob/master/client_module/source/filesystem/… */ +# define VIR_BEEGFS_MAGIC 0x19830326 /* formerly fhgfs */ # define PROC_MOUNTS "/proc/mounts" @@ -3469,6 +3471,7 @@ static const struct virFileSharedFsData virFileSharedFs[] = { { .fstype = VIR_FILE_SHFS_CEPH, .magic = CEPH_SUPER_MAGIC }, { .fstype = VIR_FILE_SHFS_GPFS, .magic = GPFS_SUPER_MAGIC }, { .fstype = VIR_FILE_SHFS_ACFS, .magic = VIR_ACFS_MAGIC }, + { .fstype = VIR_FILE_SHFS_BEEGFS, .magic = VIR_BEEGFS_MAGIC }, }; @@ -3719,7 +3722,8 @@ int virFileIsSharedFS(const char *path) VIR_FILE_SHFS_GPFS| VIR_FILE_SHFS_QB | VIR_FILE_SHFS_ACFS | - VIR_FILE_SHFS_GLUSTERFS); + VIR_FILE_SHFS_GLUSTERFS | + VIR_FILE_SHFS_BEEGFS); } diff --git a/src/util/virfile.h b/src/util/virfile.h index b75a7cc53b..60bb1d64e7 100644 --- a/src/util/virfile.h +++ b/src/util/virfile.h @@ -226,6 +226,7 @@ enum { VIR_FILE_SHFS_QB = (1 << 8), /* Quobyte shared filesystem */ VIR_FILE_SHFS_ACFS = (1 << 9), /* Oracle ASM Cluster File System */ VIR_FILE_SHFS_GLUSTERFS = (1 << 10), /* gluster's FUSE-based client */ + VIR_FILE_SHFS_BEEGFS = (1 << 11), /* BeeGFS/fhGFS */ }; int virFileIsSharedFSType(const char *path, unsigned int fstypes) ATTRIBUTE_NONNULL(1); -- 2.41.0

2 1

[PATCH] ci: Switch to OpenSUSE Leap 15.5
by Michal Privoznik 25 Jul '23

25 Jul '23

OpenSUSE Leap was released recently (2023-06-07). Refresh our CI with latest lcitool which brings this minor update. Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- Green pipeline: https://gitlab.com/MichalPrivoznik/libvirt/-/pipelines/944440144 ci/containers/opensuse-leap-15.Dockerfile | 2 +- ci/gitlab/builds.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/ci/containers/opensuse-leap-15.Dockerfile b/ci/containers/opensuse-leap-15.Dockerfile index ca11edf808..dfb5b27bbb 100644 --- a/ci/containers/opensuse-leap-15.Dockerfile +++ b/ci/containers/opensuse-leap-15.Dockerfile @@ -4,7 +4,7 @@ # # https://gitlab.com/libvirt/libvirt-ci -FROM registry.opensuse.org/opensuse/leap:15.4 +FROM registry.opensuse.org/opensuse/leap:15.5 RUN zypper update -y && \ zypper install -y \ diff --git a/ci/gitlab/builds.yml b/ci/gitlab/builds.yml index 8910a19c79..f4e67b363e 100644 --- a/ci/gitlab/builds.yml +++ b/ci/gitlab/builds.yml @@ -300,7 +300,7 @@ x86_64-opensuse-leap-15-local-env: needs: [] allow_failure: false variables: - IMAGE: registry.opensuse.org/opensuse/leap:15.4 + IMAGE: registry.opensuse.org/opensuse/leap:15.5 NAME: opensuse-leap-15 RPM: skip -- 2.41.0

2 1

[PATCH v2] docs: Mention vhostuser for queues and queue_size
by Han Han 25 Jul '23

25 Jul '23

These two attributes are supported for vhost-user-blk as well. Signed-off-by: Han Han <hhan(a)redhat.com> --- Update the supported version of the queues attribute of vhost-user-blk v1: https://listman.redhat.com/archives/libvir-list/2023-July/240836.html docs/formatdomain.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index 4af0b82569..973de8dd4f 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst @@ -3275,9 +3275,10 @@ paravirtualized driver is specified via the ``disk`` element. "virtio" ``bus`` and "pci" or "ccw" ``address`` types. :since:`Since 1.2.8 (QEMU 2.1)` - The optional ``queues`` attribute specifies the number of virt queues for - virtio-blk. ( :since:`Since 3.9.0` ) + virtio-blk ( :since:`Since 3.9.0` ) or vhost-user-blk + ( :since `Since 7.1.0` ) - The optional ``queue_size`` attribute specifies the size of each virt - queue for virtio-blk. ( :since:`Since 7.8.0` ) + queue for virtio-blk or vhost-user-blk. ( :since:`Since 7.8.0` ) - For virtio disks, `Virtio-related options`_ can also be set. ( :since:`Since 3.5.0` ) - The optional ``metadata_cache`` subelement controls aspects related to the -- 2.41.0

2 1