November 2023 - Devel - libvirt List Archives

[PATCH 0/4] Support for dirty-limit live migration
by Hyman Huang 21 Dec '23

21 Dec '23

v2: - mark the VIR_MIGRATE_DIRTY_LIMIT flag since 9.10.0 v1: The dirty-limit functionality for live migration was introduced since qemu>=8.1. In the live migration scenario, it implements the force convergence using the dirty-limit approach, which results in better reliable read performance. A straightforward dirty-limit capability for live migration is added by this patchset. Users might not care about other dirty-limit arguments like "x-vcpu-dirty-limit-period" or "vcpu-dirty-limit," thus do not expose them to Libvirt and Keep the default configurations and values in place. For more details about dirty-limit, please see the following reference: https://lore.kernel.org/qemu- devel/169024923116.19090.10825599068950039132-0(a)git.sr.ht/ Hyman Huang (4): Add VIR_MIGRATE_DIRTY_LIMIT flag qemu_migration: Implement VIR_MIGRATE_DIRTY_LIMIT flag virsh: Add support for VIR_MIGRATE_DIRTY_LIMIT flag NEWS: document support for dirty-limit live migration NEWS.rst | 8 ++++++++ docs/manpages/virsh.rst | 10 +++++++++- include/libvirt/libvirt-domain.h | 5 +++++ src/libvirt-domain.c | 8 ++++++++ src/qemu/qemu_migration.c | 8 ++++++++ src/qemu/qemu_migration.h | 1 + src/qemu/qemu_migration_params.c | 6 ++++++ src/qemu/qemu_migration_params.h | 1 + tools/virsh-domain.c | 10 ++++++++++ 9 files changed, 56 insertions(+), 1 deletion(-) -- 2.39.1

2 9

[PATCH v1] qemu_driver: Don't handle the EOF event if monitor changed
by tugy＠chinatelecom.cn 07 Dec '23

07 Dec '23

From: Guoyi Tu <tugy(a)chinatelecom.cn> Currently, libvirt creates a thread pool with only on thread to handle all qemu monitor events for virtual machines, In the cases that if the thread gets stuck while handling a monitor EOF event, such as unable to kill the virtual machine process or release resources, the events of other virtual machine will be also blocked, which will lead to the abnormal behavior of other virtual machines. For instance, when another virtual machine completes a shutdown operation and the monitor EOF event has been queued but remains unprocessed, we immediately destroy and start the virtual machine again, at a later time when EOF event get processed, the processMonitorEOFEvent() will kill the virtual machine that just started. To address this issue, in the processMonitorEOFEvent(), we check whether the current virtual machine's monitor object matches the one at the time the event was generated. If they do not match, we immediately return. Signed-off-by: Guoyi Tu <tugy(a)chinatelecom.cn> Signed-off-by: dengpengcheng <dengpc12(a)chinatelecom.cn> --- src/qemu/qemu_domain.c | 2 +- src/qemu/qemu_driver.c | 11 +++++++++-- src/qemu/qemu_process.c | 2 +- 3 files changed, 11 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 953808fcfe..435ee621df 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -11470,7 +11470,6 @@ qemuProcessEventFree(struct qemuProcessEvent *event) case QEMU_PROCESS_EVENT_NETDEV_STREAM_DISCONNECTED: case QEMU_PROCESS_EVENT_NIC_RX_FILTER_CHANGED: case QEMU_PROCESS_EVENT_SERIAL_CHANGED: - case QEMU_PROCESS_EVENT_MONITOR_EOF: case QEMU_PROCESS_EVENT_GUEST_CRASHLOADED: g_free(event->data); break; @@ -11484,6 +11483,7 @@ qemuProcessEventFree(struct qemuProcessEvent *event) case QEMU_PROCESS_EVENT_UNATTENDED_MIGRATION: case QEMU_PROCESS_EVENT_RESET: case QEMU_PROCESS_EVENT_NBDKIT_EXITED: + case QEMU_PROCESS_EVENT_MONITOR_EOF: case QEMU_PROCESS_EVENT_LAST: break; } diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index d00d2a27c6..57acafb48b 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -3854,7 +3854,8 @@ processJobStatusChangeEvent(virDomainObj *vm, static void processMonitorEOFEvent(virQEMUDriver *driver, - virDomainObj *vm) + virDomainObj *vm, + qemuMonitor *mon) { qemuDomainObjPrivate *priv = vm->privateData; int eventReason = VIR_DOMAIN_EVENT_STOPPED_SHUTDOWN; @@ -3863,6 +3864,12 @@ processMonitorEOFEvent(virQEMUDriver *driver, unsigned int stopFlags = 0; virObjectEvent *event = NULL; + if (priv->mon != mon) { + VIR_DEBUG("Domain %p '%s' has been shutdown, ignoring EOF", + vm, vm->def->name); + return; + } + if (qemuProcessBeginStopJob(vm, VIR_JOB_DESTROY, true) < 0) return; @@ -4082,7 +4089,7 @@ static void qemuProcessEventHandler(void *data, void *opaque) processJobStatusChangeEvent(vm, processEvent->data); break; case QEMU_PROCESS_EVENT_MONITOR_EOF: - processMonitorEOFEvent(driver, vm); + processMonitorEOFEvent(driver, vm, processEvent->data); break; case QEMU_PROCESS_EVENT_PR_DISCONNECT: processPRDisconnectEvent(vm); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index f32e82bbd1..ff54e947fe 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -316,7 +316,7 @@ qemuProcessHandleMonitorEOF(qemuMonitor *mon, } qemuProcessEventSubmit(vm, QEMU_PROCESS_EVENT_MONITOR_EOF, - 0, 0, NULL); + 0, 0, priv->mon); /* We don't want this EOF handler to be called over and over while the * thread is waiting for a job. -- 2.17.1

3 3

[libvirt PATCH] test: remove redundant cpuTestGuestCPUID test
by Jonathon Jongsma 06 Dec '23

06 Dec '23

DO_TEST_CPUID(arch, host, json) is a multipart test. It consists of the following tests: - cpuTestHostCPUID() - cpuTestGuestCPUID(with JSON_* flag) - cpuTestCPUIDSignature() - DO_TEST_JSON(): - if json==JSON_MODELS: - cpuTestGuestCPUID(without JSON_* flag) - cpuTestJSONCPUID() - cputestJSONSignature() Notice that for tests with json==JSON_MODELS, cpuTestGuestCPUID() is actually called twice but with different arguments. The first one passes JSON_MODELS to the test function, while the second one passes 0. The main difference in behavior when calling cpuTestGuestCPUID() with or without the flag is that in the first case, it parses the captured qemu output from $ARCH-cpuid-$CPU.json. It extracts the cpu model list from that JSON, and uses that to filter out possible cpu models to match. In other words, it tries to match the cpu to a model that was supported by the qemu version that was used to generate this JSON file. When it finds a match, it generates a cpu definition and compares the xml form of that definition with the file $ARCH-cpuid-$CPU-guest.xml. When called without the JSON_MODELS flag, it simply attempts to match it against the full libvirt cpu map and doesn't attempt to filter out any matches based on the JSON qemu cpu model list. After it finds a match, it generates an xml definition for the cpu and compares it to the same file listed above. So if these two invocations disagree on the cpu match (e.g. because libvirt has added a cpu model to its cpu map that matches better than one that was supported by the version of qemu that generated the JSON file) the test will fail. This duplicate call to cpuTestGuestCPUID() was originally added in commit 49c945a6f5c885394507f88086cc2f9461df7c27. The original justification for that commit was to fix test failures when the Qemu driver was disabled. But since DO_TEST_JSON() is #defined empty when qemu is disabled, this particular invocation would not even be executed in this scenario, so it doesn't seem relevant. Signed-off-by: Jonathon Jongsma <jjongsma(a)redhat.com> --- tests/cputest.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/tests/cputest.c b/tests/cputest.c index b3253e3116..93cd0e12a7 100644 --- a/tests/cputest.c +++ b/tests/cputest.c @@ -993,10 +993,6 @@ mymain(void) #if WITH_QEMU # define DO_TEST_JSON(arch, host, json) \ do { \ - if (json == JSON_MODELS) { \ - DO_TEST(arch, cpuTestGuestCPUID, host, host, \ - NULL, NULL, 0, NULL, 0, 0); \ - } \ if (json != JSON_NONE) { \ DO_TEST(arch, cpuTestJSONCPUID, host, host, \ NULL, NULL, 0, NULL, json, 0); \ -- 2.41.0

2 1

[libvirt PATCH v2] rpc: don't try to spawn non-existant daemon
by Daniel P. Berrangé 06 Dec '23

06 Dec '23

If libvirt is built in client only mode, the libvirtd/virtqemud/etc daemons won't exist. If the client is told to connect to a local hypervisor, it'll see the socket doesn't exist, try to spawn the daemon and then re-try connecting to the socket for a few seconds. Ultimately this will fail because the daemon doesn't exist and the user gets an error message error: Failed to connect socket to '/run/user/1000/libvirt/virtqemud-sock': No such file or directory technically this is accurate, but it doesn't help identify the root cause. With this change it will now report error: binary 'virtqemud' does not exist in $PATH: No such file or directory and will skip all the socket connect retries Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- Last time it was suggested that virCommandGetBinaryPath could be extended to always check whether the binary exists. I started doing that and realized it was a bad idea as this method runs in a context which might not have permission to access to the binary we are about to run, as we've not changed user/group ID yet. So I'm re-posting this targetted fix, with format string fixup. src/rpc/virnetsocket.c | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/src/rpc/virnetsocket.c b/src/rpc/virnetsocket.c index b58f7a6b8f..0215c99c73 100644 --- a/src/rpc/virnetsocket.c +++ b/src/rpc/virnetsocket.c @@ -123,9 +123,19 @@ VIR_ONCE_GLOBAL_INIT(virNetSocket); #ifndef WIN32 static int virNetSocketForkDaemon(const char *binary) { - g_autoptr(virCommand) cmd = virCommandNewArgList(binary, - "--timeout=120", - NULL); + g_autofree char *binarypath = virFindFileInPath(binary); + g_autoptr(virCommand) cmd = NULL; + + if (!binarypath) { + virReportSystemError(ENOENT, + _("binary '%1$s' does not exist in $PATH"), + binary); + return -1; + } + + cmd = virCommandNewArgList(binarypath, + "--timeout=120", + NULL); virCommandAddEnvPassCommon(cmd); virCommandAddEnvPass(cmd, "XDG_CACHE_HOME"); -- 2.41.0

2 1

[libvirt PATCH 0/1] rpcgen: Skip generator tests on macOS
by Andrea Bolognani 01 Dec '23

01 Dec '23

Based on: https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/GLJL… Test pipeline: https://gitlab.com/abologna/libvirt/-/pipelines/1091405064 Since we're supposed to tag 9.10.0 tomorrow, go for the quick and dirty workaround. Andrea Bolognani (1): rpcgen: Skip generator tests on macOS scripts/rpcgen/meson.build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) -- 2.43.0

2 3

[PATCH 0/2] Report better error message with no socket to connect to
by Martin Kletzander 01 Dec '23

01 Dec '23

When opening a libvirt connection with no URI set and no daemon running the error message might be confusing at times, so these patches are trying to mitigate such confusion. Martin Kletzander (2): Report first tried socket from remoteProbeSystemDriverFromSocket Report better error message in remoteGetUNIXSocket src/remote/remote_daemon_dispatch.c | 2 +- src/remote/remote_sockets.c | 22 ++++++++++++++++++---- src/remote/remote_sockets.h | 2 +- 3 files changed, 20 insertions(+), 6 deletions(-) -- 2.43.0

5 25

[libvirt PATCH v2] scripts/rpcgen: fix 64 unsigned int test on macOS
by Daniel P. Berrangé 30 Nov '23

30 Nov '23

macOS XDR library is an oddball using xdr_u_int64_t instead of xdr_uint64_t which everyone else has. The code generator already does the right thing, but the test program previously generated with the Linux rpcgen program does not compile on macOS due to this. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- Changed in v2: - Put compat logic in test_demo.c instead of demo.c, since the latter is liable to be re-generated scripts/rpcgen/tests/test_demo.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/rpcgen/tests/test_demo.c b/scripts/rpcgen/tests/test_demo.c index d6be9e236d..94f1002ac8 100644 --- a/scripts/rpcgen/tests/test_demo.c +++ b/scripts/rpcgen/tests/test_demo.c @@ -3,6 +3,10 @@ #include <rpc/xdr.h> #include <stdbool.h> +#ifdef __APPLE__ +# define xdr_uint64_t xdr_u_int64_t +#endif + #include "demo.h" #include "demo.c" -- 2.41.0

3 9

[libvirt PATCH] scripts/rpcgen: fix 64 unsigned int test on macOS
by Daniel P. Berrangé 30 Nov '23

30 Nov '23

macOS XDR library is an oddball using xdr_u_int64_t instead of xdr_uint64_t which everyone else has. The code generator already does the right thing, but the test program previously generated with the Linux rpcgen program does not compile on macOS due to this. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com> --- scripts/rpcgen/tests/demo.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/scripts/rpcgen/tests/demo.c b/scripts/rpcgen/tests/demo.c index 182ed448f0..56a50239dc 100644 --- a/scripts/rpcgen/tests/demo.c +++ b/scripts/rpcgen/tests/demo.c @@ -1,4 +1,8 @@ +#ifdef __APPLE__ +# define xdr_uint64_t xdr_u_int64_t +#endif + void xdr_TestStruct_clear(TestStruct *objp) { xdr_free((xdrproc_t)xdr_TestStruct, (char *)objp); -- 2.41.0

2 4

[libvirt PATCH] rpcgen: Pass 3 arguments to xdrproc_t calls
by Andrea Bolognani 30 Nov '23

30 Nov '23

The test_demo program currently fails to compile on macOS with too few arguments to function call, expected 3, have 2 ret = !!proc(&xdr, vorig); ~~~~ ^ Way back in 2013, commit 9fa3a8ab6fd8 handled this situation for the main library code. Apply the same fix here. Signed-off-by: Andrea Bolognani <abologna(a)redhat.com> --- scripts/rpcgen/tests/test_demo.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/scripts/rpcgen/tests/test_demo.c b/scripts/rpcgen/tests/test_demo.c index a48ceccd58..931f25f982 100644 --- a/scripts/rpcgen/tests/test_demo.c +++ b/scripts/rpcgen/tests/test_demo.c @@ -22,7 +22,7 @@ static void test_xdr(xdrproc_t proc, void *vorig, void *vnew, const char *testna /* Step 1: serialize the vorig and compare to the data in test .bin files */ xdrmem_create(&xdr, buf, buflen, XDR_ENCODE); - ret = !!proc(&xdr, vorig); + ret = !!proc(&xdr, vorig, 0); g_assert_cmpint(ret, ==, !fail); if (fail) @@ -49,7 +49,7 @@ static void test_xdr(xdrproc_t proc, void *vorig, void *vnew, const char *testna /* Step 2: de-serialize the state to create a new object */ xdrmem_create(&xdr, buf, buflen, XDR_DECODE); - ret = !!proc(&xdr, vnew); + ret = !!proc(&xdr, vnew, 0); g_assert_cmpint(ret, ==, true); actlen = xdr_getpos(&xdr); @@ -63,7 +63,7 @@ static void test_xdr(xdrproc_t proc, void *vorig, void *vnew, const char *testna xdrmem_create(&xdr, buf, buflen, XDR_ENCODE); - ret = !!proc(&xdr, vnew); + ret = !!proc(&xdr, vnew, 0); g_assert_cmpint(ret, ==, true); actlen = xdr_getpos(&xdr); -- 2.43.0

2 1

[libvirt PATCH 0/5] rpcgen: Fix test coverage
by Andrea Bolognani 30 Nov '23

30 Nov '23

Right now we don't have pytest installed in any of the CI build environments, and tests are only executed when pytest is present, so the test coverage is exactly zero. This series changes things so that we actually run the test suite as part of the CI pipeline, which immediately triggers a build failure on macOS: In file included from ../scripts/rpcgen/tests/test_demo.c:7: ../scripts/rpcgen/tests/demo.c:402:10: error: call to undeclared function 'xdr_uint64_t'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] if (!xdr_uint64_t(xdrs, &objp->suh)) ^ ../scripts/rpcgen/tests/test_demo.c:25:29: error: too few arguments to function call, expected 3, have 2 ret = !!proc(&xdr, vorig); ~~~~ ^ ../scripts/rpcgen/tests/test_demo.c:52:28: error: too few arguments to function call, expected 3, have 2 ret = !!proc(&xdr, vnew); ~~~~ ^ ../scripts/rpcgen/tests/test_demo.c:66:28: error: too few arguments to function call, expected 3, have 2 ret = !!proc(&xdr, vnew); ~~~~ ^ 4 errors generated. I haven't looked into that failure at all, but it looks like something that we might want to fix before 9.10.0 is released? libvirt itself seems to build fine, so perhaps it's just the test suite that needs fixing. I leave the call to people who are more familiar with the rpcgen situation. Andrea Bolognani (5): rpcgen: Don't skip all tests when pytest is missing rpcgen: Organize meson tests into suites rpcgen: Skip tests if tests are disabled rpcgen: Reformat meson files ci: Refresh generated files ci/buildenv/almalinux-8.sh | 1 + ci/buildenv/alpine-317.sh | 1 + ci/buildenv/alpine-edge.sh | 1 + ci/buildenv/centos-stream-8.sh | 1 + ci/buildenv/centos-stream-9.sh | 1 + ci/buildenv/debian-11-cross-aarch64.sh | 1 + ci/buildenv/debian-11-cross-armv6l.sh | 1 + ci/buildenv/debian-11-cross-armv7l.sh | 1 + ci/buildenv/debian-11-cross-i686.sh | 1 + ci/buildenv/debian-11-cross-mips64el.sh | 1 + ci/buildenv/debian-11-cross-mipsel.sh | 1 + ci/buildenv/debian-11-cross-ppc64le.sh | 1 + ci/buildenv/debian-11-cross-s390x.sh | 1 + ci/buildenv/debian-11.sh | 1 + ci/buildenv/debian-12-cross-aarch64.sh | 1 + ci/buildenv/debian-12-cross-armv6l.sh | 1 + ci/buildenv/debian-12-cross-armv7l.sh | 1 + ci/buildenv/debian-12-cross-i686.sh | 1 + ci/buildenv/debian-12-cross-mips64el.sh | 1 + ci/buildenv/debian-12-cross-mipsel.sh | 1 + ci/buildenv/debian-12-cross-ppc64le.sh | 1 + ci/buildenv/debian-12-cross-s390x.sh | 1 + ci/buildenv/debian-12.sh | 1 + ci/buildenv/debian-sid-cross-aarch64.sh | 1 + ci/buildenv/debian-sid-cross-armv6l.sh | 1 + ci/buildenv/debian-sid-cross-armv7l.sh | 1 + ci/buildenv/debian-sid-cross-i686.sh | 1 + ci/buildenv/debian-sid-cross-mips64el.sh | 1 + ci/buildenv/debian-sid-cross-ppc64le.sh | 1 + ci/buildenv/debian-sid-cross-s390x.sh | 1 + ci/buildenv/debian-sid.sh | 1 + ci/buildenv/fedora-37.sh | 1 + ci/buildenv/fedora-38-cross-mingw32.sh | 1 + ci/buildenv/fedora-38-cross-mingw64.sh | 1 + ci/buildenv/fedora-38.sh | 1 + ci/buildenv/fedora-rawhide-cross-mingw32.sh | 1 + ci/buildenv/fedora-rawhide-cross-mingw64.sh | 1 + ci/buildenv/fedora-rawhide.sh | 1 + ci/buildenv/opensuse-leap-15.sh | 1 + ci/buildenv/opensuse-tumbleweed.sh | 1 + ci/buildenv/ubuntu-2004.sh | 1 + ci/buildenv/ubuntu-2204.sh | 1 + ci/cirrus/freebsd-12.vars | 2 +- ci/cirrus/freebsd-13.vars | 2 +- ci/cirrus/macos-13.vars | 2 +- ci/cirrus/macos-14.vars | 2 +- ci/containers/almalinux-8.Dockerfile | 1 + ci/containers/alpine-317.Dockerfile | 1 + ci/containers/alpine-edge.Dockerfile | 1 + ci/containers/centos-stream-8.Dockerfile | 1 + ci/containers/centos-stream-9.Dockerfile | 1 + .../debian-11-cross-aarch64.Dockerfile | 1 + .../debian-11-cross-armv6l.Dockerfile | 1 + .../debian-11-cross-armv7l.Dockerfile | 1 + ci/containers/debian-11-cross-i686.Dockerfile | 1 + .../debian-11-cross-mips64el.Dockerfile | 1 + .../debian-11-cross-mipsel.Dockerfile | 1 + .../debian-11-cross-ppc64le.Dockerfile | 1 + .../debian-11-cross-s390x.Dockerfile | 1 + ci/containers/debian-11.Dockerfile | 1 + .../debian-12-cross-aarch64.Dockerfile | 1 + .../debian-12-cross-armv6l.Dockerfile | 1 + .../debian-12-cross-armv7l.Dockerfile | 1 + ci/containers/debian-12-cross-i686.Dockerfile | 1 + .../debian-12-cross-mips64el.Dockerfile | 1 + .../debian-12-cross-mipsel.Dockerfile | 1 + .../debian-12-cross-ppc64le.Dockerfile | 1 + .../debian-12-cross-s390x.Dockerfile | 1 + ci/containers/debian-12.Dockerfile | 1 + .../debian-sid-cross-aarch64.Dockerfile | 1 + .../debian-sid-cross-armv6l.Dockerfile | 1 + .../debian-sid-cross-armv7l.Dockerfile | 1 + .../debian-sid-cross-i686.Dockerfile | 1 + .../debian-sid-cross-mips64el.Dockerfile | 1 + .../debian-sid-cross-ppc64le.Dockerfile | 1 + .../debian-sid-cross-s390x.Dockerfile | 1 + ci/containers/debian-sid.Dockerfile | 1 + ci/containers/fedora-37.Dockerfile | 1 + .../fedora-38-cross-mingw32.Dockerfile | 1 + .../fedora-38-cross-mingw64.Dockerfile | 1 + ci/containers/fedora-38.Dockerfile | 1 + .../fedora-rawhide-cross-mingw32.Dockerfile | 1 + .../fedora-rawhide-cross-mingw64.Dockerfile | 1 + ci/containers/fedora-rawhide.Dockerfile | 1 + ci/containers/opensuse-leap-15.Dockerfile | 1 + ci/containers/opensuse-tumbleweed.Dockerfile | 1 + ci/containers/ubuntu-2004.Dockerfile | 1 + ci/containers/ubuntu-2204.Dockerfile | 1 + scripts/rpcgen/meson.build | 17 ++++++----- scripts/rpcgen/rpcgen/meson.build | 10 +++---- scripts/rpcgen/tests/meson.build | 29 ++++++++++--------- 91 files changed, 118 insertions(+), 30 deletions(-) -- 2.43.0

3 9