February 2021 - Devel - libvirt List Archives

[PATCH v2 RESEND 00/13] Introduce virtio-mem <memory/> model
by Michal Privoznik 22 Feb '21

22 Feb '21

Resend of: https://listman.redhat.com/archives/libvir-list/2021-February/msg00534.html We had couple of changes made (new capabilities added mostly) and the original patch set no long applies cleanly. This is pure rebase of the original patch set, no functional change (hence I'm keeping version number). Michal Prívozník (13): virhostmem: Introduce virHostMemGetTHPSize() qemu_process: Deduplicate code in qemuProcessNeedHugepagesPath() qemu_process: Drop needless check in qemuProcessNeedMemoryBackingPath() qemu_capabilities: Introduce QEMU_CAPS_DEVICE_VIRTIO_MEM_PCI conf: Introduce virtio-mem <memory/> model qemu: Build command line for virtio-mem qemu: Wire up <memory/> live update qemu: Wire up MEMORY_DEVICE_SIZE_CHANGE event Introduce MEMORY_DEVICE_SIZE_CHANGE event qemu: Refresh the actual size of virtio-mem on monitor reconnect virsh: Introduce update-memory command news: document recent virtio memory addition kbase: Document virtio-mem NEWS.rst | 7 + docs/formatdomain.rst | 58 ++++- docs/kbase/index.rst | 4 + docs/kbase/memorydevices.rst | 160 ++++++++++++++ docs/kbase/meson.build | 1 + docs/manpages/virsh.rst | 30 +++ docs/schemas/domaincommon.rng | 16 ++ examples/c/misc/event-test.c | 17 ++ include/libvirt/libvirt-domain.h | 23 ++ src/conf/domain_conf.c | 115 +++++++++- src/conf/domain_conf.h | 15 ++ src/conf/domain_event.c | 84 +++++++ src/conf/domain_event.h | 10 + src/conf/domain_validate.c | 39 ++++ src/libvirt_private.syms | 5 + src/qemu/qemu_alias.c | 10 +- src/qemu/qemu_capabilities.c | 2 + src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 13 +- src/qemu/qemu_domain.c | 67 +++++- src/qemu/qemu_domain.h | 1 + src/qemu/qemu_domain_address.c | 38 +++- src/qemu/qemu_driver.c | 209 +++++++++++++++++- src/qemu/qemu_hotplug.c | 18 ++ src/qemu/qemu_hotplug.h | 5 + src/qemu/qemu_monitor.c | 37 ++++ src/qemu/qemu_monitor.h | 27 +++ src/qemu/qemu_monitor_json.c | 95 ++++++-- src/qemu/qemu_monitor_json.h | 5 + src/qemu/qemu_process.c | 101 +++++++-- src/qemu/qemu_validate.c | 22 +- src/remote/remote_daemon_dispatch.c | 30 +++ src/remote/remote_driver.c | 32 +++ src/remote/remote_protocol.x | 15 +- src/remote_protocol-structs | 7 + src/security/security_apparmor.c | 1 + src/security/security_dac.c | 2 + src/security/security_selinux.c | 2 + src/util/virhostmem.c | 63 ++++++ src/util/virhostmem.h | 3 + tests/domaincapsmock.c | 9 + .../caps_5.1.0.x86_64.xml | 1 + .../caps_5.2.0.x86_64.xml | 1 + .../caps_6.0.0.x86_64.xml | 1 + ...mory-hotplug-virtio-mem.x86_64-latest.args | 48 ++++ .../memory-hotplug-virtio-mem.xml | 64 ++++++ tests/qemuxml2argvtest.c | 1 + ...emory-hotplug-virtio-mem.x86_64-latest.xml | 1 + tests/qemuxml2xmltest.c | 1 + tools/virsh-domain.c | 170 ++++++++++++++ 50 files changed, 1619 insertions(+), 68 deletions(-) create mode 100644 docs/kbase/memorydevices.rst create mode 100644 tests/qemuxml2argvdata/memory-hotplug-virtio-mem.x86_64-latest.args create mode 100644 tests/qemuxml2argvdata/memory-hotplug-virtio-mem.xml create mode 120000 tests/qemuxml2xmloutdata/memory-hotplug-virtio-mem.x86_64-latest.xml -- 2.26.2

4 25

[libvirt PATCH] NEWS: mention support for vhost-user-blk
by Pavel Hrdina 22 Feb '21

22 Feb '21

Signed-off-by: Pavel Hrdina <phrdina(a)redhat.com> --- NEWS.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/NEWS.rst b/NEWS.rst index 440dbf2601..56a9afe749 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -47,6 +47,11 @@ v7.1.0 (unreleased) to set the hostdev device's MAC address (which is a necessary part of the alternate <interface type='hostdev'>). + * Introduce ``<disk type='vhostuser'>`` support + + Introduces a support for QEMU vhost-user-blk device that can be used + together with qemu-storage-daemon to provide disks for VMs. + * **Improvements** * **Bug fixes** -- 2.29.2

2 2

[PATCH] NEWS: Mention snapshot quiesce rollback bugfix
by Peter Krempa 22 Feb '21

22 Feb '21

Signed-off-by: Peter Krempa <pkrempa(a)redhat.com> --- NEWS.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/NEWS.rst b/NEWS.rst index 440dbf2601..8d639253ff 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -51,6 +51,20 @@ v7.1.0 (unreleased) * **Bug fixes** + * qemu: Fix disk quiescing rollback when creating external snapshots + + If the qemu guest agent call to freeze filesystems failed when creating + an external snapshot with ``VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE`` flag the + filesystems would be unconditionally thawed. This could cause problems when + the filesystems were frozen by an explicit call to ``virDomainFSFreeze`` + since the guest agent then rejects any further freeze attempts once are + filesystems frozen, an explicit freeze followed by a quiesced snapshot + would fail and thaw filesystems. + + Users are also encouraged to use ``virDomainFSFreeze/Thaw`` manually instead + of relying on ``VIR_DOMAIN_SNAPSHOT_CREATE_QUIESCE`` if they need finer + grained control. + v7.0.0 (2021-01-15) =================== -- 2.29.2

2 1

[libvirt PATCH 0/2] followup fix for cpu quota limits
by Pavel Hrdina 22 Feb '21

22 Feb '21

Pavel Hrdina (2): domain_validate: use defines for cpu period and quota limits docs: use proper cpu quota value in our documentation docs/formatdomain.rst | 8 ++++---- docs/manpages/virsh.rst | 2 +- docs/schemas/domaincommon.rng | 2 +- src/conf/domain_validate.c | 20 +++++++++++++------- 4 files changed, 19 insertions(+), 13 deletions(-) -- 2.29.2

2 3

[PATCH 00/25] Clear pointers in virJSONValue(Object|Array)Append and other cleanups
by Peter Krempa 20 Feb '21

20 Feb '21

Peter Krempa (25): virLockDaemonPreExecRestart: Refactor memory cleanup virLogDaemonPreExecRestart: Refactor memory cleanup virLogHandlerPreExecRestart: Refactor memory cleanup virNetDaemonPreExecRestart: Refactor memory cleanup virNetServerServicePreExecRestart: Refactor memory cleanup virNetServerClientPreExecRestart: Refactor memory cleanup virNetServerPreExecRestart: Drop error reporting from virJSONValueObjectAppend* calls virNetServerPreExecRestart: Refactor memory cleanup virLockSpacePreExecRestart: Refactor memory cleanup qemuAgentMakeCommand: Refactor memory cleanup virJSONValueObjectInsert: Clear @value on successful insertion virJSONValueCopy: Don't use virJSONValue(Object|Array)Append virJSONValue(Array|Object)Append*: Simplify handling of appended object virJSONValueNewArrayFromBitmap: Refactor cleanup virJSONValueObjectAddVArgs: Use autofree for the temporary bitmap virJSONValueObjectAppend: Clear pointer when taking ownership of passed value qemuAgentMakeStringsArray: Refactor cleanup virMACMapHashDumper: Refactor array addition testQEMUSchemaValidateObjectMergeVariantMember: Fix theoretical leak virJSONValueArrayAppend: Clear pointer when taking ownership of passed value qemuMonitorJSONTransactionAdd: Refactor cleanup qemuAgentSetVCPUsCommand: Refactor cleanup virJSONParserHandle*: Refactor memory cleanup and drop NULL checks virJSONValueNewNumber: Take ownership of passed string virJSONParserInsertValue: Take double pointer for @value src/locking/lock_daemon.c | 77 ++++---- src/logging/log_daemon.c | 52 +++--- src/logging/log_handler.c | 40 ++-- src/network/leaseshelper.c | 3 +- src/node_device/node_device_driver.c | 4 +- src/qemu/qemu_agent.c | 89 ++++----- src/qemu/qemu_block.c | 19 +- src/qemu/qemu_command.c | 3 +- src/qemu/qemu_firmware.c | 27 +-- src/qemu/qemu_migration_params.c | 4 +- src/qemu/qemu_monitor_json.c | 58 +++--- src/rpc/virnetdaemon.c | 25 +-- src/rpc/virnetserver.c | 79 +++----- src/rpc/virnetserverclient.c | 24 +-- src/rpc/virnetserverservice.c | 34 ++-- src/util/virjson.c | 261 +++++++++++---------------- src/util/virjson.h | 7 +- src/util/virlease.c | 2 +- src/util/virlockspace.c | 47 ++--- src/util/virmacmap.c | 13 +- tests/testutilsqemuschema.c | 4 +- 21 files changed, 322 insertions(+), 550 deletions(-) -- 2.29.2

2 28

[PATCH v2 00/15] qemu: migrate block bitmaps when migrating storage
by Peter Krempa 19 Feb '21

19 Feb '21

When we are copying storage we should also preserve the block dirty bitmaps. This series implements their migration. For standard migration with shared storage we let qemu flush the bitmaps to disk and reload them on destination. There is possibility to migrate them using the migration stream too but it's not implemented in this series. v2: Patches from v1: - patches 4-7 from v1 were pushed already - patch 8 from v1 was dropped This version: - patch 1 was updated to capture the latest capabilities - patch 4 is new, replaces patch 8 from v1 - all other patches apply review feedback from Jirka - patch 14 has one additional change (thus I didn't carry Jirka's R-b): The migration of bitmaps is enabled based on QEMU_CAPS_MIGRATION_PARAM_BLOCK_BITMAP_MAPPING instead of QEMU_CAPS_INCREMENTAL_BACKUP. Peter Krempa (15): qemucapabilitiesdata: Update test data for qemu-6.0 on x86_64 qemu: capabilities: Introduce QEMU_CAPS_MIGRATION_PARAM_BLOCK_BITMAP_MAPPING qemu: Probe whether an image is 'qcow2 v2' from query-named-block-nodes qemu: migration: Create qcow2 v3 images for VIR_MIGRATE_NON_SHARED_DISK qemu: monitor: Introduce qemuMonitorBitmapRemove qemu: blockjob: Use qemuMonitorBitmapRemove for single bitmap removal qemu: migration_params: Add infrastructure for 'dirty-bitmaps' migration feature qemu: migration_cookie: Add XML handling for setting up bitmap migration qemu: migration_cookie: Add helpers for transforming the cookie into migration params qemu: domain: Store list of temporary bitmaps for migration in status XML tests: qemustatusxml2xml: Add status XML from migration with bitmaps tests: qemumigrationcookie: Add testing for block dirty bitmap migration qemu: migration: Clean up temporary bitmaps when cancelling a migration qemu: migration: Migrate block dirty bitmaps corresponding to checkpoints qemu: capabilities: Enable QEMU_CAPS_INCREMENTAL_BACKUP src/qemu/qemu_blockjob.c | 24 +- src/qemu/qemu_capabilities.c | 9 +- src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_domain.c | 87 +- src/qemu/qemu_domain.h | 15 + src/qemu/qemu_migration.c | 369 ++++++++- src/qemu/qemu_migration_cookie.c | 261 ++++++ src/qemu/qemu_migration_cookie.h | 42 + src/qemu/qemu_migration_params.c | 29 + src/qemu/qemu_migration_params.h | 5 + src/qemu/qemu_monitor.c | 13 + src/qemu/qemu_monitor.h | 8 + src/qemu/qemu_monitor_json.c | 35 + src/qemu/qemu_monitor_json.h | 6 + tests/meson.build | 2 +- tests/qemublocktest.c | 2 + tests/qemublocktestdata/bitmap/synthetic.json | 2 +- tests/qemublocktestdata/bitmap/synthetic.out | 1 + .../caps_6.0.0.x86_64.replies | 741 ++++++++++-------- .../caps_6.0.0.x86_64.xml | 23 +- .../nbd-bitmaps-xml2xml-in.xml | 52 ++ .../nbd-bitmaps-xml2xml-migparams.json | 25 + .../nbd-bitmaps-xml2xml-out.xml | 51 ++ tests/qemumigrationcookiexmltest.c | 166 +++- tests/qemumonitorjsontest.c | 2 + .../migration-out-nbd-bitmaps-in.xml | 574 ++++++++++++++ .../migration-out-nbd-bitmaps-out.xml | 1 + tests/qemustatusxml2xmltest.c | 1 + 28 files changed, 2172 insertions(+), 375 deletions(-) create mode 100644 tests/qemumigrationcookiexmldata/nbd-bitmaps-xml2xml-in.xml create mode 100644 tests/qemumigrationcookiexmldata/nbd-bitmaps-xml2xml-migparams.json create mode 100644 tests/qemumigrationcookiexmldata/nbd-bitmaps-xml2xml-out.xml create mode 100644 tests/qemustatusxml2xmldata/migration-out-nbd-bitmaps-in.xml create mode 120000 tests/qemustatusxml2xmldata/migration-out-nbd-bitmaps-out.xml -- 2.29.2

2 19

[PATCH 0/4] qemuxml2*test: Significantly improve performance by caching qemu capabilities and qapi schema
by Peter Krempa 19 Feb '21

19 Feb '21

This results in 1/3 of the original execution time. Peter Krempa (4): qemuxml2argvtest: Cache QAPI schema between tests testCompareXMLToArgvValidateSchema: Improve and fix helper for testing everything testQemuInfoSetArgs: Use curly braces in else section qemu*xml2*test: Cache capabilities between tests tests/qemustatusxml2xmltest.c | 3 ++- tests/qemuxml2argvtest.c | 26 +++++++++++++++++++------- tests/qemuxml2xmltest.c | 3 ++- tests/testutilsqemu.c | 24 +++++++++++++++++++----- tests/testutilsqemu.h | 2 ++ 5 files changed, 44 insertions(+), 14 deletions(-) -- 2.29.2

2 5

[PATCH v2] qemu: Fix libvirt hang due to early TPM device stop
by Stefan Berger 19 Feb '21

19 Feb '21

This patch partially reverts commit 5cde9dee where the qemuExtDevicesStop() was moved to a location before the QEMU process is stopped. It may be alright to tear down some devices before QEMU is stopped, but it doesn't work for the external TPM (swtpm) which assumes that QEMU sends it a signal to stop it before libvirt may try to clean it up. So this patch moves the virFileDeleteTree() calls after the call to qemuExtDevicesStop() so that the pid file of virtiofsd is not deleted before that call. Afftected libvirt versions are 6.10 and 7.0. Fixes: 5cde9dee8c70b17c458d031ab6cf71dce476eea2 Cc: Masayoshi Mizuma <m.mizuma(a)jp.fujitsu.com> Signed-off-by: Stefan Berger <stefanb(a)linux.ibm.com> --- src/qemu/qemu_process.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 7feb35e609..d930ff9a74 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -7659,11 +7659,6 @@ void qemuProcessStop(virQEMUDriverPtr driver, /* Do this before we delete the tree and remove pidfile. */ qemuProcessKillManagedPRDaemon(vm); - qemuExtDevicesStop(driver, vm); - - virFileDeleteTree(priv->libDir); - virFileDeleteTree(priv->channelTargetDir); - ignore_value(virDomainChrDefForeach(vm->def, false, qemuProcessCleanupChardevDevice, @@ -7677,10 +7672,15 @@ void qemuProcessStop(virQEMUDriverPtr driver, qemuDomainCleanupRun(driver, vm); + qemuExtDevicesStop(driver, vm); + qemuDBusStop(driver, vm); vm->def->id = -1; + virFileDeleteTree(priv->libDir); + virFileDeleteTree(priv->channelTargetDir); + /* Stop autodestroy in case guest is restarted */ qemuProcessAutoDestroyRemove(driver, vm); -- 2.26.2

2 1

[PATCH] qemu: Fix libvirt hang due to early TPM device stop
by Stefan Berger 19 Feb '21

19 Feb '21

This patch partially reverts commit 5cde9dee where the qemuExtDevicesStop() was moved to a location before the QEMU process is stopped. It may be alright to tear down some devices before QEMU is stopped, but it doesn't work for the external TPM (swtpm) which assumes that QEMU sends it a signal to stop it before libvirt may try to clean it up. So this patch introduces a function qemuExtDevicesStopEarly() to accommodate those devices that can/must be torn down early, which seems to be the case with virtio and its virtiofsd, and recreates the location of the call to qemuExtDevicesStop() from before the regression for all other devices. Fixes: 5cde9dee8c70b17c458d031ab6cf71dce476eea2 Cc: Masayoshi Mizuma <m.mizuma(a)jp.fujitsu.com> Signed-off-by: Stefan Berger <stefanb(a)linux.ibm.com> --- src/qemu/qemu_extdevice.c | 27 ++++++++++++++++++++------- src/qemu/qemu_extdevice.h | 4 ++++ src/qemu/qemu_process.c | 5 ++++- 3 files changed, 28 insertions(+), 8 deletions(-) diff --git a/src/qemu/qemu_extdevice.c b/src/qemu/qemu_extdevice.c index 8fe7ceaa10..96fdc9a589 100644 --- a/src/qemu/qemu_extdevice.c +++ b/src/qemu/qemu_extdevice.c @@ -205,7 +205,27 @@ qemuExtDevicesStart(virQEMUDriverPtr driver, return 0; } +/* qemuExtDevicesStopEarly stops devices that may be stopped + * before QEMU terminates + */ +void +qemuExtDevicesStopEarly(virQEMUDriverPtr driver, + virDomainObjPtr vm) +{ + virDomainDefPtr def = vm->def; + size_t i; + for (i = 0; i < def->nfss; i++) { + virDomainFSDefPtr fs = def->fss[i]; + + if (fs->fsdriver == VIR_DOMAIN_FS_DRIVER_TYPE_VIRTIOFS) + qemuVirtioFSStop(driver, vm, fs); + } +} + +/* qemuExtDevicesStop stops devices that may only be stopped + * after QEMU terminated + */ void qemuExtDevicesStop(virQEMUDriverPtr driver, virDomainObjPtr vm) @@ -236,13 +256,6 @@ qemuExtDevicesStop(virQEMUDriverPtr driver, if (actualType == VIR_DOMAIN_NET_TYPE_ETHERNET && net->downscript) virNetDevRunEthernetScript(net->ifname, net->downscript); } - - for (i = 0; i < def->nfss; i++) { - virDomainFSDefPtr fs = def->fss[i]; - - if (fs->fsdriver == VIR_DOMAIN_FS_DRIVER_TYPE_VIRTIOFS) - qemuVirtioFSStop(driver, vm, fs); - } } diff --git a/src/qemu/qemu_extdevice.h b/src/qemu/qemu_extdevice.h index 49373a15a1..df0cbd6993 100644 --- a/src/qemu/qemu_extdevice.h +++ b/src/qemu/qemu_extdevice.h @@ -51,6 +51,10 @@ int qemuExtDevicesStart(virQEMUDriverPtr driver, ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) G_GNUC_WARN_UNUSED_RESULT; +void qemuExtDevicesStopEarly(virQEMUDriverPtr driver, + virDomainObjPtr vm) + ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2); + void qemuExtDevicesStop(virQEMUDriverPtr driver, virDomainObjPtr vm) ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 7feb35e609..b5b2ddef4c 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -7659,7 +7659,7 @@ void qemuProcessStop(virQEMUDriverPtr driver, /* Do this before we delete the tree and remove pidfile. */ qemuProcessKillManagedPRDaemon(vm); - qemuExtDevicesStop(driver, vm); + qemuExtDevicesStopEarly(driver, vm); virFileDeleteTree(priv->libDir); virFileDeleteTree(priv->channelTargetDir); @@ -7677,6 +7677,9 @@ void qemuProcessStop(virQEMUDriverPtr driver, qemuDomainCleanupRun(driver, vm); + /* tear down external devices after QEMU is gone */ + qemuExtDevicesStop(driver, vm); + qemuDBusStop(driver, vm); vm->def->id = -1; -- 2.26.2

2 1

[PATCH] event-test: Properly terminate strings printed from callbacks
by Kristina Hanicova 19 Feb '21

19 Feb '21

Stdio was buffering strings in functions: myDomainEventBlockJobCallback, myDomainEventBlockThresholdCallback, myDomainEventMemoryFailureCallback. It caused flushing the printed strings from callbacks at the end of a run, not gradually. The solution is to add \n at the end of each string. Signed-off-by: Kristina Hanicova <khanicov(a)redhat.com> --- examples/c/misc/event-test.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/examples/c/misc/event-test.c b/examples/c/misc/event-test.c index f164e825e1..76d4f3f6e8 100644 --- a/examples/c/misc/event-test.c +++ b/examples/c/misc/event-test.c @@ -938,7 +938,7 @@ myDomainEventBlockJobCallback(virConnectPtr conn G_GNUC_UNUSED, const char *eventName = opaque; printf("%s EVENT: Domain %s(%d) block job callback '%s' disk '%s', " - "type '%s' status '%s'", + "type '%s' status '%s'\n", __func__, virDomainGetName(dom), virDomainGetID(dom), eventName, disk, blockJobTypeToStr(type), blockJobStatusToStr(status)); return 0; @@ -956,7 +956,7 @@ myDomainEventBlockThresholdCallback(virConnectPtr conn G_GNUC_UNUSED, { /* Casts to uint64_t to work around mingw not knowing %lld */ printf("%s EVENT: Domain %s(%d) block threshold callback dev '%s'(%s), " - "threshold: '%" PRIu64 "', excess: '%" PRIu64 "'", + "threshold: '%" PRIu64 "', excess: '%" PRIu64 "'\n", __func__, virDomainGetName(dom), virDomainGetID(dom), dev, NULLSTR(path), (uint64_t)threshold, (uint64_t)excess); return 0; @@ -972,7 +972,7 @@ myDomainEventMemoryFailureCallback(virConnectPtr conn G_GNUC_UNUSED, void *opaque G_GNUC_UNUSED) { printf("%s EVENT: Domain %s(%d) memory failure: recipient '%d', " - "aciont '%d', flags '%d'", __func__, virDomainGetName(dom), + "aciont '%d', flags '%d'\n", __func__, virDomainGetName(dom), virDomainGetID(dom), recipient, action, flags); return 0; } -- 2.29.2

2 1