- Devel - libvirt List Archives

[libvirt] [PATCH 0/3] qemu: guest dedicated crypto adapters
by Boris Fiuczynski 15 Oct '18

15 Oct '18

This patch series introduces initial libvirt support for guest dedicated crypto adapters on S390. It allows to specify a vfio-ap mediated device in a domain. Extensive documentation about AP is available in patch 6 of the QEMU patch series. KVM/kernel: guest dedicated crypto adapters https://lkml.org/lkml/2018/9/26/25 QEMU: s390x: vfio-ap: guest dedicated crypto adapters https://lists.gnu.org/archive/html/qemu-devel/2018-10/msg01993.html The qemu patch series has been included into master. https://git.qemu.org/?p=qemu.git;a=commit;h=69ac8c4cb93f2685839ff7b857cef30… Boris Fiuczynski (3): qemu: add vfio-ap capability qemu: vfio-ap device support news: Update news for vfio-ap support docs/formatdomain.html.in | 3 ++- docs/news.xml | 9 +++++++++ docs/schemas/domaincommon.rng | 1 + src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 8 ++++++++ src/qemu/qemu_domain_address.c | 4 ++++ src/util/virmdev.c | 3 ++- src/util/virmdev.h | 1 + 9 files changed, 30 insertions(+), 2 deletions(-) -- 2.17.0

1 3

[libvirt] [RFC 0/7] Warn at runtime when deprecated features are used
by Andrea Bolognani 15 Oct '18

15 Oct '18

Background ========== We have plenty of features in libvirt, some of which were designed at a time when the virtualization story was much more straightforward than the multi-architecture, multi-hypervisor, multi-machine world we currently live in and, while we have found ways to keep the APIs chugging along, the result is sometimes somewhat confusing for users and application developers, as well as requiring libvirt developers themselves to spend quite a bit of collective time working around decisions that, in hindsight, turn out to have been less than fortunate. Two concrete examples are considered here: one is the virConnectNumOfDomains() API which, while known to be racy and having non-racy alternatives, can still be used by developers without getting any kind of warning in the process; the other one is the ability to define a domain without specifying the machine type, which is becoming increasingly problematic now with some x86_64 features being limited to q35 and downstreams looking to push for its adoption, as well as being a manifestation of the more general problem of libvirt's default being sometimes too conservative and at odds with the existence of slimmed-down QEMU binaries being built with reducing the total attack surface in mind. Having a proper deprecation story in libvirt would allow us to point users and developers towards the recommended solution in each case, be it using a different API or querying libosinfo for information; after a generous grace period, we could then remove the problematic functionality altogether. This would be a more conservative version of the process we already have in place for dropping support for older QEMU releases, which recently has allowed us to ax massive chunks of effectively dead code and simplify parts of libvirt quite significantly. This series explores one possible approach to the problem and aims to spark project-wide discussion around the topic. Further work ============ * Fix the known issues listed below as well as all not-yet-known issues that will undoubtably surface through discussion :) * Introduce a mechanism to catch use of deprecated APIs at build time, similar to GLib's G_DISABLE_DEPRECATED, to help application developers proactively move off problematic APIs. * Create a formal deprecation policy with well-defined rules and time scales in the spirit of the existing one covering our relationship with QEMU. Know issues =========== * For the more granular (and more interesting) type of deprecation shown in patch 6/7, warnings are not being reported back to the client as expected. I believe this is caused by the RPC code looking for either a failure, in which case the virError is transmitted, or a success, in which case the actual return value is: we'll have to figure out a way for the error to travel across the wire regardless of whether or not the API call was ultimately successful if we want clients to actually receive warnings when non-local drivers are involved. Andrea Bolognani (7): util: Add 'level' argument to virReportErrorHelper() util: Introduce virReportWarning() tools: Print warnings in virsh util: Introduce VIR_ERR_DEPRECATED_FEATURE Deprecate virConnectNumOfDomains() Deprecate missing machine type in virDomainDefineXMLFlags() tools: Force virsh to use deprecated features include/libvirt/virterror.h | 1 + src/access/viraccessdriverpolkit.c | 2 +- src/access/viraccessmanager.c | 2 +- src/conf/domain_conf.c | 11 ++++++++--- src/datatypes.h | 30 ++++++++++++++++++++++++++---- src/libvirt-domain.c | 6 ++++++ src/libvirt.c | 1 + src/util/virbuffer.c | 4 ++-- src/util/virconf.c | 6 ++++-- src/util/virerror.c | 10 +++++++++- src/util/virerror.h | 15 ++++++++++----- src/util/virkeyfile.c | 6 ++++-- src/util/virxml.c | 2 +- tools/virsh-domain-monitor.c | 2 ++ tools/vsh.c | 3 +++ 15 files changed, 79 insertions(+), 22 deletions(-) -- 2.17.1

10 59

[libvirt] [PATCH 0/2] util: storage: Fix parsing of URIs as disk sources
by Peter Krempa 12 Oct '18

12 Oct '18

see 2/2 for explanation Peter Krempa (2): util: storage: Rename '@path' argument of virStorageSourceParseBackingURI util: storage: Properly parse URIs with missing trailing slash src/util/virstoragefile.c | 28 +++++++++++++++++++++------- tests/qemublocktest.c | 3 +++ tests/virstoragetest.c | 20 ++++++++++++++++++++ 3 files changed, 44 insertions(+), 7 deletions(-) -- 2.17.1

3 4

[libvirt] [RFC PATCH 0/3] qemu: guest dedicated crypto adapters
by Boris Fiuczynski 12 Oct '18

12 Oct '18

This RFC patch series introduces initial libvirt support for guest dedicated crypto adapters on S390. It essentially allows to specify a vfio-ap mediated device in a domain. Extensive documentation about AP is available in patch 6 of the QEMU patch series. KVM/kernel: guest dedicated crypto adapters https://lkml.org/lkml/2018/9/26/25 QEMU: s390x: vfio-ap: guest dedicated crypto adapters https://lists.gnu.org/archive/html/qemu-devel/2018-09/msg03538.html Boris Fiuczynski (3): qemu: add vfio-ap capability qemu: vfio-ap device support news: Update news for vfio-ap support docs/formatdomain.html.in | 3 ++- docs/news.xml | 9 +++++++++ docs/schemas/domaincommon.rng | 1 + src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 8 ++++++++ src/qemu/qemu_domain_address.c | 4 ++++ src/util/virmdev.c | 3 ++- src/util/virmdev.h | 1 + 9 files changed, 30 insertions(+), 2 deletions(-) -- 2.17.0

3 10

[libvirt] [PATCH v1] rpc: reproducible genprotocol output
by Olaf Hering 12 Oct '18

12 Oct '18

If the same source gets built twice ('build same source on different hosts at different times') the resulting files may differ. Fix this by sorting the hash keys before usage. Signed-off-by: Olaf Hering <olaf(a)aepfle.de> --- src/rpc/genprotocol.pl | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/rpc/genprotocol.pl b/src/rpc/genprotocol.pl index 6ce268c024..6baa4f22d5 100755 --- a/src/rpc/genprotocol.pl +++ b/src/rpc/genprotocol.pl @@ -104,7 +104,7 @@ while (<RPCGEN>) { if (keys %uses >= 1) { my $i = 1; - foreach (keys %uses) { + foreach (sort(keys %uses)) { $i = $uses{$_}; unshift @function, (" char **objp_cpp$i = (char **) (void *) &$_;\n");

2 1

[libvirt] [PATCH] virsh: Fix regression with duplicated error messages
by Eric Blake 12 Oct '18

12 Oct '18

Commit 4f4c3b13 (v3.3) fixed an issue where cleaning libvirt objects lost error messages, by adding code to copy the libvirt error into last_error prior to cleanup paths. However, it caused a regression: some errors are now printed twice, because libvirt still remembers in its thread-local storage that an error was set. For example: $ virsh -c test:///default snapshot-delete test blah error: Domain snapshot not found: no domain snapshot with matching name 'blah' error: Domain snapshot not found: no domain snapshot with matching name 'blah' Fix things by telling libvirt to discard any thread-local errors at the same time virsh prints an error message (whether or not the libvirt error is the same as what is stored in last_error). Update the virsh-undefine testsuite (partially reverting portions of commit b620bdee, by removing -q, to more easily pinpoint which commands are causing which messages), now that there is only one error message instead of two. Signed-off-by: Eric Blake <eblake(a)redhat.com> --- tests/virsh-undefine | 20 ++++++++++++++++---- tools/vsh.c | 1 + 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/tests/virsh-undefine b/tests/virsh-undefine index 974b0c71f7..4a9f68dd39 100755 --- a/tests/virsh-undefine +++ b/tests/virsh-undefine @@ -30,34 +30,46 @@ fail=0 # connection is opened to the test driver, it starts life with a new # persistent running domain named 'test' with a different uuid, so # testing this command requires batch mode use of virsh. -$abs_top_builddir/tools/virsh -q -c test:///default \ +$abs_top_builddir/tools/virsh -c test:///default \ 'dominfo test; undefine test; dominfo test' > out1 2>&1 test $? = 0 || fail=1 sed '/^Persistent/n; /:/d' < out1 > out cat <<\EOF > exp || fail=1 Persistent: yes + +Domain test has been undefined + Persistent: no + EOF compare exp out || fail=1 # A similar diagnostic when specifying a domain ID -$abs_top_builddir/tools/virsh -q -c test:///default \ +$abs_top_builddir/tools/virsh -c test:///default \ 'dominfo 1; undefine 1; dominfo 1' > out1 2>&1 test $? = 0 || fail=1 sed '/^Persistent/n; /:/d' < out1 > out cat <<\EOF > exp || fail=1 Persistent: yes + +Domain 1 has been undefined + Persistent: no + EOF compare exp out || fail=1 # Succeed, now: first shut down, then undefine, both via name. -$abs_top_builddir/tools/virsh -q -c test:///default \ +$abs_top_builddir/tools/virsh -c test:///default \ 'shutdown test; undefine test; dominfo test' > out 2>&1 test $? = 1 || fail=1 cat <<\EOF > expout || fail=1 +Domain test is being shutdown + +Domain test has been undefined + error: failed to get domain 'test' -error: Domain not found + EOF compare expout out || fail=1 diff --git a/tools/vsh.c b/tools/vsh.c index 9ea3c4b96a..de887a9e76 100644 --- a/tools/vsh.c +++ b/tools/vsh.c @@ -276,6 +276,7 @@ vshResetLibvirtError(void) { virFreeError(last_error); last_error = NULL; + virResetLastError(); } /* -- 2.17.1

2 1

[libvirt] [RFC] Faster libvirtd restart with nwfilter rules
by Nikolay Shirokovskiy 12 Oct '18

12 Oct '18

Hi, all. On fat hosts which are capable to run hundreds of VMs restarting libvirtd makes it's services unavailable for a long time if VMs use network filters. In my tests each of 100 VMs has no-promisc [1] and no-mac-spoofing filters and executing virsh list right after daemon restart takes appoximately 140s if no firewalld is running (that is ebtables/iptables/ip6tables commands are used to configure kernel tables). The problem is daemon does not even start to read from client connections because state drivers are not initialized. Initialization is blocked in state drivers autostart which grabs VMs locks. And VMs locks are hold by VMs reconnection code. Each VM reloads network tables on reconnection and this reloading is serialized on updateMutex in gentech nwfilter driver. Workarounding autostart won't help much because even if state drivers will initialize listing VM won't be possible because listing VMs takes each VM lock one by one too. However managing VM that passed reconnection phase will be possible which takes same 140s in worst case. Note that this issue is only applicable if we use filters configuration that don't need ip learning. In the latter case situation is different because reconnection code spawns new thread that apply network rules only after ip is learned from traffic and this thread does not grab VM lock. As result VMs are managable but reloading filters in background takes appoximately those same 140s. I guess managing network filters during this period can have issues too. Anyway this situation does not look good so fixing the described issue by spawning threads even without ip learning does not look nice to me. What speed up is possible on conservative approach? First we can remove for test purpuses firewall ruleLock, gentech dirver updateMutex and filter object mutex which do not serve function in restart scenario. This gives 36s restart time. The speed up is archived because heavy fork/preexec steps are now run concurrently. Next we can try to reduce fork/preexec time. To estimate its contibution alone let's bring back the above locks. It turns out the most time takes fork itself and closing 8k (on my system) file descriptors in preexec. Using vfork gives 2x boost and so does dropping mass close. (I check this mass close contribution because I not quite understand the purpose of this step - libvirt typically set close-on-exec flag on it's descriptors). So this two optimizations alone can result in restart time of 30s. Unfortunately combining the above two approaches does not give boost multiple of them along. The reason is due to concurrency and high number of VMs (100) preexec boost does not have significant role and using vfork dininishes concurrency as it freezes all parent threads before execve. So dropping locks and closes gives 33s restart time and adding vfork to this gives 25s restart time. Another approach is to use --atomic-file option for ebtables (iptables/ip6tables unfortunately does not have one). The idea is to save table to file/edit file/commit table to kernel. I hoped this could give performance boost because we don't need to load/store kernel network table for a single rule update. In order to isolate approaches I also dropped all ip/ip6 updates which can not be done this way. In this approach we can not drop ruleLock in firewall because no other VM threads should change tables between save/commit. This approach gives restart time 25s. But this approach is broken anyway as we can not be sure another application doesn't change newtork table between save/commit in which case these changes will be lost. After all I think we need to move in a different direction. We can add API to all binaries and firewalld to execute many commands in one run. We can pass commands as arguments or wrote them into file which is then given to binary. Then libvirt itself can update for example bridge network table in couple of commands. The exact number depends on new API. For example if we add option to delete chains recursively and an option not to fail on NOENT error we can change table in one command (no listing current rules is required). [1] no-promisc filter <filter name='no-promisc' chain='root' priority='-750'> <uuid>6d055022-1192-4a3d-ae1f-576baa5564b6</uuid> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='ff:ff:ff:ff:ff:ff'/> </rule> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='$MAC'/> </rule> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='33:33:00:00:00:00' dstmacmask='ff:ff:00:00:00:00'/> </rule> <rule action='drop' direction='in' priority='500'> <mac/> </rule> <rule action='return' direction='in' priority='500'> <mac dstmacaddr='01:00:5e:00:00:00' dstmacmask='ff:ff:ff:80:00:00'/> </rule> </filter>

3 6

[libvirt] [PATCH] qemu: Remove unused qemuProcessAutostartAll
by John Ferlan 12 Oct '18

12 Oct '18

The function was never defined in source, just the protoype. Signed-off-by: John Ferlan <jferlan(a)redhat.com> --- Pushed as trivial. src/qemu/qemu_process.h | 1 - 1 file changed, 1 deletion(-) diff --git a/src/qemu/qemu_process.h b/src/qemu/qemu_process.h index c2f7c2b5d2..2037467c94 100644 --- a/src/qemu/qemu_process.h +++ b/src/qemu/qemu_process.h @@ -47,7 +47,6 @@ int qemuProcessDestroyMemoryBackingPath(virQEMUDriverPtr driver, virDomainObjPtr vm, virDomainMemoryDefPtr mem); -void qemuProcessAutostartAll(virQEMUDriverPtr driver); void qemuProcessReconnectAll(virQEMUDriverPtr driver); typedef struct _qemuProcessIncomingDef qemuProcessIncomingDef; -- 2.17.1

1 0

[libvirt] [glib PATCH 0/3] Add ich9 sound device + _domain_os_get_machine()
by Fabiano Fidêncio 11 Oct '18

11 Oct '18

The first patch is needed to properly support Q35 machine type on GNOME Boxes (or any other app using libvirt-glib) as the ICH9 audio device should be used together with Q35 machine type, The second patch is mostly a convenience for the apps and the third one just adds a test case for the second. Fabiano Fidêncio (3): gconfig-domain-sound: Add ich9 sound device gconfig: add _domain_os_get_machine() tests: Add testes for _domain_os_{set,get}_machine() libvirt-gconfig/libvirt-gconfig-domain-os.c | 7 +++++++ libvirt-gconfig/libvirt-gconfig-domain-os.h | 1 + libvirt-gconfig/libvirt-gconfig-domain-sound.h | 3 ++- libvirt-gconfig/libvirt-gconfig.sym | 1 + tests/test-gconfig.c | 2 ++ tests/xml/gconfig-domain-os.xml | 2 +- 6 files changed, 14 insertions(+), 2 deletions(-) -- 2.19.1

2 4

[libvirt] [PATCH v2 1/2] add nodeset='all' for interleave mode
by Peng Hao 11 Oct '18

11 Oct '18

sometimes we hoped that the memory of vm can be evenly distributed in all nodes according to interleave mode. But different hosts has different node number. So we add nodeset='all' for interleave mode. Signed-off-by: Peng Hao <peng.hao2(a)zte.com.cn> --- src/conf/numa_conf.c | 77 ++++++++++++++++++++++++++++++++++++------- 1 files changed, 64 insertions(+), 13 deletions(-) diff --git a/src/conf/numa_conf.c b/src/conf/numa_conf.c index 97a3ca4..a336a62 100644 --- a/src/conf/numa_conf.c +++ b/src/conf/numa_conf.c @@ -29,6 +29,9 @@ #include "virnuma.h" #include "virstring.h" +#if WITH_NUMACTL +#include <numa.h> +#endif /* * Distance definitions defined Conform ACPI 2.0 SLIT. * See include/linux/topology.h @@ -66,6 +69,7 @@ typedef virDomainNumaNode *virDomainNumaNodePtr; struct _virDomainNuma { struct { bool specified; + bool allnode; virBitmapPtr nodeset; virDomainNumatuneMemMode mode; virDomainNumatunePlacement placement; @@ -259,13 +263,21 @@ virDomainNumatuneParseXML(virDomainNumaPtr numa, tmp = virXMLPropString(node, "nodeset"); if (tmp) { - if (virBitmapParse(tmp, &nodeset, VIR_DOMAIN_CPUMASK_LEN) < 0) - goto cleanup; - - if (virBitmapIsAllClear(nodeset)) { + if (STREQ(tmp, "all") && !virNumaIsAvailable()) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, - _("Invalid value of 'nodeset': %s"), tmp); + _("Invalid nodeset=%s when numactl is not supported"), tmp); goto cleanup; + } else if (STREQ(tmp, "all") && virNumaIsAvailable()) { + numa->memory.allnode = true; + } else { + if (virBitmapParse(tmp, &nodeset, VIR_DOMAIN_CPUMASK_LEN) < 0) + goto cleanup; + + if (virBitmapIsAllClear(nodeset)) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("Invalid value of 'nodeset': %s"), tmp); + goto cleanup; + } } VIR_FREE(tmp); @@ -319,10 +331,14 @@ virDomainNumatuneFormatXML(virBufferPtr buf, virBufferAsprintf(buf, "<memory mode='%s' ", tmp); if (numatune->memory.placement == VIR_DOMAIN_NUMATUNE_PLACEMENT_STATIC) { - if (!(nodeset = virBitmapFormat(numatune->memory.nodeset))) - return -1; - virBufferAsprintf(buf, "nodeset='%s'/>\n", nodeset); - VIR_FREE(nodeset); + if (numatune->memory.allnode == true) { + virBufferAddLit(buf, "nodeset='all'/>\n"); + } else { + if (!(nodeset = virBitmapFormat(numatune->memory.nodeset))) + return -1; + virBufferAsprintf(buf, "nodeset='%s'/>\n", nodeset); + VIR_FREE(nodeset); + } } else if (numatune->memory.placement) { tmp = virDomainNumatunePlacementTypeToString(numatune->memory.placement); virBufferAsprintf(buf, "placement='%s'/>\n", tmp); @@ -489,6 +505,37 @@ virDomainNumatuneMaybeFormatNodeset(virDomainNumaPtr numatune, return 0; } +#if WITH_NUMACTL +static int +makeAllnodeBitmap(virDomainNumaPtr numa) +{ + size_t i = 0, maxnode = 0; + virBitmapPtr bitmap = NULL; + + if ((bitmap = virBitmapNew(VIR_DOMAIN_CPUMASK_LEN)) == NULL) + return -1; + virBitmapClearAll(bitmap); + maxnode = numa_max_node(); + for (i = 0; i <= maxnode; i++) { + if (virBitmapSetBit(bitmap, i) < 0) { + virBitmapFree(bitmap); + return -1; + } + } + + virBitmapFree(numa->memory.nodeset); + numa->memory.nodeset = bitmap; + + return 0; +} +#else +static int +makeAllnodeBitmap(virDomainNumaPtr numa) +{ + return -1; +} +#endif + int virDomainNumatuneSet(virDomainNumaPtr numa, bool placement_static, @@ -538,20 +585,28 @@ virDomainNumatuneSet(virDomainNumaPtr numa, } if (placement == VIR_DOMAIN_NUMATUNE_PLACEMENT_DEFAULT) { - if (numa->memory.nodeset || placement_static) + if (numa->memory.nodeset || placement_static || numa->memory.allnode) placement = VIR_DOMAIN_NUMATUNE_PLACEMENT_STATIC; else placement = VIR_DOMAIN_NUMATUNE_PLACEMENT_AUTO; } if (placement == VIR_DOMAIN_NUMATUNE_PLACEMENT_STATIC && - !numa->memory.nodeset) { + !numa->memory.nodeset && !numa->memory.allnode) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("nodeset for NUMA memory tuning must be set " "if 'placement' is 'static'")); goto cleanup; } + if (placement == VIR_DOMAIN_NUMATUNE_PLACEMENT_STATIC && + mode == VIR_DOMAIN_NUMATUNE_MEM_INTERLEAVE && + numa->memory.allnode && virNumaIsAvailable()) { + + if (makeAllnodeBitmap(numa) < 0) + goto cleanup; + } + /* setting nodeset when placement auto is invalid */ if (placement == VIR_DOMAIN_NUMATUNE_PLACEMENT_AUTO && numa->memory.nodeset) { -- 1.8.3.1

3 5