- Devel - libvirt List Archives

[libvirt] [PATCH 00/11] introduce virFileCache and use it for qemu capabilities
by Pavel Hrdina 19 Jul '17

19 Jul '17

Pavel Hrdina (11): util/virhash: add name parameter to virHashSearch util: introduce virFileCache tests: add virfilecachetest qemu: move libvirt ctime and version into _virQEMUCaps struct qemu: move virQEMUCapsIsValid before its usage and make it static qemu: move libvirt ctime and version check into virQEMUCapsIsValid qemu: don't pass qemuctime into virQEMUCapsIsValid qemu: separate virQEMUCapsInitCached out of virQEMUCapsNewForBinaryInternal qemu: introduce struct _virQEMUCapsCachePriv qemu: switch QEMU capabilities to use virFileCache qemu: privatize _virQEMUCapsCachePriv struct po/POTFILES.in | 1 + src/Makefile.am | 1 + src/conf/virdomainobjlist.c | 2 +- src/conf/virnetworkobj.c | 4 +- src/conf/virsecretobj.c | 2 +- src/libvirt_private.syms | 9 + src/qemu/qemu_capabilities.c | 517 +++++++-------------- src/qemu/qemu_capabilities.h | 22 +- src/qemu/qemu_capspriv.h | 18 +- src/qemu/qemu_conf.h | 3 +- src/qemu/qemu_driver.c | 2 +- src/util/virfilecache.c | 391 ++++++++++++++++ src/util/virfilecache.h | 136 ++++++ src/util/virhash.c | 11 +- src/util/virhash.h | 2 +- src/xen/xm_internal.c | 5 +- tests/Makefile.am | 12 + tests/qemucapabilitiestest.c | 4 +- tests/qemucapsprobe.c | 2 +- tests/testutilsqemu.c | 14 +- tests/testutilsqemu.h | 3 +- ...a15b1658aa16923e46497dd8deeb6be287ddb0ca0.cache | 1 + ...ae4bc3a28e75bc3e262757001e8b953580f5e75ef.cache | 1 + tests/virfilecachetest.c | 238 ++++++++++ tests/virhashtest.c | 2 +- 25 files changed, 1008 insertions(+), 395 deletions(-) create mode 100644 src/util/virfilecache.c create mode 100644 src/util/virfilecache.h create mode 100644 tests/virfilecachedata/5f3154560c130108b282a2aa15b1658aa16923e46497dd8deeb6be287ddb0ca0.cache create mode 120000 tests/virfilecachedata/9ca150bf3119b75dcac8e8bae4bc3a28e75bc3e262757001e8b953580f5e75ef.cache create mode 100644 tests/virfilecachetest.c -- 2.13.2

2 24

[libvirt] [PATCH] qemu: avoid deadlock on domain object enter monitor fail
by Wang King 19 Jul '17

19 Jul '17

Should be followed with qemuDomainObjExitMonitor only if qemuDomainObjEnterMonitorAsync returns 0. --- src/qemu/qemu_migration.c | 2 +- src/qemu/qemu_process.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c23fffe..dc56263 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -108,7 +108,7 @@ qemuMigrationCheckTLSCreds(virQEMUDriverPtr driver, qemuMonitorMigrationParams migParams = { 0 }; if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) < 0) - goto cleanup; + return -1; if (qemuMonitorGetMigrationParams(priv->mon, &migParams) < 0) goto cleanup; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index c2c3e8b..8d3cfe0 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -4349,7 +4349,7 @@ qemuProcessSetupBalloon(virQEMUDriverPtr driver, return 0; if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) < 0) - goto cleanup; + return -1; if (vm->def->memballoon->period) qemuMonitorSetMemoryStatsPeriod(priv->mon, vm->def->memballoon, -- 2.8.3

2 1

[libvirt] [PATCH] security: apparmor: Properly link with storage driver in helper program
by Peter Krempa 19 Jul '17

19 Jul '17

The refactor to split up storage driver into modules broke the apparmor helper program, since that did not initialize the storage driver properly and thus detection of the backing chain could not work. Register the storage driver backends explicitly. Unfortunately it's now necessary to link with the full storage driver to satisfy dependencies of the loadable modules. --- src/Makefile.am | 2 +- src/security/virt-aa-helper.c | 8 ++++++++ 2 files changed, 9 insertions(+), 1 deletion(-) diff --git a/src/Makefile.am b/src/Makefile.am index 399d031dd..e637dfd91 100644 --- a/src/Makefile.am +++ b/src/Makefile.am @@ -3281,7 +3281,7 @@ virt_aa_helper_LDADD = \ libvirt.la \ libvirt_conf.la \ libvirt_util.la \ - libvirt_driver_storage_impl.la \ + libvirt_driver_storage.la \ ../gnulib/lib/libgnu.la if WITH_DTRACE_PROBES virt_aa_helper_LDADD += libvirt_probes.lo diff --git a/src/security/virt-aa-helper.c b/src/security/virt-aa-helper.c index 695272076..a751d6deb 100644 --- a/src/security/virt-aa-helper.c +++ b/src/security/virt-aa-helper.c @@ -56,6 +56,7 @@ #include "virgettext.h" #include "storage/storage_source.h" +#include "storage/storage_backend.h" #define VIR_FROM_THIS VIR_FROM_SECURITY @@ -926,6 +927,11 @@ get_files(vahControl * ctl) goto cleanup; } + if (virStorageBackendDriversRegister(false) < 0) { + vah_error(ctl, 0, _("failed to register storage driver backend")); + goto cleanup; + } + for (i = 0; i < ctl->def->ndisks; i++) { virDomainDiskDefPtr disk = ctl->def->disks[i]; @@ -1283,6 +1289,8 @@ main(int argc, char **argv) exit(EXIT_FAILURE); } + virFileActivateDirOverride(argv[0]); + /* Initialize the log system */ virLogSetFromEnv(); -- 2.13.2

2 2

[libvirt] [PATCH] virFileInData: Report an error if unable to reposition file
by Michal Privoznik 19 Jul '17

19 Jul '17

The purpose of this function is to tell if the current position in given FD is in data section or a hole and how much bytes there is remaining until the end of the section. This is achieved by couple of lseeks(). The most important part is that we reposition the FD back, so that the position is unchanged from the caller POV. And until now the final lseek() back to the original position was done with no check for errors. And I was convinced that that's okay since nothing can go wrong. However, John persuaded me, that it's better to be safe than sorry. Therefore, lets check if the final lseek() succeeded and if it doesn't report an error. Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- src/util/virfile.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/util/virfile.c b/src/util/virfile.c index d444b32f8..2f28e83f4 100644 --- a/src/util/virfile.c +++ b/src/util/virfile.c @@ -3900,8 +3900,12 @@ virFileInData(int fd, ret = 0; cleanup: /* At any rate, reposition back to where we started. */ - if (cur != (off_t) -1) - ignore_value(lseek(fd, cur, SEEK_SET)); + if (cur != (off_t) -1 && + lseek(fd, cur, SEEK_SET) == (off_t) -1) { + virReportSystemError(errno, "%s", + _("unable to restore position in file")); + ret = -1; + } return ret; } -- 2.13.0

2 2

Re: [libvirt] [PATCH] spapr: make default PHB optionnal
by Andrea Bolognani 19 Jul '17

19 Jul '17

[libvir-list added to the loop] On Tue, 2017-07-04 at 10:47 +0200, Greg Kurz wrote: > On Tue, 4 Jul 2017 17:29:01 +1000 David Gibson <david(a)gibson.dropbear.id.au> wrote: > > On Mon, Jul 03, 2017 at 06:48:25PM +0200, Greg Kurz wrote: > > > > > > The sPAPR machine always create a default PHB during initialization, even > > > if -nodefaults was passed on the command line. This forces the user to > > > rely on -global if she wants to set properties of the default PHB, such > > > as numa_node. > > > > > > This patch introduces a new machine create-default-phb property to control > > > whether the default PHB must be created or not. It defaults to on in order > > > to preserve old setups (which is also the motivation to not alter the > > > current behavior of -nodefaults). > > > > > > If create-default-phb is set to off, the default PHB isn't created, nor > > > any other device usually created with it. It is mandatory to provide > > > a PHB on the command line to be able to use PCI devices (otherwise QEMU > > > won't start). For example, the following creates a PHB with the same > > > mappings as the default PHB and also sets the NUMA affinity: > > > > > > -machine type=pseries,create-default-phb=off \ > > > -numa node,nodeid=0 -device spapr-pci-host-bridge,index=0,numa_node=0 > > > > So, I agree that the distinction between default devices that are > > disabled with -nodefaults and default devices that aren't is a big > > mess in qemu configuration. But on the other hand this only addresses > > one tiny aspect of that, and in the meantime means we will silently > > ignore some other configuration options in some conditions. > > > > So, what's the immediate benefit / use case for this? > > With the current code base, the only way to set properties of the default > PHB, is to pass -global spapr-pci-host-bridge.prop=value for each property. > The immediate benefit of this patch is to unify the way libvirt passes > PHB description to the command line: > > ie, do: > > -machine type=pseries,create-default-phb=off \ > -device spapr-pci-host-bridge,prop1=a,prop2=b,prop3=c \ > -device spapr-pci-host-bridge,prop1=d,prop2=e,prop3=f > > instead of: > > -machine type=pseries \ > -global spapr-pci-host-bridge.prop1=a \ > -global spapr-pci-host-bridge.prop2=b \ > -global spapr-pci-host-bridge.prop3=c \ > -device spapr-pci-host-bridge,prop1=d,prop2=e,prop3=f So, I'm thinking about this mostly in terms of NUMA nodes because that's the use case I'm aware of. The problem with using -global is not that it requires using a different syntax to set properties for the default PHB, but rather that such properties are then inherited by all other PHBs unless explicitly overridden. Not creating the default PHB at all would solve the issue. On the other hand, libvirt would then need to either 1) only allow setting NUMA nodes for PHBs if QEMU supports the new option, leaving QEMU < 2.10 users behind; or 2) implement handling for both the new and old behavior. I'm not sure we could get away with 1), and going for 2) means more work both for QEMU and libvirt developers for very little actual gain, so I'd be inclined to scrap this and just build the libvirt glue on top of the existing interface. That is, of course, unless 1) having a random selection of PHBs not assigned to any NUMA node is a sensible use case. This is something we just can't do reliably with the current interface: we can decide to set the NUMA node only for say, PHBs 1 and 3 leaving 0 and 2 alone, but once we set it for the default PHB we *have* to set it for all remaining ones as well. libvirt will by default assign emulated devices to the default PHB, so I would rather expect users to leave that one alone and set a NUMA node for all other PHBs; or 2) there are other properties outside of numa_node we might want to deal with; or 3) it turns out it's okay to require a recent QEMU :) -- Andrea Bolognani / Red Hat / Virtualization

4 4

[libvirt] [PATCH] docs: schema: make disk driver name attribute optional
by Jim Fehlig 19 Jul '17

19 Jul '17

/domain/devices/disk/driver/@name is not a required or mandatory attribute according to formatdomain, and indeed it was agreed on IRC that the attribute is "optional for input, recommended (but not required) for output". Currently the schema requires the attribute, causing virt-xml-validate to fail on disk config where the driver name is not explicitly specified. E.g. # cat test.xml | grep -A 5 cdrom <disk type='file' device='cdrom'> <driver type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> # virt-xml-validate test.xml Relax-NG validity error : Extra element devices in interleave test.xml:21: element devices: Relax-NG validity error : Element domain failed to validate content test.xml fails to validate Relaxing the name attribute to be optional fixes the validation # virt-xml-validate test.xml test.xml validates Signed-off-by: Jim Fehlig <jfehlig(a)suse.com> --- docs/schemas/domaincommon.rng | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/schemas/domaincommon.rng b/docs/schemas/domaincommon.rng index fc1a40f96..a49ce9303 100644 --- a/docs/schemas/domaincommon.rng +++ b/docs/schemas/domaincommon.rng @@ -1804,9 +1804,11 @@ </element> </define> <define name="driverFormat"> - <attribute name="name"> - <ref name="genericName"/> - </attribute> + <optional> + <attribute name="name"> + <ref name="genericName"/> + </attribute> + </optional> <optional> <attribute name='type'> <choice> -- 2.13.1

2 2

[libvirt] Xen device section defaults miss name='qemu'
by Christian Ehrhardt 19 Jul '17

19 Jul '17

Hi, we have found an issue caused by 321a28c6 "libxl: set default disk format in device post-parse" (and maybe more changes I couldn't identify yet). TL;DR: The auto-added driver section in xen changed and now causes issues. An example to to trigger: 1. Create a XEN xml with a disk device of type cdrom, but do not add a driver section, like. <disk type='file' device='cdrom'> <target dev='hdb' bus='ide'/> <readonly/> </disk> 2. on virsh define if that XML 2b. You could also "virsh edit" any working xml file, remove the <driver../> which will trigger the same 2c. You could also define via virt-manager as it does not explicitly specify the device section What happens is that before the changes this auto-added a driver section like: <driver name='qemu' type='raw'/> But now it does only add <driver type='raw'/> Which fails to verify like: Interestingly the same is not true for KVM, there the section is as it was in the past which made it more likely the post-parse step might be involved. Also rather expected, if one adds a full <driver name='qemu' type='raw'/> in the XML libvirt doesn't have to provide (broken) defaults and things work as they should - yet from a user with formerly working XMLs or using virt-manager this is a regression. I have beg your pardon I seem to not be experienced enough on this part of libvirt to provide a valid fix - I tried but nothing good came out so far. For now I reverted 321a28c6 which also affects a test later added by 4cd3f241. With those changes it will not add anything which lets us start Xen VMs for now, but it clearly isn't a solution. I wanted to ask you if you could take a look into this? -- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd

3 7

[libvirt] [PATCH] test/Makefile.am: drop WARN_CFLAGS from LDFLAGS
by Ján Tomko 19 Jul '17

19 Jul '17

Introduced by commit 0832c58, with the intention to link with the stack protector library. Another instance introduced by commit 4cbc15d which separated commandhelper_LDADD from LDADDS. Not needed because per commit 71b54636, automake should pass all the CFLAGS to the linker. --- tests/Makefile.am | 2 -- 1 file changed, 2 deletions(-) diff --git a/tests/Makefile.am b/tests/Makefile.am index 19986dc99..3d3038e62 100644 --- a/tests/Makefile.am +++ b/tests/Makefile.am @@ -76,7 +76,6 @@ GNULIB_LIBS = \ ../gnulib/lib/libgnu.la LDADDS = \ - $(WARN_CFLAGS) \ $(NO_INDIRECT_LDFLAGS) \ $(PROBES_O) \ $(GNULIB_LIBS) \ @@ -933,7 +932,6 @@ commandtest_LDADD = $(LDADDS) commandhelper_SOURCES = \ commandhelper.c commandhelper_LDADD = \ - $(WARN_CFLAGS) \ $(NO_INDIRECT_LDFLAGS) \ $(PROBES_O) \ ../src/libvirt_util.la \ -- 2.13.0

2 1

[libvirt] [RFC PATCH v2 0/4] numa: describe sibling nodes distances
by Wim Ten Have 18 Jul '17

18 Jul '17

From: Wim ten Have <wim.ten.have(a)oracle.com> This patch extents guest domain administration adding support to advertise node sibling distances when configuring HVM numa guests. NUMA (non-uniform memory access), a method of configuring a cluster of nodes within a single multiprocessing system such that it shares processor local memory amongst others improving performance and the ability of the system to be expanded. A NUMA system could be illustrated as shown below. Within this 4-node system, every socket is equipped with its own distinct memory. The whole typically resembles a SMP (symmetric multiprocessing) system being a "tightly-coupled," "share everything" system in which multiple processors are working under a single operating system and can access each others' memory over multiple "Bus Interconnect" paths. +-----+-----+-----+ +-----+-----+-----+ | M | CPU | CPU | | CPU | CPU | M | | E | | | | | | E | | M +- Socket0 -+ +- Socket3 -+ M | | O | | | | | | O | | R | CPU | CPU <---------> CPU | CPU | R | | Y | | | | | | Y | +-----+--^--+-----+ +-----+--^--+-----+ | | | Bus Interconnect | | | +-----+--v--+-----+ +-----+--v--+-----+ | M | | | | | | M | | E | CPU | CPU <---------> CPU | CPU | E | | M | | | | | | M | | O +- Socket1 -+ +- Socket2 -+ O | | R | | | | | | R | | Y | CPU | CPU | | CPU | CPU | Y | +-----+-----+-----+ +-----+-----+-----+ In contrast there is the limitation of a flat SMP system, not illustrated. Here, as sockets are added, the bus (data and address path), under high activity, gets overloaded and easily becomes a performance bottleneck. NUMA adds an intermediate level of memory shared amongst a few cores per socket as illustrated above, so that data accesses do not have to travel over a single bus. Unfortunately the way NUMA does this adds its own limitations. This, as visualized in the illustration above, happens when data is stored in memory associated with Socket2 and is accessed by a CPU (core) in Socket0. The processors use the "Bus Interconnect" to create gateways between the sockets (nodes) enabling inter-socket access to memory. These "Bus Interconnect" hops add data access delays when a CPU (core) accesses memory associated with a remote socket (node). For terminology we refer to sockets as "nodes" where access to each others' distinct resources such as memory make them "siblings" with a designated "distance" between them. A specific design is described under the ACPI (Advanced Configuration and Power Interface Specification) within the chapter explaining the system's SLIT (System Locality Distance Information Table). These patches extend core libvirt's XML description of a virtual machine's hardware to include NUMA distance information for sibling nodes, which is then passed to Xen guests via libxl. Recently qemu landed support for constructing the SLIT since commit 0f203430dd ("numa: Allow setting NUMA distance for different NUMA nodes"), hence these core libvirt extensions can also help other drivers in supporting this feature. The XML changes made allow to describe the <cell> (or node/sockets) <distances> amongst <sibling> node identifiers and propagate these towards the numa domain functionality finally adding support to libxl. [below is an example illustrating a 4 node/socket <cell> setup] <cpu> <numa> <cell id='0' cpus='0,4-7' memory='2097152' unit='KiB'> <distances> <sibling id='0' value='10'/> <sibling id='1' value='21'/> <sibling id='2' value='31'/> <sibling id='3' value='41'/> </distances> </cell> <cell id='1' cpus='1,8-10,12-15' memory='2097152' unit='KiB'> <distances> <sibling id='0' value='21'/> <sibling id='1' value='10'/> <sibling id='2' value='21'/> <sibling id='3' value='31'/> </distances> </cell> <cell id='2' cpus='2,11' memory='2097152' unit='KiB'> <distances> <sibling id='0' value='31'/> <sibling id='1' value='21'/> <sibling id='2' value='10'/> <sibling id='3' value='21'/> </distances> </cell> <cell id='3' cpus='3' memory='2097152' unit='KiB'> <distances> <sibling id='0' value='41'/> <sibling id='1' value='31'/> <sibling id='2' value='21'/> <sibling id='3' value='10'/> </distances> </cell> </numa> </cpu> By default on libxl, if no <distances> are given to describe the SLIT data between different <cell>s, this patch will default to a scheme using 10 for local and 21 for any remote node/socket, which is the assumption of guest OS when no SLIT is specified. While SLIT is optional, libxl requires that distances are set nonetheless. On Linux systems the SLIT detail can be listed with help of the 'numactl -H' command. An above HVM guest as described would on such prompt with below output. [root@f25 ~]# numactl -H available: 4 nodes (0-3) node 0 cpus: 0 4 5 6 7 node 0 size: 1988 MB node 0 free: 1743 MB node 1 cpus: 1 8 9 10 12 13 14 15 node 1 size: 1946 MB node 1 free: 1885 MB node 2 cpus: 2 11 node 2 size: 2011 MB node 2 free: 1912 MB node 3 cpus: 3 node 3 size: 2010 MB node 3 free: 1980 MB node distances: node 0 1 2 3 0: 10 21 31 41 1: 21 10 21 31 2: 31 21 10 21 3: 41 31 21 10 Wim ten Have (4): numa: describe siblings distances within cells libxl: vnuma support xenconfig: add domxml conversions for xen-xl xlconfigtest: add tests for numa cell sibling distances docs/formatdomain.html.in | 64 ++++- docs/schemas/basictypes.rng | 8 + docs/schemas/cputypes.rng | 18 ++ src/conf/cpu_conf.c | 2 +- src/conf/numa_conf.c | 260 +++++++++++++++++- src/conf/numa_conf.h | 25 +- src/libvirt_private.syms | 6 + src/libxl/libxl_conf.c | 124 +++++++++ src/xenconfig/xen_xl.c | 303 +++++++++++++++++++++ .../test-fullvirt-vnuma-nodistances.cfg | 26 ++ .../test-fullvirt-vnuma-nodistances.xml | 53 ++++ tests/xlconfigdata/test-fullvirt-vnuma.cfg | 26 ++ tests/xlconfigdata/test-fullvirt-vnuma.xml | 81 ++++++ tests/xlconfigtest.c | 4 + 14 files changed, 993 insertions(+), 7 deletions(-) create mode 100644 tests/xlconfigdata/test-fullvirt-vnuma-nodistances.cfg create mode 100644 tests/xlconfigdata/test-fullvirt-vnuma-nodistances.xml create mode 100644 tests/xlconfigdata/test-fullvirt-vnuma.cfg create mode 100644 tests/xlconfigdata/test-fullvirt-vnuma.xml -- 2.9.4

3 9

[libvirt] [PATCH 0/2] Fix a couple issues found w/ vHBA logic
by John Ferlan 18 Jul '17

18 Jul '17

Patch 1 fixes something seen whilst working through patch 2. Long description in patch 2 describes the problem. John Ferlan (2): storage: Alter check for default managed setting conf: Fix vHBA checkParent logic for pool creation src/conf/node_device_conf.c | 50 ++++++++++++++++++++++++++++++++------ src/storage/storage_backend_scsi.c | 6 ++--- 2 files changed, 45 insertions(+), 11 deletions(-) -- 2.9.4

2 5