October 2016 - Devel - libvirt List Archives

[libvirt] RFC: Exposing "ready" bool (of `query-block-jobs`) or QMP BLOCK_JOB_READY event
by Kashyap Chamarthy 08 Oct '16

08 Oct '16

Backround --------- For QEMU block device jobs, the "ready" boolean field (part of QMP `query-block-jobs`) was introduced in commit ef6dbf1 (available in QEMU v2.2.0 or above): http://git.qemu.org/?p=qemu.git;a=commitdiff;h=ef6dbf1e4 -- blockjob: Add "ready" field "When a block job signals readiness, this is currently reported only through QMP. If qemu wants to use block jobs for internal tasks, there needs to be another way to correctly detect when a block job may be completed. For this reason, introduce a bool "ready" which is set when the block job may be completed." And, libvirt was fixed to use the above field in this commit (available in libvirt v1.2.18 or above): http://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=eae5924 -- qemu: Update state of block job to READY only if it actually is ready RFC --- Currently libvirt block APIs (& consequently higher-level applications like Nova which use these APIs) rely on polling for job completion via virDomainGetBlockJobInfo(), which uses QMP `query-block-jobs`, and waits for QEMU to report "offset" == "len", which translates to libvirt "cur" == "end". Based on this, libvirt can take an action (whether to gracefully abort, or pivot to the copy in case of a COPY job). Since QEMU reports the "ready": true field (followed by a BLOCK_JOB_READY QMP event). It would be helpful if libvirt expose this via an API, so upper layers could instead use that, rather than polling. Problem scenario ---------------- When virDomainBlockRebase() is invoked to start a copy job, then aborting the said copy operation with virDomainBlockJobAbort() + flag VIR_DOMAIN_BLOCK_JOB_ABORT_PIVOT can result in a potential race condition (due to the way the virDomainGetBlockJobInfo() reports the job status) where the pivot operation fails. Race condition window ~~~~~~~~~~~~~~~~~~~~~ libvirt finds cur==end AND sends a pivot request, all in the window before QEMU would have sent "ready": true field [emitted as part of the QMP `query-block-jobs` command's response, indicating that the job has actually completed], however the pivot request fails because it requires "ready": true. So Eric Blake suggests: QEMU 2.0 or 1.x probably had a synchronous setup where you could never observer cur==end on a non-ready job. But I don't remember enough history to point to when QEMU switched jobs to be a bit more asynchronous. Maybe there was no qemu regression - maybe it was BECAUSE of other block-job additions in 2.2 that offset==len was no longer reliable. I don't know that for sure. But what it DOES sound like is that IF qemu reports "ready": false, offset==len is not reliable, and libvirt should be taught to fudge that. And hopefully, QEMU too old to report "ready:" at all is reliable with regards to offset==len, because that's all we have to go by. For now, I filed this upstream libvirt bug: https://bugzilla.redhat.com/show_bug.cgi?id=1382165 -- virDomainGetBlockJobInfo: Adjust job reporting based on QEMU stats & the "ready" field of `query-block-jobs` However, exposing the "ready" boolean from QMP `query-block-jobs` might be worth considering. -- /kashyap

4 8

[libvirt] [PATCH 0/3] Clean up some perf event issues
by John Ferlan 07 Oct '16

07 Oct '16

Resolve a couple of recently filed issues regarding perf events. Details in each patch. John Ferlan (3): docs: Alter descriptions of perf cpu_cycles util: Clear up some perf error messages util: Check/ignore already disabled event docs/formatdomain.html.in | 2 +- src/libvirt-domain.c | 2 +- src/util/virperf.c | 13 ++++++++----- src/util/virperf.h | 2 +- tools/virsh.pod | 9 +++++---- 5 files changed, 16 insertions(+), 12 deletions(-) -- 2.7.4

2 8

[libvirt] [libvirt-perl PATCH 0/2] Add new flags
by John Ferlan 07 Oct '16

07 Oct '16

Patch 1 is related to the on list iotune length/duration parameter Patch 2 is related to the already pushed vcpu hotpluggable flag These would be pushed once I push the blkdeviotune changes - the order was only because I made the blkdeviotune changes, then built and found that the vcpu hotpluggable flag cause test failure. John Ferlan (2): Add blkdeviotune length parameters Add VIR_DOMAIN_VCPU_HOTPLUGGABLE Changes | 3 ++- Virt.xs | 13 +++++++++++++ lib/Sys/Virt/Domain.pm | 29 +++++++++++++++++++++++++++++ 3 files changed, 44 insertions(+), 1 deletion(-) -- 2.7.4

2 5

[libvirt] PCP libvirt Plugin
by Marko Myllynen 07 Oct '16

07 Oct '16

Hi, FYI, I've written a PCP plugin (PMDA in PCP parlance) to support most hypervisor / domain information and metrics available over the libvirt Python API, it's up to date as of libvirt 2.3 (so it already supports the recently added perf event metrics). In case you haven't heard about PCP, here's a brief intro: The Performance Co-Pilot (PCP, http://www.pcp.io/) system is a toolkit for collecting, archiving, and processing performance metrics. It supports system and application metrics from local and remote hosts, archives, and containers. A typical Linux PCP installation offers over 1,000 metrics by default and is easily extensible with its own plugins, or PMDAs ("Performance Metrics Domain Agents"). In addition to very complete /proc based statistics, readily available PCP PMDAs provide support components like 389 Directory Server, Apache, Ceph, GFS2, Gluster, MySQL, NFS, Oracle, Postfix, PostgreSQL, Samba, and Sendmail, among others. PCP also runs on many platforms, including Linux, Mac OS X, FreeBSD, NetBSD, Solaris, and Windows. And here's Quick Guide to get started with PCP: http://www.pcp.io/docs/guide.html (So PCP comes with handy clients like pmrep(1) for custom metrics reporting, sophisticated logging and inferencing tools, and support for Grafana based Web UI.) The libvirt metrics available with PCP libvirt PMDA and their descriptions are listed below, they include all the recently added perf event metrics as well as combined and per-device metrics for each VM. The plugin is available for Fedora in updates-testing and for Ubuntu from https://bintray.com/pcp/ . As this is a recent addition, the plugin or some of the later additions like per-device metrics have not propagated to all distributions yet. For those using PCP already, they may also manually grab the needed files from the below page without the need to upgrade any PCP packages: https://myllynen.fedorapeople.org/pmdalibvirt/ Upstream code lives here: http://oss.sgi.com/cgi-bin/gitweb.cgi?p=pcp/pcp.git;a=tree;f=src/pmdas/libv… I wonder could this be mentioned at https://libvirt.org/apps.html ? Here's the list of the PCP metrics available with the current git code: libvirt.dominfo.container [VM container] libvirt.dominfo.description [VM description] libvirt.dominfo.memory.boot [VM memory, at boot] libvirt.dominfo.memory.current [VM memory, current] libvirt.dominfo.memory.max [VM memory, maximum] libvirt.dominfo.name [VM name] libvirt.dominfo.os.type [VM OS type] libvirt.dominfo.title [VM title] libvirt.dominfo.type [VM type] libvirt.dominfo.uuid [VM UUID] libvirt.dominfo.vcpu.current [VM vCPUs, current] libvirt.dominfo.vcpu.max [VM vCPUs, maximum] libvirt.domstats.balloon.current [VM balloon size, current] libvirt.domstats.balloon.maximum [VM balloon size, maximum] libvirt.domstats.block.all.allocation [VM backing imgs, total allocation] libvirt.domstats.block.all.backingIndex [VM block devs, backing chain imgs] libvirt.domstats.block.all.capacity [VM backing imgs, total capacity] libvirt.domstats.block.all.fl.reqs [VM block devs, total fl reqs] libvirt.domstats.block.all.fl.times [VM block devs, total fl times] libvirt.domstats.block.all.name [VM block devs, all names] libvirt.domstats.block.all.path [VM block devs, all paths] libvirt.domstats.block.all.physical [VM backing imgs, total physical] libvirt.domstats.block.all.rd.bytes [VM block devs, total rd bytes] libvirt.domstats.block.all.rd.reqs [VM block devs, total rd reqs] libvirt.domstats.block.all.rd.times [VM block devs, total rd times] libvirt.domstats.block.all.wr.bytes [VM block devs, total wr bytes] libvirt.domstats.block.all.wr.reqs [VM block devs, total wr reqs] libvirt.domstats.block.all.wr.times [VM block devs, total wr times] libvirt.domstats.block.allocation [VM backing img, allocation] libvirt.domstats.block.backingIndex [VM block dev, backing chain img] libvirt.domstats.block.capacity [VM backing img, capacity] libvirt.domstats.block.count [VM block devs, count] libvirt.domstats.block.fl.reqs [VM block dev, fl reqs] libvirt.domstats.block.fl.times [VM block dev, fl times] libvirt.domstats.block.name [VM block dev, name] libvirt.domstats.block.path [VM block dev, path] libvirt.domstats.block.physical [VM backing img, physical] libvirt.domstats.block.rd.bytes [VM block dev, rd bytes] libvirt.domstats.block.rd.reqs [VM block dev, rd reqs] libvirt.domstats.block.rd.times [VM block dev, rd times] libvirt.domstats.block.wr.bytes [VM block dev, wr bytes] libvirt.domstats.block.wr.reqs [VM block dev, wr reqs] libvirt.domstats.block.wr.times [VM block dev, wr times] libvirt.domstats.cpu.system [VM CPU time, system] libvirt.domstats.cpu.time [VM CPU time, total] libvirt.domstats.cpu.user [VM CPU time, user] libvirt.domstats.mem.actual [VM memory, balloon size] libvirt.domstats.mem.available [VM memory, available] libvirt.domstats.mem.major_fault [VM memory, major faults] libvirt.domstats.mem.minor_fault [VM memory, minor faults] libvirt.domstats.mem.rss [VM memory, proc RSS] libvirt.domstats.mem.swap_in [VM memory, swapped in total] libvirt.domstats.mem.swap_out [VM memory, swapped out total] libvirt.domstats.mem.unused [VM memory, unused] libvirt.domstats.net.all.name [VM NICs, all names] libvirt.domstats.net.all.rx.bytes [VM NICs, total rx bytes] libvirt.domstats.net.all.rx.drop [VM NICs, total rx drop] libvirt.domstats.net.all.rx.errs [VM NICs, total rx errs] libvirt.domstats.net.all.rx.pkts [VM NICs, total rx pkts] libvirt.domstats.net.all.tx.bytes [VM NICs, total tx bytes] libvirt.domstats.net.all.tx.drop [VM NICs, total tx drop] libvirt.domstats.net.all.tx.errs [VM NICs, total tx errs] libvirt.domstats.net.all.tx.pkts [VM NICs, total tx pkts] libvirt.domstats.net.count [VM NICs, count] libvirt.domstats.net.name [VM NIC, name] libvirt.domstats.net.rx.bytes [VM NIC, rx bytes] libvirt.domstats.net.rx.drop [VM NIC, rx drop] libvirt.domstats.net.rx.errs [VM NIC, rx errs] libvirt.domstats.net.rx.pkts [VM NIC, rx pkts] libvirt.domstats.net.tx.bytes [VM NIC, tx bytes] libvirt.domstats.net.tx.drop [VM NIC, tx drop] libvirt.domstats.net.tx.errs [VM NIC, tx errs] libvirt.domstats.net.tx.pkts [VM NIC, tx pkts] libvirt.domstats.perf.cache_misses [VM perf stats, cache misses] libvirt.domstats.perf.cache_references [VM perf stats, cache references] libvirt.domstats.perf.cmt [VM perf stats, cmt] libvirt.domstats.perf.cpu_cycles [VM perf stats, CPU cycles/instr] libvirt.domstats.perf.instructions [VM perf stats, instructions count] libvirt.domstats.perf.mbml [VM perf stats, mbml] libvirt.domstats.perf.mbmt [VM perf stats, mbmt] libvirt.domstats.vcpu.all.state [VM vCPUs, total state] libvirt.domstats.vcpu.all.time [VM vCPUs, total time] libvirt.domstats.vcpu.all.wait [VM vCPUs, total wait] libvirt.domstats.vcpu.current [VM vCPUs, current] libvirt.domstats.vcpu.maximum [VM vCPUs, maximum] libvirt.domstats.vcpu.state [VM vCPU, state] libvirt.domstats.vcpu.time [VM vCPU, time] libvirt.domstats.vcpu.wait [VM vCPU, wait] libvirt.hv.devices [Libvirt devices] libvirt.hv.domains.active [Libvirt domains, active] libvirt.hv.domains.inactive [Libvirt domains, inactive] libvirt.hv.driver [Libvirt driver] libvirt.hv.interfaces.active [Libvirt interfaces, active] libvirt.hv.interfaces.inactive [Libvirt interfaces, inactive] libvirt.hv.networkfilters [Libvirt network filters] libvirt.hv.networks.active [Libvirt networks, active] libvirt.hv.networks.inactive [Libvirt networks, inactive] libvirt.hv.secrets [Libvirt secrets] libvirt.hv.storagepools.active [Libvirt storage pools, active] libvirt.hv.storagepools.inactive [Libvirt storage pools, inactive] libvirt.hv.uri [Libvirt URI] libvirt.hv.version [Libvirt version] Thanks, -- Marko Myllynen

3 4

[libvirt] [PATCH] virCgroupDetect: Print pid as long long
by Michal Privoznik 07 Oct '16

07 Oct '16

Pids are signed, report them as such. Before: Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine/qemu-4-fedora.libvirt-qemu/vcpu0 for pid 18446744073709551615 After: Detected mount/mapping 2:cpuset at /sys/fs/cgroup/cpuset in /machine/qemu-4-fedora.libvirt-qemu/vcpu0 for pid -1 Signed-off-by: Michal Privoznik <mprivozn(a)redhat.com> --- src/util/vircgroup.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/util/vircgroup.c b/src/util/vircgroup.c index 8b52816..0a06a8a 100644 --- a/src/util/vircgroup.c +++ b/src/util/vircgroup.c @@ -732,11 +732,12 @@ virCgroupDetect(virCgroupPtr group, return -1; } - VIR_DEBUG("Detected mount/mapping %zu:%s at %s in %s for pid %llu", i, + VIR_DEBUG("Detected mount/mapping %zu:%s at %s in %s for pid %lld", + i, virCgroupControllerTypeToString(i), group->controllers[i].mountPoint, group->controllers[i].placement, - (unsigned long long)pid); + (long long) pid); } return 0; -- 2.8.4

3 3

[libvirt] [PATCH] private_syms: add virLogFilterListFree to libvirt_private.syms
by Erik Skultety 07 Oct '16

07 Oct '16

Commit 660468b1 forgot to add it, so let's add it now to prevent future linker issues. Signed-off-by: Erik Skultety <eskultet(a)redhat.com> --- Pushed under trivial rule src/libvirt_private.syms | 1 + 1 file changed, 1 insertion(+) diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index c4af57d..923afd1 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -1872,6 +1872,7 @@ virLockSpaceReleaseResourcesForOwner; virLogDefineFilter; virLogDefineOutput; virLogFilterFree; +virLogFilterListFree; virLogGetDefaultPriority; virLogGetFilters; virLogGetNbFilters; -- 2.5.5

1 0

[libvirt] [PATCH] qemu_process: add pid of vm in domain log
by Chen Hanxiao 06 Oct '16

06 Oct '16

From: Chen Hanxiao <chenhanxiao(a)gmail.com> Add pid of VM in domain log. We used to show this info in debug log. For example: If a process send SIGKILL to a qemu process, we could find something in audit logs. Then the pid of VM in domain log will be helpful. Signed-off-by: Chen Hanxiao <chenhanxiao(a)gmail.com> --- src/qemu/qemu_process.c | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 27d04a4..8510a89 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -5392,11 +5392,14 @@ qemuProcessLaunch(virConnectPtr conn, _("Domain %s didn't show up"), vm->def->name); rv = -1; } - VIR_DEBUG("QEMU vm=%p name=%s running with pid=%llu", - vm, vm->def->name, (unsigned long long)vm->pid); + qemuDomainLogContextWrite(logCtxt, + "QEMU vm=%p name=%s running with pid=%llu\n", + vm, + vm->def->name, + (unsigned long long)vm->pid); } else { - VIR_DEBUG("QEMU vm=%p name=%s failed to spawn", - vm, vm->def->name); + qemuDomainLogContextWrite(logCtxt, "QEMU vm=%p name=%s failed to spawn", + vm, vm->def->name); } VIR_DEBUG("Writing early domain status to disk"); -- 1.8.3.1

3 2

[libvirt] [PATCH] m4: Drop PKG_PROG_PKG_CONFIG compatibility code
by Andrea Bolognani 06 Oct '16

06 Oct '16

This was needed for RHEL 4 vintage distributions, which we haven't supported for a long time now. --- m4/virt-pkgconfig-back-compat.m4 | 23 ----------------------- 1 file changed, 23 deletions(-) delete mode 100644 m4/virt-pkgconfig-back-compat.m4 diff --git a/m4/virt-pkgconfig-back-compat.m4 b/m4/virt-pkgconfig-back-compat.m4 deleted file mode 100644 index 7eab84b..0000000 --- a/m4/virt-pkgconfig-back-compat.m4 +++ /dev/null @@ -1,23 +0,0 @@ -dnl -dnl To support the old pkg-config from RHEL4 vintage, we need -dnl to define the PKG_PROG_PKG_CONFIG macro if its not already -dnl present -m4_ifndef([PKG_PROG_PKG_CONFIG], - [AC_DEFUN([PKG_PROG_PKG_CONFIG], -[m4_pattern_forbid([^_?PKG_[A-Z_]+$]) -m4_pattern_allow([^PKG_CONFIG(_PATH)?$]) -AC_ARG_VAR([PKG_CONFIG], [path to pkg-config utility])dnl -if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then - AC_PATH_TOOL([PKG_CONFIG], [pkg-config]) -fi -if test -n "$PKG_CONFIG"; then - _pkg_min_version=m4_default([$1], [0.9.0]) - AC_MSG_CHECKING([pkg-config is at least version $_pkg_min_version]) - if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then - AC_MSG_RESULT([yes]) - else - AC_MSG_RESULT([no]) - PKG_CONFIG="" - fi -fi[]dnl -])]) -- 2.7.4

2 1

[libvirt] Question about QEMU + hugepages + NUMA memory + migration
by Sam Bobroff 06 Oct '16

06 Oct '16

Hi libvirt people, I've been looking at a (probable) bug and I'm not sure how to progress. The situation is a bit complicated and involves both QEMU and libvirt (and I think it may have been looked at already) so I would really appreciate some advice on how to approach it. I'm using a pretty recent master version of libvirt from git and I'm testing on a ppc64le host with a similar guest but this doesn't seem to be arch-specific. If I create a QEMU guest (e.g. via virt-install) that requests both hugepage backing on the host and NUMA memory placement on the host, the NUMA placement seems to be ignored. If I do: # echo 0 > /proc/sys/vm/nr_hugepages # echo 512 > /sys/devices/system/node/node0/hugepages/hugepages-16384kB/nr_hugepages # virt-install --name tmp --memory=4096 --graphics none --memorybacking hugepages=yes --disk none --import --wait 0 --numatune=8 ... then hugepages are allocated on node 0 and the machine starts successfully, which seems like a bug. I believe it should fail to start due to insufficient memory, and in fact that is what happens if cgroup support isn't detected in the host: there seems to be a fall-back path in libvirt (probably using mbind()) that works as I would expect. Note: the relevant part of the guest XML seems to be this: »·······<memoryBacking> »·······»·······<hugepages/> »·······</memoryBacking> »·······<numatune> »·······»·······<memory mode='strict' nodeset='8'/> »·······</numatune> It seems fairly clear what is happening: although QEMU is capable of allocating hugepages on specific NUMA nodes (using "memory-backend-file") libvirt is not passing those options to QEMU in this situation. I investigated this line of reasoning and if I hack libvirt to pass those options to QEMU it does indeed fix the problem... but it renders the machine state migration-incompatible with unfixed versions. This seems to have been why this hasn't been fixed already :-( So what can we do? I assume it's not acceptible to just break migration with a bugfix, and I can only think of two ways to fix migration: (a) Add a new flag to the XML, and for guests without the flag, maintain the old buggy behaviour (and therefore migration compatability). (b) Hack QEMU so that migration can succeed between un-fixed and fixed versions. (And possibly also in the reverse direction?) I don't like (a) because it's visible in the XML, and would have to be carried forever (or at least a long time?). I don't really like (b) either because it's tricky, and even if it could be made to work reliably, it would add mess and risk to the migration code. I'm not sure how the QEMU community would feel about it either. However, I did hack up some code and it worked at least in some simple cases. Can anyone see a better approach? Is anyone already working on this? Thanks, Sam.

2 5

[libvirt] [PATCH] virt-yajl: Fix detection of yajl requirements
by Martin Kletzander 06 Oct '16

06 Oct '16

Running the output of qemu -help doesn't make any sense. We should be looking for libvirt being mentioned in the output. This worked by accident, let's make it work as expected it to. Signed-off-by: Martin Kletzander <mkletzan(a)redhat.com> --- m4/virt-yajl.m4 | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/m4/virt-yajl.m4 b/m4/virt-yajl.m4 index adf2819eeb66..8c452adca653 100644 --- a/m4/virt-yajl.m4 +++ b/m4/virt-yajl.m4 @@ -26,7 +26,7 @@ AC_DEFUN([LIBVIRT_CHECK_YAJL],[ AC_PATH_PROGS([QEMU], [qemu-kvm qemu kvm qemu-system-x86_64], [], [$PATH:/usr/bin:/usr/libexec]) if test -x "$QEMU"; then - if `$QEMU -help | grep libvirt` >/dev/null; then + if $QEMU -help 2>/dev/null | grep -q libvirt; then with_yajl=yes else [qemu_version_sed='s/.*ersion $[0-9.,]*$.*/\1/'] -- 2.10.1

2 1