[libvirt] [PATCH v7 0/7] qemu: expand domain memory statistics

From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> QEMU reports timestamp and available along with other memory statistics. This information was not saved into domain statistics. 1st to 3rd patches add new stats to reports. Also, to collect all balloon statistics for all guests it was necessary to make several libvirt requests (one per VE).. 4th and 5th patch allows doing this via qemuConnectGetAllDomainStats in one request. 6th patch speed-ups DomainMemoryStats by caching so that requests would not gain much additional overhead from previous patches. 7th patch adds probbing and can be dropped. Changes since v1: * Enum numeration fixed * Macro getting "usage" field fixed Changes since v2: * Previous patches were on wrong branch * qemu's stat name was "stat-available-memory" Changes since v3: * domstats patch added Changes since v4: * Formatted and rephrased commit messages * Fixed libvirt crash, caused by simultaneous incorrect QUERY job execution Changes since v5: * Updated virsh.pod * Splitted patch about domstats into 2 patches * Do not report balloon.current as balloon.actual Changes since v6: * caching added for DomainMemoryStats Derbyshev Dmitry (5): virsh: Add balloon stats description to .pod qemu: expand domain memory statistics with 'usable' qemu: expand domain memory statistics with 'last-update' timestamp qemu: split qemuDomainMemoryStats into internal and external functions qemu: return balloon statistics when all domain statistics reported Igor Redko (2): qemu: add cache for DomainMemoryStats qemu-probes: add probes for cache MemoryStats cache include/libvirt/libvirt-domain.h | 11 ++- src/libvirt-domain.c | 5 ++ src/libvirt_qemu_probes.d | 4 + src/qemu/qemu_domain.h | 6 ++ src/qemu/qemu_driver.c | 187 +++++++++++++++++++++++++++++++++++---- src/qemu/qemu_monitor_json.c | 24 +++-- tools/virsh-domain-monitor.c | 4 + tools/virsh.pod | 29 +++++- 8 files changed, 242 insertions(+), 28 deletions(-) -- 1.9.5.msysgit.0

From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> Description for existing balloon stats was missing for dommemstat. Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- tools/virsh.pod | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/tools/virsh.pod b/tools/virsh.pod index d7cd10e..4ebf7ad 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -806,7 +806,19 @@ on hypervisor. Get memory stats for a running domain. -Depending on the hypervisor a variety of statistics can be returned +Availability of these fields depends on hypervisor. Unsupported fields are +missing from the output. Other fields may appear if communicating with a newer +version of libvirtd. + +B<Explanation of fields>: + swap_in - The amount of data read from swap space (in kB) + swap_out - The amount of memory written out to swap space (in kB) + major_fault - The number of page faults then disk IO was required + minor_fault - The number of other page faults + unused - The amount of memory left unused by the system (in kB) + available - The amount of usable memory as seen by the domain (in kB) + actual - Current balloon value (in KB) + rss - Resident Set Size of the running domain's process (in kB) For QEMU/KVM with a memory balloon, setting the optional I<--period> to a value larger than 0 in seconds will allow the balloon driver to return -- 1.9.5.msysgit.0

On Wed, Jul 13, 2016 at 01:42:12PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com>
Description for existing balloon stats was missing for dommemstat.
Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- tools/virsh.pod | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-)
diff --git a/tools/virsh.pod b/tools/virsh.pod index d7cd10e..4ebf7ad 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -806,7 +806,19 @@ on hypervisor.
Get memory stats for a running domain.
-Depending on the hypervisor a variety of statistics can be returned +Availability of these fields depends on hypervisor. Unsupported fields are +missing from the output. Other fields may appear if communicating with a newer +version of libvirtd. + +B<Explanation of fields>: + swap_in - The amount of data read from swap space (in kB) + swap_out - The amount of memory written out to swap space (in kB) + major_fault - The number of page faults then disk IO was required
s/then/where/
+ minor_fault - The number of other page faults + unused - The amount of memory left unused by the system (in kB) + available - The amount of usable memory as seen by the domain (in kB) + actual - Current balloon value (in KB) + rss - Resident Set Size of the running domain's process (in kB)
For QEMU/KVM with a memory balloon, setting the optional I<--period> to a value larger than 0 in seconds will allow the balloon driver to return

From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> Currently 'memtotal' in virtio drivers and qemu corresponds to 'available' in libvirt. Because of that we introduce libvirt 'usable' parameter, which maps to 'stat-available-memory' balloon statistics. As balloon statistics isn't reported in hmp, so no modification is made in qemu_monitor_text.c. Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- include/libvirt/libvirt-domain.h | 8 +++++++- src/libvirt-domain.c | 3 +++ src/qemu/qemu_monitor_json.c | 4 ++++ tools/virsh-domain-monitor.c | 2 ++ tools/virsh.pod | 2 ++ 5 files changed, 18 insertions(+), 1 deletion(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 7ea93aa..3fb482b 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -604,10 +604,16 @@ typedef enum { VIR_DOMAIN_MEMORY_STAT_RSS = 7, /* + * How big the balloon can be inflated without pushing the guest system + * to swap, corresponds to 'Available' in /proc/meminfo + */ + VIR_DOMAIN_MEMORY_STAT_USABLE = 8, + + /* * The number of statistics supported by this version of the interface. * To add new statistics, add them to the enum and increase this value. */ - VIR_DOMAIN_MEMORY_STAT_NR = 8, + VIR_DOMAIN_MEMORY_STAT_NR = 9, # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_MEMORY_STAT_LAST = VIR_DOMAIN_MEMORY_STAT_NR diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 4e71a94..1467030 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -5986,6 +5986,9 @@ virDomainGetInterfaceParameters(virDomainPtr domain, * The amount of memory which is not being used for any purpose (in kb). * VIR_DOMAIN_MEMORY_STAT_AVAILABLE: * The total amount of memory available to the domain's OS (in kb). + * VIR_DOMAIN_MEMORY_STAT_USABLE: + * How big the balloon can be inflated without pushing the guest system + * to swap, corresponds to 'Available' in /proc/meminfo * VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON: * Current balloon value (in kb). * diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index bb426dc..c83e0cc 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -1719,7 +1719,11 @@ int qemuMonitorJSONGetMemoryStats(qemuMonitorPtr mon, VIR_DOMAIN_MEMORY_STAT_UNUSED, 1024); GET_BALLOON_STATS("stat-total-memory", VIR_DOMAIN_MEMORY_STAT_AVAILABLE, 1024); + GET_BALLOON_STATS("stat-available-memory", + VIR_DOMAIN_MEMORY_STAT_USABLE, 1024); + ret = got; + cleanup: virJSONValueFree(cmd); virJSONValueFree(reply); diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index c712fa5..7f30da2 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -369,6 +369,8 @@ cmdDomMemStat(vshControl *ctl, const vshCmd *cmd) vshPrint(ctl, "unused %llu\n", stats[i].val); if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_AVAILABLE) vshPrint(ctl, "available %llu\n", stats[i].val); + if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_USABLE) + vshPrint(ctl, "usable %llu\n", stats[i].val); if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON) vshPrint(ctl, "actual %llu\n", stats[i].val); if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_RSS) diff --git a/tools/virsh.pod b/tools/virsh.pod index 4ebf7ad..5444f91 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -819,6 +819,8 @@ B<Explanation of fields>: available - The amount of usable memory as seen by the domain (in kB) actual - Current balloon value (in KB) rss - Resident Set Size of the running domain's process (in kB) + usable - The amount of memory which can be reclaimed by balloon +without causing host swapping (in KB) For QEMU/KVM with a memory balloon, setting the optional I<--period> to a value larger than 0 in seconds will allow the balloon driver to return -- 1.9.5.msysgit.0

On Wed, Jul 13, 2016 at 01:42:13PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com>
Currently 'memtotal' in virtio drivers and qemu corresponds to 'available' in libvirt. Because of that we introduce libvirt 'usable' parameter, which maps to 'stat-available-memory' balloon statistics. As balloon statistics isn't reported in hmp, so no modification is made in qemu_monitor_text.c.
Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- include/libvirt/libvirt-domain.h | 8 +++++++- src/libvirt-domain.c | 3 +++ src/qemu/qemu_monitor_json.c | 4 ++++ tools/virsh-domain-monitor.c | 2 ++ tools/virsh.pod | 2 ++ 5 files changed, 18 insertions(+), 1 deletion(-)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 7ea93aa..3fb482b 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -604,10 +604,16 @@ typedef enum { VIR_DOMAIN_MEMORY_STAT_RSS = 7,
/* + * How big the balloon can be inflated without pushing the guest system
s/big/much/
+ * to swap, corresponds to 'Available' in /proc/meminfo + */ + VIR_DOMAIN_MEMORY_STAT_USABLE = 8, + + /* * The number of statistics supported by this version of the interface. * To add new statistics, add them to the enum and increase this value. */ - VIR_DOMAIN_MEMORY_STAT_NR = 8, + VIR_DOMAIN_MEMORY_STAT_NR = 9,
# ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_MEMORY_STAT_LAST = VIR_DOMAIN_MEMORY_STAT_NR diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 4e71a94..1467030 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -5986,6 +5986,9 @@ virDomainGetInterfaceParameters(virDomainPtr domain, * The amount of memory which is not being used for any purpose (in kb). * VIR_DOMAIN_MEMORY_STAT_AVAILABLE: * The total amount of memory available to the domain's OS (in kb). + * VIR_DOMAIN_MEMORY_STAT_USABLE: + * How big the balloon can be inflated without pushing the guest system
s/big/much/
+ * to swap, corresponds to 'Available' in /proc/meminfo * VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON: * Current balloon value (in kb). * diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index bb426dc..c83e0cc 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -1719,7 +1719,11 @@ int qemuMonitorJSONGetMemoryStats(qemuMonitorPtr mon, VIR_DOMAIN_MEMORY_STAT_UNUSED, 1024); GET_BALLOON_STATS("stat-total-memory", VIR_DOMAIN_MEMORY_STAT_AVAILABLE, 1024); + GET_BALLOON_STATS("stat-available-memory", + VIR_DOMAIN_MEMORY_STAT_USABLE, 1024); + ret = got; + cleanup: virJSONValueFree(cmd); virJSONValueFree(reply); diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index c712fa5..7f30da2 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -369,6 +369,8 @@ cmdDomMemStat(vshControl *ctl, const vshCmd *cmd) vshPrint(ctl, "unused %llu\n", stats[i].val); if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_AVAILABLE) vshPrint(ctl, "available %llu\n", stats[i].val); + if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_USABLE) + vshPrint(ctl, "usable %llu\n", stats[i].val); if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON) vshPrint(ctl, "actual %llu\n", stats[i].val); if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_RSS) diff --git a/tools/virsh.pod b/tools/virsh.pod index 4ebf7ad..5444f91 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -819,6 +819,8 @@ B<Explanation of fields>: available - The amount of usable memory as seen by the domain (in kB) actual - Current balloon value (in KB) rss - Resident Set Size of the running domain's process (in kB) + usable - The amount of memory which can be reclaimed by balloon
trailing white space
+without causing host swapping (in KB)
For QEMU/KVM with a memory balloon, setting the optional I<--period> to a value larger than 0 in seconds will allow the balloon driver to return
ACK

From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> QEMU reports timestamp along with other memory statistics, but this information is not reported by libvirt statistics API. It could be useful to determine if the data reported is fresh or not. Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- include/libvirt/libvirt-domain.h | 5 ++++- src/libvirt-domain.c | 2 ++ src/qemu/qemu_monitor_json.c | 22 ++++++++++++---------- tools/virsh-domain-monitor.c | 2 ++ tools/virsh.pod | 1 + 5 files changed, 21 insertions(+), 11 deletions(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 3fb482b..01a2649 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -609,11 +609,14 @@ typedef enum { */ VIR_DOMAIN_MEMORY_STAT_USABLE = 8, + /* Timestamp of the last update of statistics, in seconds. */ + VIR_DOMAIN_MEMORY_STAT_LAST_UPDATE = 9, + /* * The number of statistics supported by this version of the interface. * To add new statistics, add them to the enum and increase this value. */ - VIR_DOMAIN_MEMORY_STAT_NR = 9, + VIR_DOMAIN_MEMORY_STAT_NR = 10, # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_MEMORY_STAT_LAST = VIR_DOMAIN_MEMORY_STAT_NR diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 1467030..413f8af 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -5991,6 +5991,8 @@ virDomainGetInterfaceParameters(virDomainPtr domain, * to swap, corresponds to 'Available' in /proc/meminfo * VIR_DOMAIN_MEMORY_STAT_ACTUAL_BALLOON: * Current balloon value (in kb). + * VIR_DOMAIN_MEMORY_STAT_LAST_UPDATE + * Timestamp of the last statistic * * Returns: The number of stats provided or -1 in case of failure. */ diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index c83e0cc..6b955f8 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -1633,10 +1633,10 @@ qemuMonitorJSONGetBalloonInfo(qemuMonitorPtr mon, * rates and/or whether data has been collected since a previous cycle. * It's currently unused. */ -#define GET_BALLOON_STATS(FIELD, TAG, DIVISOR) \ - if (virJSONValueObjectHasKey(statsdata, FIELD) && \ +#define GET_BALLOON_STATS(OBJECT, FIELD, TAG, DIVISOR) \ + if (virJSONValueObjectHasKey(OBJECT, FIELD) && \ (got < nr_stats)) { \ - if (virJSONValueObjectGetNumberUlong(statsdata, FIELD, &mem) < 0) { \ + if (virJSONValueObjectGetNumberUlong(OBJECT, FIELD, &mem) < 0) { \ VIR_DEBUG("Failed to get '%s' value", FIELD); \ } else { \ /* Not being collected? No point in providing bad data */ \ @@ -1707,20 +1707,22 @@ int qemuMonitorJSONGetMemoryStats(qemuMonitorPtr mon, goto cleanup; } - GET_BALLOON_STATS("stat-swap-in", + GET_BALLOON_STATS(statsdata, "stat-swap-in", VIR_DOMAIN_MEMORY_STAT_SWAP_IN, 1024); - GET_BALLOON_STATS("stat-swap-out", + GET_BALLOON_STATS(statsdata, "stat-swap-out", VIR_DOMAIN_MEMORY_STAT_SWAP_OUT, 1024); - GET_BALLOON_STATS("stat-major-faults", + GET_BALLOON_STATS(statsdata, "stat-major-faults", VIR_DOMAIN_MEMORY_STAT_MAJOR_FAULT, 1); - GET_BALLOON_STATS("stat-minor-faults", + GET_BALLOON_STATS(statsdata, "stat-minor-faults", VIR_DOMAIN_MEMORY_STAT_MINOR_FAULT, 1); - GET_BALLOON_STATS("stat-free-memory", + GET_BALLOON_STATS(statsdata, "stat-free-memory", VIR_DOMAIN_MEMORY_STAT_UNUSED, 1024); - GET_BALLOON_STATS("stat-total-memory", + GET_BALLOON_STATS(statsdata, "stat-total-memory", VIR_DOMAIN_MEMORY_STAT_AVAILABLE, 1024); - GET_BALLOON_STATS("stat-available-memory", + GET_BALLOON_STATS(statsdata, "stat-available-memory", VIR_DOMAIN_MEMORY_STAT_USABLE, 1024); + GET_BALLOON_STATS(data, "last-update", + VIR_DOMAIN_MEMORY_STAT_LAST_UPDATE, 1); ret = got; diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index 7f30da2..77aa272 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -375,6 +375,8 @@ cmdDomMemStat(vshControl *ctl, const vshCmd *cmd) vshPrint(ctl, "actual %llu\n", stats[i].val); if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_RSS) vshPrint(ctl, "rss %llu\n", stats[i].val); + if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_LAST_UPDATE) + vshPrint(ctl, "last_update %llu\n", stats[i].val); } ret = true; diff --git a/tools/virsh.pod b/tools/virsh.pod index 5444f91..6af94d1 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -821,6 +821,7 @@ B<Explanation of fields>: rss - Resident Set Size of the running domain's process (in kB) usable - The amount of memory which can be reclaimed by balloon without causing host swapping (in KB) + last-update - Timestamp of the last update of statistics (in seconds) For QEMU/KVM with a memory balloon, setting the optional I<--period> to a value larger than 0 in seconds will allow the balloon driver to return -- 1.9.5.msysgit.0

On Wed, Jul 13, 2016 at 01:42:14PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com>
QEMU reports timestamp along with other memory statistics, but this information is not reported by libvirt statistics API. It could be useful to determine if the data reported is fresh or not.
Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> ---
ACK

From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> Is necessary to call it from other contexts, such as qemuDomainGetStatsBalloon. Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- src/qemu/qemu_driver.c | 55 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 35 insertions(+), 20 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index f8d9afe..6fa8d01 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -10959,32 +10959,22 @@ qemuDomainGetInterfaceParameters(virDomainPtr dom, return ret; } +/* This functions assumes that job QEMU_JOB_QUERY is started by a caller */ + static int -qemuDomainMemoryStats(virDomainPtr dom, - virDomainMemoryStatPtr stats, - unsigned int nr_stats, - unsigned int flags) +qemuDomainMemoryStatsInternal(virQEMUDriverPtr driver, + virDomainObjPtr vm, + virDomainMemoryStatPtr stats, + unsigned int nr_stats) + { - virQEMUDriverPtr driver = dom->conn->privateData; - virDomainObjPtr vm; int ret = -1; long rss; - virCheckFlags(0, -1); - - if (!(vm = qemuDomObjFromDomain(dom))) - goto cleanup; - - if (virDomainMemoryStatsEnsureACL(dom->conn, vm->def) < 0) - goto cleanup; - - if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_QUERY) < 0) - goto cleanup; - if (!virDomainObjIsActive(vm)) { virReportError(VIR_ERR_OPERATION_INVALID, "%s", _("domain is not running")); - goto endjob; + return -1; } if (vm->def->memballoon && @@ -10995,7 +10985,7 @@ qemuDomainMemoryStats(virDomainPtr dom, ret = -1; if (ret < 0 || ret >= nr_stats) - goto endjob; + return ret; } else { ret = 0; } @@ -11009,7 +10999,32 @@ qemuDomainMemoryStats(virDomainPtr dom, ret++; } - endjob: + return ret; +} + +static int +qemuDomainMemoryStats(virDomainPtr dom, + virDomainMemoryStatPtr stats, + unsigned int nr_stats, + unsigned int flags) +{ + virQEMUDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm; + int ret = -1; + + virCheckFlags(0, -1); + + if (!(vm = qemuDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainMemoryStatsEnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_QUERY) < 0) + goto cleanup; + + ret = qemuDomainMemoryStatsInternal(driver, vm, stats, nr_stats); + qemuDomainObjEndJob(driver, vm); cleanup: -- 1.9.5.msysgit.0

On Wed, Jul 13, 2016 at 01:42:15PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com>
Is necessary to call it from other contexts, such as qemuDomainGetStatsBalloon.
Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- src/qemu/qemu_driver.c | 55 ++++++++++++++++++++++++++++++++------------------ 1 file changed, 35 insertions(+), 20 deletions(-)
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index f8d9afe..6fa8d01 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -10959,32 +10959,22 @@ qemuDomainGetInterfaceParameters(virDomainPtr dom, return ret; }
+/* This functions assumes that job QEMU_JOB_QUERY is started by a caller */ +
This was already pointed out, we don't put a new line between function and a comment for that function.
static int -qemuDomainMemoryStats(virDomainPtr dom, - virDomainMemoryStatPtr stats, - unsigned int nr_stats, - unsigned int flags) +qemuDomainMemoryStatsInternal(virQEMUDriverPtr driver, + virDomainObjPtr vm, + virDomainMemoryStatPtr stats, + unsigned int nr_stats) + { - virQEMUDriverPtr driver = dom->conn->privateData; - virDomainObjPtr vm; int ret = -1; long rss;
ACK

From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> To collect all balloon statistics for all guests it was necessary to make several libvirt requests. Now it's possible to get all balloon statiscs via single connectGetAllDomainStats call. Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- src/qemu/qemu_driver.c | 39 +++++++++++++++++++++++++++++++++++++-- tools/virsh.pod | 12 +++++++++++- 2 files changed, 48 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 6fa8d01..70f3faa 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -18511,15 +18511,29 @@ qemuDomainGetStatsCpu(virQEMUDriverPtr driver ATTRIBUTE_UNUSED, return 0; } + +#define STORE_MEM_RECORD(TAG, NAME) { \ + if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_ ##TAG) \ + if (virTypedParamsAddULLong(&record->params, \ + &record->nparams, \ + maxparams, \ + "balloon." NAME, \ + stats[i].val) < 0) \ + return -1; \ +} + static int -qemuDomainGetStatsBalloon(virQEMUDriverPtr driver ATTRIBUTE_UNUSED, +qemuDomainGetStatsBalloon(virQEMUDriverPtr driver, virDomainObjPtr dom, virDomainStatsRecordPtr record, int *maxparams, - unsigned int privflags ATTRIBUTE_UNUSED) + unsigned int privflags) { qemuDomainObjPrivatePtr priv = dom->privateData; + virDomainMemoryStatStruct stats[VIR_DOMAIN_MEMORY_STAT_NR]; + int nr_stats; unsigned long long cur_balloon = 0; + size_t i; int err = 0; if (!virDomainDefHasMemballoon(dom->def)) { @@ -18544,8 +18558,29 @@ qemuDomainGetStatsBalloon(virQEMUDriverPtr driver ATTRIBUTE_UNUSED, virDomainDefGetMemoryTotal(dom->def)) < 0) return -1; + if (err || !HAVE_JOB(privflags)) + return 0; + + nr_stats = qemuDomainMemoryStatsInternal(driver, dom, stats, + VIR_DOMAIN_MEMORY_STAT_NR); + if (nr_stats < 0) + return 0; + + for (i = 0; i < nr_stats; i++) { + STORE_MEM_RECORD(SWAP_IN, "swap_in") + STORE_MEM_RECORD(SWAP_OUT, "swap_out") + STORE_MEM_RECORD(MAJOR_FAULT, "major_fault") + STORE_MEM_RECORD(MINOR_FAULT, "minor_fault") + STORE_MEM_RECORD(UNUSED, "unused") + STORE_MEM_RECORD(AVAILABLE, "available") + STORE_MEM_RECORD(RSS, "rss") + STORE_MEM_RECORD(LAST_UPDATE, "last-update") + STORE_MEM_RECORD(USABLE, "usable") + } + return 0; } +#undef STORE_MEM_RECORD static int diff --git a/tools/virsh.pod b/tools/virsh.pod index 6af94d1..cda1874 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -899,7 +899,17 @@ I<--cpu-total> returns: I<--balloon> returns: "balloon.current" - the memory in kiB currently used, -"balloon.maximum" - the maximum memory in kiB allowed +"balloon.maximum" - the maximum memory in kiB allowed, +"balloon.swap_in" - the amount of data read from swap space (in kB), +"balloon.swap_out" - the amount of memory written out to swap space (in kB), +"balloon.major_fault" - the number of page faults then disk IO was required, +"balloon.minor_fault" - the number of other page faults, +"balloon.unused" - the amount of memory left unused by the system (in kB), +"balloon.available" - the amount of usable memory as seen by the domain (in kB), +"balloon.rss" - Resident Set Size of running domain's process (in kB), +"balloon.usable" - the amount of memory which can be reclaimed by balloon +without causing host swapping (in KB), +"balloon.last-update" - timestamp of the last update of statistics (in seconds), I<--vcpu> returns: "vcpu.current" - current number of online virtual CPUs, -- 1.9.5.msysgit.0

On Wed, Jul 13, 2016 at 01:42:16PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com>
To collect all balloon statistics for all guests it was necessary to make several libvirt requests. Now it's possible to get all balloon statiscs via single connectGetAllDomainStats call.
Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- src/qemu/qemu_driver.c | 39 +++++++++++++++++++++++++++++++++++++-- tools/virsh.pod | 12 +++++++++++- 2 files changed, 48 insertions(+), 3 deletions(-)
diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 6fa8d01..70f3faa 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -18511,15 +18511,29 @@ qemuDomainGetStatsCpu(virQEMUDriverPtr driver ATTRIBUTE_UNUSED, return 0; }
+ +#define STORE_MEM_RECORD(TAG, NAME) { \ + if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_ ##TAG) \ + if (virTypedParamsAddULLong(&record->params, \ + &record->nparams, \ + maxparams, \ + "balloon." NAME, \ + stats[i].val) < 0) \ + return -1; \ +}
This isn't a correct way how to write a MACRO. For starters, the syntax isn't the same as for functions. If you need to ensure that it is a separate block the correct way is to do something like this: #define MACRO() \ do { \ #body of the macro \ } while (0) But in this case it's not required to use the do-while construct. Next point to this MACRO is that it uses other variables not passed as parameters. It's OK to do that but only inside a function and right before its usage and also you should #undef it right away. For example: #define MACRO() MACRO() MACRO() MACRO() #undef MACRO
+ static int -qemuDomainGetStatsBalloon(virQEMUDriverPtr driver ATTRIBUTE_UNUSED, +qemuDomainGetStatsBalloon(virQEMUDriverPtr driver, virDomainObjPtr dom, virDomainStatsRecordPtr record, int *maxparams, - unsigned int privflags ATTRIBUTE_UNUSED) + unsigned int privflags) { qemuDomainObjPrivatePtr priv = dom->privateData; + virDomainMemoryStatStruct stats[VIR_DOMAIN_MEMORY_STAT_NR]; + int nr_stats; unsigned long long cur_balloon = 0; + size_t i; int err = 0;
if (!virDomainDefHasMemballoon(dom->def)) { @@ -18544,8 +18558,29 @@ qemuDomainGetStatsBalloon(virQEMUDriverPtr driver ATTRIBUTE_UNUSED, virDomainDefGetMemoryTotal(dom->def)) < 0) return -1;
+ if (err || !HAVE_JOB(privflags)) + return 0;
There is no need to check the *err* variable, it's used only to indicate whether the "balloon.current" should be reported or not. I would change the condition to this: if (!HAVE_JOB(privflags) || !virDomainObjIsActive(dom)) return 0;
+ + nr_stats = qemuDomainMemoryStatsInternal(driver, dom, stats, + VIR_DOMAIN_MEMORY_STAT_NR); + if (nr_stats < 0) + return 0; + + for (i = 0; i < nr_stats; i++) { + STORE_MEM_RECORD(SWAP_IN, "swap_in") + STORE_MEM_RECORD(SWAP_OUT, "swap_out") + STORE_MEM_RECORD(MAJOR_FAULT, "major_fault") + STORE_MEM_RECORD(MINOR_FAULT, "minor_fault") + STORE_MEM_RECORD(UNUSED, "unused") + STORE_MEM_RECORD(AVAILABLE, "available") + STORE_MEM_RECORD(RSS, "rss") + STORE_MEM_RECORD(LAST_UPDATE, "last-update") + STORE_MEM_RECORD(USABLE, "usable") + } + return 0; } +#undef STORE_MEM_RECORD
static int diff --git a/tools/virsh.pod b/tools/virsh.pod index 6af94d1..cda1874 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -899,7 +899,17 @@ I<--cpu-total> returns:
I<--balloon> returns: "balloon.current" - the memory in kiB currently used, -"balloon.maximum" - the maximum memory in kiB allowed +"balloon.maximum" - the maximum memory in kiB allowed, +"balloon.swap_in" - the amount of data read from swap space (in kB), +"balloon.swap_out" - the amount of memory written out to swap space (in kB), +"balloon.major_fault" - the number of page faults then disk IO was required, +"balloon.minor_fault" - the number of other page faults, +"balloon.unused" - the amount of memory left unused by the system (in kB), +"balloon.available" - the amount of usable memory as seen by the domain (in kB), +"balloon.rss" - Resident Set Size of running domain's process (in kB), +"balloon.usable" - the amount of memory which can be reclaimed by balloon +without causing host swapping (in KB), +"balloon.last-update" - timestamp of the last update of statistics (in seconds),
I<--vcpu> returns: "vcpu.current" - current number of online virtual CPUs, -- 1.9.5.msysgit.0
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list

From: Igor Redko <redkoi@virtuozzo.com> Communication with qemu monitor is time-consuming and there is additional overhead for converting qemu_binary->json->libvirt_binary. This patch tries to avoid unnecessary qmp queries. Knowing period of balloon statistics update cycle and timestamps of last update provided by qemu, it is possible to make cache of last response and reuse it for next requests. Signed-off-by: Igor Redko <redkoi@virtuozzo.com> Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- src/qemu/qemu_domain.h | 6 +++ src/qemu/qemu_driver.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 105 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 114db98..90cddfd 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -213,6 +213,12 @@ struct _qemuDomainObjPrivate { qemuDomainUnpluggingDevice unplug; + unsigned long long cacheExpires; + unsigned int cacheNrStats; + virDomainMemoryStatStruct cacheStats[VIR_DOMAIN_MEMORY_STAT_NR]; + + virCond unplugFinished; /* signals that unpluggingDevice was unplugged */ + const char *unpluggingDevice; /* alias of the device that is being unplugged */ char **qemuDevices; /* NULL-terminated list of devices aliases known to QEMU */ bool hookRun; /* true if there was a hook run over this domain */ diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 70f3faa..616b87d 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -10962,6 +10962,96 @@ qemuDomainGetInterfaceParameters(virDomainPtr dom, /* This functions assumes that job QEMU_JOB_QUERY is started by a caller */ static int +qemuDomainCacheMemoryStats(virDomainObjPtr vm, + virDomainMemoryStatPtr stats, + unsigned int nr_stats, + unsigned int flags) +{ + unsigned long long now; + unsigned long long expires; + unsigned long long timestamp; + size_t i; + virDomainDefPtr def; + qemuDomainObjPrivatePtr priv; + int ret = -1; + + if (nr_stats <= 0 || nr_stats > VIR_DOMAIN_MEMORY_STAT_NR) + goto cleanup; + + if (virDomainObjGetDefs(vm, flags, &def, NULL) < 0) + goto cleanup; + + if (def->memballoon->period <= 0) + goto cleanup; + + if (virTimeMillisNow(&now) < 0) + goto cleanup; + + priv = vm->privateData; + timestamp = now; + + for (i = 0; i < nr_stats; i++){ + if (stats[i].tag == VIR_DOMAIN_MEMORY_STAT_LAST_UPDATE){ + /* QEMU timestamps are in secs */ + timestamp = stats[i].val * 1000; + break; + } + } + /* period also in secs */ + expires = (timestamp > now) ? now : timestamp + def->memballoon->period * 1000; + + if (expires < now || expires <= priv->cacheExpires) + goto cleanup; + + priv->cacheExpires = expires; + memcpy(priv->cacheStats, stats, nr_stats * sizeof(*stats)); + priv->cacheNrStats = nr_stats; + ret = nr_stats; + + cleanup: + return ret; +} + +static int +qemuDomainCachedMemoryStats(virDomainObjPtr vm, + virDomainMemoryStatPtr stats, + unsigned int nr_stats, + unsigned int flags) +{ + unsigned long long now; + virDomainDefPtr def; + qemuDomainObjPrivatePtr priv; + int ret = -1; + + if (virTimeMillisNow(&now) < 0) + goto cleanup; + + priv = vm->privateData; + + if (nr_stats <= 0 || nr_stats > priv->cacheNrStats) + goto cleanup; + + if (now > priv->cacheExpires){ + goto cleanup; + } + + if (virDomainObjGetDefs(vm, flags, &def, NULL) < 0) + goto cleanup; + + /* If period was changed or system time driffted we drop caches */ + if (now + def->memballoon->period * 1000 < priv->cacheExpires){ + priv->cacheNrStats = 0; + goto cleanup; + } + + memcpy(stats, priv->cacheStats, nr_stats * sizeof(*stats)); + return nr_stats; + + cleanup: + return ret; +} + +static int qemuDomainMemoryStatsInternal(virQEMUDriverPtr driver, virDomainObjPtr vm, virDomainMemoryStatPtr stats, @@ -10977,6 +11067,11 @@ qemuDomainMemoryStatsInternal(virQEMUDriverPtr driver, return -1; } + ret = qemuDomainCachedMemoryStats(vm, stats, nr_stats, 0); + + if (ret > 0) + goto cleanup; + if (vm->def->memballoon && vm->def->memballoon->model == VIR_DOMAIN_MEMBALLOON_MODEL_VIRTIO) { qemuDomainObjEnterMonitor(driver, vm); @@ -10985,7 +11080,7 @@ qemuDomainMemoryStatsInternal(virQEMUDriverPtr driver, ret = -1; if (ret < 0 || ret >= nr_stats) - return ret; + goto cache; } else { ret = 0; } @@ -10999,6 +11094,9 @@ qemuDomainMemoryStatsInternal(virQEMUDriverPtr driver, ret++; } + cache: + qemuDomainCacheMemoryStats(vm, stats, ret, 0); + cleanup: return ret; } -- 1.9.5.msysgit.0

On Wed, Jul 13, 2016 at 01:42:17PM +0300, Derbyshev Dmitriy wrote:
From: Igor Redko <redkoi@virtuozzo.com>
Communication with qemu monitor is time-consuming and there is additional overhead for converting qemu_binary->json->libvirt_binary.
This patch tries to avoid unnecessary qmp queries. Knowing period of balloon statistics update cycle and timestamps of last update provided by qemu, it is possible to make cache of last response and reuse it for next requests.
I don't think that this is necessary. It doesn't take that much time to get the memory stats and if someone want's to spam libvirt about those stats, well it's pointless. NACK from me to this patch and the last one, but let's see if someone else has a different opinion. Pavel

On Wed, Jul 27, 2016 at 01:09:38PM +0200, Pavel Hrdina wrote:
On Wed, Jul 13, 2016 at 01:42:17PM +0300, Derbyshev Dmitriy wrote:
From: Igor Redko <redkoi@virtuozzo.com>
Communication with qemu monitor is time-consuming and there is additional overhead for converting qemu_binary->json->libvirt_binary.
This patch tries to avoid unnecessary qmp queries. Knowing period of balloon statistics update cycle and timestamps of last update provided by qemu, it is possible to make cache of last response and reuse it for next requests.
I don't think that this is necessary. It doesn't take that much time to get the memory stats and if someone want's to spam libvirt about those stats, well it's pointless. NACK from me to this patch and the last one, but let's see if someone else has a different opinion.
Yeah, I don't really see a need for this. The stats refresh period is exposed in the XML, so an application using libvirt can simply avoid calling this libvirt API more frequently than the configured stats period. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

From: Igor Redko <redkoi@virtuozzo.com>
Communication with qemu monitor is time-consuming and there is additional overhead for converting qemu_binary->json->libvirt_binary.
This patch tries to avoid unnecessary qmp queries. Knowing period of balloon statistics update cycle and timestamps of last update provided by qemu, it is possible to make cache of last response and reuse it for next requests. I don't think that this is necessary. It doesn't take that much time to get the memory stats and if someone want's to spam libvirt about those stats, well it's
On Wed, Jul 13, 2016 at 01:42:17PM +0300, Derbyshev Dmitriy wrote: pointless. NACK from me to this patch and the last one, but let's see if someone else has a different opinion. Yeah, I don't really see a need for this. The stats refresh period is exposed in the XML, so an application using libvirt can simply avoid calling this libvirt API more frequently than the configured stats
On Wed, Jul 27, 2016 at 01:09:38PM +0200, Pavel Hrdina wrote: period.
Regards, Daniel Our use-case may be somewhat different from the one you have in mind: several almost independent applications sending requests for new stats every now and then. Even though each one of these applications may reduce the frequency of its own send request, overall load on the system is still high. We could get around that by making one application devoted to caching
7/27/2016 2:14 PM, Daniel P. Berrange пишет: these requests, but it is more natural to do this work in libvirt instead. Thanks for your reply, Dmitry

From: Igor Redko <redkoi@virtuozzo.com> Signed-off-by: Igor Redko <redkoi@virtuozzo.com> Signed-off-by: Derbyshev Dmitry <dderbyshev@virtuozzo.com> --- src/libvirt_qemu_probes.d | 4 ++++ src/qemu/qemu_driver.c | 7 +++++++ 2 files changed, 11 insertions(+) diff --git a/src/libvirt_qemu_probes.d b/src/libvirt_qemu_probes.d index e4449a9..1a36026 100644 --- a/src/libvirt_qemu_probes.d +++ b/src/libvirt_qemu_probes.d @@ -19,4 +19,8 @@ provider libvirt { probe qemu_monitor_io_read(void *mon, const char *buf, unsigned int len, int ret, int errno); probe qemu_monitor_io_write(void *mon, const char *buf, unsigned int len, int ret, int errno); probe qemu_monitor_io_send_fd(void *mon, int fd, int ret, int errno); + + # Memory statistics caching + probe qemu_memstats_cache_expires(unsigned long long old_expire, unsigned long long new_expire); + probe qemu_memstats_cache_expired(unsigned long long expire, unsigned long long now); }; diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 616b87d..d14009e 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -105,6 +105,11 @@ #include "virnuma.h" #include "dirname.h" #include "network/bridge_driver.h" +#include "virprobe.h" + +#ifdef WITH_DTRACE_PROBES +# include "libvirt_qemu_probes.h" +#endif #define VIR_FROM_THIS VIR_FROM_QEMU @@ -11000,6 +11005,7 @@ qemuDomainCacheMemoryStats(virDomainObjPtr vm, /* period also in secs */ expires = (timestamp > now) ? now : timestamp + def->memballoon->period * 1000; + PROBE(QEMU_MEMSTATS_CACHE_EXPIRES, "old=%llu new=%llu", priv->cacheExpires, expires); if (expires < now || expires <= priv->cacheExpires) goto cleanup; @@ -11032,6 +11038,7 @@ qemuDomainCachedMemoryStats(virDomainObjPtr vm, goto cleanup; if (now > priv->cacheExpires){ + PROBE(QEMU_MEMSTATS_CACHE_EXPIRED, "old=%llu new=%llu", priv->cacheExpires, now); goto cleanup; } -- 1.9.5.msysgit.0

On Wed, Jul 13, 2016 at 01:42:11PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com>
There are some things that needs to be updated, but if you're OK with me updating those issues I can push the first 5 patches for you with the issues fixed. Pavel

7/27/2016 2:10 PM, Pavel Hrdina пишет:
On Wed, Jul 13, 2016 at 01:42:11PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> There are some things that needs to be updated, but if you're OK with me updating those issues I can push the first 5 patches for you with the issues fixed.
Pavel I would be very glad if did so, thank you. Last 2 patches are auxiliary, while the first 5 are important to us with or without them.
If I would like to resend new version of these last 2 patches, should I send them in the same patch series or other, if first patches are already applied? Dmitry

On Wed, Jul 27, 2016 at 03:05:43PM +0300, Dmitry Derbyshev wrote:
7/27/2016 2:10 PM, Pavel Hrdina пишет:
On Wed, Jul 13, 2016 at 01:42:11PM +0300, Derbyshev Dmitriy wrote:
From: Derbyshev Dmitry <dderbyshev@virtuozzo.com> There are some things that needs to be updated, but if you're OK with me updating those issues I can push the first 5 patches for you with the issues fixed.
Pavel I would be very glad if did so, thank you. Last 2 patches are auxiliary, while the first 5 are important to us with or without them.
I've pushed 1,4,5 patches and for patches 2 and 3 only the diff to already pushed patches.
If I would like to resend new version of these last 2 patches, should I send them in the same patch series or other, if first patches are already applied?
I'm still not convinced that we need to do the caching. However if something is pushed from patch series you usually send a new patch series without the patches that are already pushed, so in this case only the last two patches. Pavel
participants (4)
-
Daniel P. Berrange
-
Derbyshev Dmitriy
-
Dmitry Derbyshev
-
Pavel Hrdina