[libvirt] [PATCH v2] qemu: bulk stats: implement (cpu) tune group.
Management applications, like oVirt, may need to setup cpu quota limits to enforce QoS for domains. For this purpose, management applications also need to check how domains are behaving with respect to CPU quota. This data is available using the virDomainGetSchedulerParameters API. This patch adds a new group to bulk stats API to obtain the same information. Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1191428 --- include/libvirt/libvirt-domain.h | 1 + src/libvirt-domain.c | 16 ++++++++ src/qemu/qemu_driver.c | 84 ++++++++++++++++++++++++++++++++++++++++ tools/virsh-domain-monitor.c | 7 ++++ tools/virsh.pod | 10 ++++- 5 files changed, 117 insertions(+), 1 deletion(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index a9d3efd..a283f93 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1723,6 +1723,7 @@ typedef enum { VIR_DOMAIN_STATS_VCPU = (1 << 3), /* return domain virtual CPU info */ VIR_DOMAIN_STATS_INTERFACE = (1 << 4), /* return domain interfaces info */ VIR_DOMAIN_STATS_BLOCK = (1 << 5), /* return domain block info */ + VIR_DOMAIN_STATS_TUNE_CPU = (1 << 6), /* return domain CPU tuning info */ } virDomainStatsTypes; typedef enum { diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 89d1eab..b451299 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -11004,6 +11004,22 @@ virConnectGetDomainCapabilities(virConnectPtr conn, * "block.<num>.physical" - physical size in bytes of the container of the * backing image as unsigned long long. * + * VIR_DOMAIN_STATS_TUNE_CPU: Return CPU tuning statistics + * and usage information. + * The typed parameter keys are in this format: + * "tune.vcpu.quota" - max allowed bandwidth, in microseconds, as + * long long integer. -1 means 'infinite'. + * "tune.vcpu.period" - timeframe on which the virtual cpu quota is + * enforced, in microseconds, as unsigned long long. + * "tune.emu.quota" - max allowed bandwidth for emulator threads, + * in microseconds, as long long integer. + * -1 means 'infinite'. + * "tune.emu.period" - timeframe on which the emulator quota is + * enforced, in microseconds, as unsigned long long. + * "tune.cpu.shares" - weight of this domain. This value is meaningful + * only if compared with the other values of + * the running domains. Expressed as unsigned long long. + * * Note that entire stats groups or individual stat fields may be missing from * the output in case they are not supported by the given hypervisor, are not * applicable for the current state of the guest domain, or their retrieval diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index ffa4e19..a810fa5 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -18768,6 +18768,89 @@ qemuDomainGetStatsBlock(virQEMUDriverPtr driver, #undef QEMU_ADD_COUNT_PARAM + +#define QEMU_ADD_PARAM_LL(record, maxparams, name, value) \ +do { \ + if (virTypedParamsAddLLong(&(record)->params, \ + &(record)->nparams, \ + maxparams, \ + name, \ + value) < 0) \ + goto cleanup; \ +} while (0) + +#define QEMU_ADD_PARAM_ULL(record, maxparams, name, value) \ +do { \ + if (virTypedParamsAddULLong(&(record)->params, \ + &(record)->nparams, \ + maxparams, \ + name, \ + value) < 0) \ + goto cleanup; \ +} while (0) + +static int +qemuDomainGetStatsCpuTune(virQEMUDriverPtr driver, + virDomainObjPtr dom, + virDomainStatsRecordPtr record, + int *maxparams, + unsigned int privflags ATTRIBUTE_UNUSED) +{ + int ret = -1; + unsigned long long shares = 0; + qemuDomainObjPrivatePtr priv = dom->privateData; + virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); + + if (!cfg->privileged || + !virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPU)) { + ret = 0; + goto cleanup; + } + + if (virCgroupGetCpuShares(priv->cgroup, &shares) < 0) { + ret = 0; + goto cleanup; + } + + QEMU_ADD_PARAM_ULL(record, maxparams, "tune.cpu.shares", shares); + + if (virCgroupSupportsCpuBW(priv->cgroup)) { + unsigned long long period = 0; + long long quota = 0; + unsigned long long emulator_period = 0; + long long emulator_quota = 0; + int err; + + err = qemuGetVcpusBWLive(dom, &period, "a); + if (!err) { + QEMU_ADD_PARAM_ULL(record, maxparams, + "tune.vcpu.period", period); + QEMU_ADD_PARAM_LL(record, maxparams, + "tune.vcpu.quota", quota); + } + + err = qemuGetEmulatorBandwidthLive(dom, priv->cgroup, + &emulator_period, &emulator_quota); + if (!err) { + QEMU_ADD_PARAM_ULL(record, maxparams, + "tune.emu.period", emulator_period); + QEMU_ADD_PARAM_LL(record, maxparams, + "tune.emu.quota", emulator_quota); + } + } + + ret = 0; + + cleanup: + virObjectUnref(cfg); + return ret; +} + +#undef QEMU_ADD_PARAM_LL + +#undef QEMU_ADD_PARAM_ULL + + typedef int (*qemuDomainGetStatsFunc)(virQEMUDriverPtr driver, virDomainObjPtr dom, @@ -18788,6 +18871,7 @@ static struct qemuDomainGetStatsWorker qemuDomainGetStatsWorkers[] = { { qemuDomainGetStatsVcpu, VIR_DOMAIN_STATS_VCPU, false }, { qemuDomainGetStatsInterface, VIR_DOMAIN_STATS_INTERFACE, false }, { qemuDomainGetStatsBlock, VIR_DOMAIN_STATS_BLOCK, true }, + { qemuDomainGetStatsCpuTune, VIR_DOMAIN_STATS_TUNE_CPU, false }, { NULL, 0, false } }; diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index 464ac11..ede433d 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -2017,6 +2017,10 @@ static const vshCmdOptDef opts_domstats[] = { .type = VSH_OT_BOOL, .help = N_("report domain block device statistics"), }, + {.name = "tune-cpu", + .type = VSH_OT_BOOL, + .help = N_("report domain cpu scheduler parameters"), + }, {.name = "list-active", .type = VSH_OT_BOOL, .help = N_("list only active domains"), @@ -2127,6 +2131,9 @@ cmdDomstats(vshControl *ctl, const vshCmd *cmd) if (vshCommandOptBool(cmd, "block")) stats |= VIR_DOMAIN_STATS_BLOCK; + if (vshCommandOptBool(cmd, "tune-cpu")) + stats |= VIR_DOMAIN_STATS_TUNE_CPU; + if (vshCommandOptBool(cmd, "list-active")) flags |= VIR_CONNECT_GET_ALL_DOMAINS_STATS_ACTIVE; diff --git a/tools/virsh.pod b/tools/virsh.pod index 1f71c91..bcc966e 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -828,6 +828,7 @@ or unique source names printed by this command. =item B<domstats> [I<--raw>] [I<--enforce>] [I<--backing>] [I<--state>] [I<--cpu-total>] [I<--balloon>] [I<--vcpu>] [I<--interface>] [I<--block>] +[I<--tune-cpu>] [[I<--list-active>] [I<--list-inactive>] [I<--list-persistent>] [I<--list-transient>] [I<--list-running>] [I<--list-paused>] [I<--list-shutoff>] [I<--list-other>]] | [I<domain> ...] @@ -846,7 +847,7 @@ behavior use the I<--raw> flag. The individual statistics groups are selectable via specific flags. By default all supported statistics groups are returned. Supported statistics groups flags are: I<--state>, I<--cpu-total>, I<--balloon>, -I<--vcpu>, I<--interface>, I<--block>. +I<--vcpu>, I<--interface>, I<--block>, I<--tune-cpu>. When selecting the I<--state> group the following fields are returned: "state.state" - state of the VM, returned as number from virDomainState enum, @@ -906,6 +907,13 @@ local file or block device, "block.<num>.capacity" - logical size of source file in bytes, "block.<num>.physical" - physical size of source file in bytes +I<--tune-cpu> returns: +"tune.vcpu.quota" - max allowed bandwidth, in microseconds, +"tune.vcpu.period" - timeframe for enforcing the vcpu quota, in microseconds, +"tune.emu.quota" - maximum bandwidth for emulator threads, in microseconds, +"tune.emu.period" - timeframe for enforcing the emu quota, in microseconds, +"tune.cpu.shares" - weight of this domain, compared to other running domains. + Selecting a specific statistics groups doesn't guarantee that the daemon supports the selected group of stats. Flag I<--enforce> forces the command to fail if the daemon doesn't support the -- 2.1.0
Version 2 addresses the comments received in v1. Please note that the the typed parameter keys are not yet changed. ----- Original Message -----
From: "Francesco Romani" <fromani@redhat.com> To: libvir-list@redhat.com Cc: "Francesco Romani" <fromani@redhat.com> Sent: Friday, March 6, 2015 3:22:50 PM Subject: [libvirt] [PATCH v2] qemu: bulk stats: implement (cpu) tune group.
Management applications, like oVirt, may need to setup cpu quota limits to enforce QoS for domains.
For this purpose, management applications also need to check how domains are behaving with respect to CPU quota. This data is available using the virDomainGetSchedulerParameters API.
This patch adds a new group to bulk stats API to obtain the same information.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1191428 --- include/libvirt/libvirt-domain.h | 1 + src/libvirt-domain.c | 16 ++++++++ src/qemu/qemu_driver.c | 84 ++++++++++++++++++++++++++++++++++++++++ tools/virsh-domain-monitor.c | 7 ++++ tools/virsh.pod | 10 ++++- 5 files changed, 117 insertions(+), 1 deletion(-)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index a9d3efd..a283f93 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1723,6 +1723,7 @@ typedef enum { VIR_DOMAIN_STATS_VCPU = (1 << 3), /* return domain virtual CPU info */ VIR_DOMAIN_STATS_INTERFACE = (1 << 4), /* return domain interfaces info */ VIR_DOMAIN_STATS_BLOCK = (1 << 5), /* return domain block info */ + VIR_DOMAIN_STATS_TUNE_CPU = (1 << 6), /* return domain CPU tuning info */ } virDomainStatsTypes;
typedef enum { diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 89d1eab..b451299 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -11004,6 +11004,22 @@ virConnectGetDomainCapabilities(virConnectPtr conn, * "block.<num>.physical" - physical size in bytes of the container of the * backing image as unsigned long long. * + * VIR_DOMAIN_STATS_TUNE_CPU: Return CPU tuning statistics + * and usage information. + * The typed parameter keys are in this format: + * "tune.vcpu.quota" - max allowed bandwidth, in microseconds, as + * long long integer. -1 means 'infinite'. + * "tune.vcpu.period" - timeframe on which the virtual cpu quota is + * enforced, in microseconds, as unsigned long long. + * "tune.emu.quota" - max allowed bandwidth for emulator threads, + * in microseconds, as long long integer. + * -1 means 'infinite'. + * "tune.emu.period" - timeframe on which the emulator quota is + * enforced, in microseconds, as unsigned long long. + * "tune.cpu.shares" - weight of this domain. This value is meaningful + * only if compared with the other values of + * the running domains. Expressed as unsigned long long. + * * Note that entire stats groups or individual stat fields may be missing from * the output in case they are not supported by the given hypervisor, are not * applicable for the current state of the guest domain, or their retrieval diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index ffa4e19..a810fa5 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -18768,6 +18768,89 @@ qemuDomainGetStatsBlock(virQEMUDriverPtr driver,
#undef QEMU_ADD_COUNT_PARAM
+ +#define QEMU_ADD_PARAM_LL(record, maxparams, name, value) \ +do { \ + if (virTypedParamsAddLLong(&(record)->params, \ + &(record)->nparams, \ + maxparams, \ + name, \ + value) < 0) \ + goto cleanup; \ +} while (0) + +#define QEMU_ADD_PARAM_ULL(record, maxparams, name, value) \ +do { \ + if (virTypedParamsAddULLong(&(record)->params, \ + &(record)->nparams, \ + maxparams, \ + name, \ + value) < 0) \ + goto cleanup; \ +} while (0) + +static int +qemuDomainGetStatsCpuTune(virQEMUDriverPtr driver, + virDomainObjPtr dom, + virDomainStatsRecordPtr record, + int *maxparams, + unsigned int privflags ATTRIBUTE_UNUSED) +{ + int ret = -1; + unsigned long long shares = 0; + qemuDomainObjPrivatePtr priv = dom->privateData; + virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); + + if (!cfg->privileged || + !virCgroupHasController(priv->cgroup, VIR_CGROUP_CONTROLLER_CPU)) { + ret = 0; + goto cleanup; + } + + if (virCgroupGetCpuShares(priv->cgroup, &shares) < 0) { + ret = 0; + goto cleanup; + } + + QEMU_ADD_PARAM_ULL(record, maxparams, "tune.cpu.shares", shares); + + if (virCgroupSupportsCpuBW(priv->cgroup)) { + unsigned long long period = 0; + long long quota = 0; + unsigned long long emulator_period = 0; + long long emulator_quota = 0; + int err; + + err = qemuGetVcpusBWLive(dom, &period, "a); + if (!err) { + QEMU_ADD_PARAM_ULL(record, maxparams, + "tune.vcpu.period", period); + QEMU_ADD_PARAM_LL(record, maxparams, + "tune.vcpu.quota", quota); + } + + err = qemuGetEmulatorBandwidthLive(dom, priv->cgroup, + &emulator_period, &emulator_quota); + if (!err) { + QEMU_ADD_PARAM_ULL(record, maxparams, + "tune.emu.period", emulator_period); + QEMU_ADD_PARAM_LL(record, maxparams, + "tune.emu.quota", emulator_quota); + } + } + + ret = 0; + + cleanup: + virObjectUnref(cfg); + return ret; +} + +#undef QEMU_ADD_PARAM_LL + +#undef QEMU_ADD_PARAM_ULL + + typedef int (*qemuDomainGetStatsFunc)(virQEMUDriverPtr driver, virDomainObjPtr dom, @@ -18788,6 +18871,7 @@ static struct qemuDomainGetStatsWorker qemuDomainGetStatsWorkers[] = { { qemuDomainGetStatsVcpu, VIR_DOMAIN_STATS_VCPU, false }, { qemuDomainGetStatsInterface, VIR_DOMAIN_STATS_INTERFACE, false }, { qemuDomainGetStatsBlock, VIR_DOMAIN_STATS_BLOCK, true }, + { qemuDomainGetStatsCpuTune, VIR_DOMAIN_STATS_TUNE_CPU, false }, { NULL, 0, false } };
diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index 464ac11..ede433d 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -2017,6 +2017,10 @@ static const vshCmdOptDef opts_domstats[] = { .type = VSH_OT_BOOL, .help = N_("report domain block device statistics"), }, + {.name = "tune-cpu", + .type = VSH_OT_BOOL, + .help = N_("report domain cpu scheduler parameters"), + }, {.name = "list-active", .type = VSH_OT_BOOL, .help = N_("list only active domains"), @@ -2127,6 +2131,9 @@ cmdDomstats(vshControl *ctl, const vshCmd *cmd) if (vshCommandOptBool(cmd, "block")) stats |= VIR_DOMAIN_STATS_BLOCK;
+ if (vshCommandOptBool(cmd, "tune-cpu")) + stats |= VIR_DOMAIN_STATS_TUNE_CPU; + if (vshCommandOptBool(cmd, "list-active")) flags |= VIR_CONNECT_GET_ALL_DOMAINS_STATS_ACTIVE;
diff --git a/tools/virsh.pod b/tools/virsh.pod index 1f71c91..bcc966e 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -828,6 +828,7 @@ or unique source names printed by this command.
=item B<domstats> [I<--raw>] [I<--enforce>] [I<--backing>] [I<--state>] [I<--cpu-total>] [I<--balloon>] [I<--vcpu>] [I<--interface>] [I<--block>] +[I<--tune-cpu>] [[I<--list-active>] [I<--list-inactive>] [I<--list-persistent>] [I<--list-transient>] [I<--list-running>] [I<--list-paused>] [I<--list-shutoff>] [I<--list-other>]] | [I<domain> ...] @@ -846,7 +847,7 @@ behavior use the I<--raw> flag. The individual statistics groups are selectable via specific flags. By default all supported statistics groups are returned. Supported statistics groups flags are: I<--state>, I<--cpu-total>, I<--balloon>, -I<--vcpu>, I<--interface>, I<--block>. +I<--vcpu>, I<--interface>, I<--block>, I<--tune-cpu>.
When selecting the I<--state> group the following fields are returned: "state.state" - state of the VM, returned as number from virDomainState enum, @@ -906,6 +907,13 @@ local file or block device, "block.<num>.capacity" - logical size of source file in bytes, "block.<num>.physical" - physical size of source file in bytes
+I<--tune-cpu> returns: +"tune.vcpu.quota" - max allowed bandwidth, in microseconds, +"tune.vcpu.period" - timeframe for enforcing the vcpu quota, in microseconds, +"tune.emu.quota" - maximum bandwidth for emulator threads, in microseconds, +"tune.emu.period" - timeframe for enforcing the emu quota, in microseconds, +"tune.cpu.shares" - weight of this domain, compared to other running domains. + Selecting a specific statistics groups doesn't guarantee that the daemon supports the selected group of stats. Flag I<--enforce> forces the command to fail if the daemon doesn't support the -- 2.1.0
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
-- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
On Fri, Mar 06, 2015 at 15:22:50 +0100, Francesco Romani wrote:
Management applications, like oVirt, may need to setup cpu quota limits to enforce QoS for domains.
For this purpose, management applications also need to check how domains are behaving with respect to CPU quota. This data is available using the virDomainGetSchedulerParameters API.
This patch adds a new group to bulk stats API to obtain the same information.
Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=1191428 --- include/libvirt/libvirt-domain.h | 1 + src/libvirt-domain.c | 16 ++++++++ src/qemu/qemu_driver.c | 84 ++++++++++++++++++++++++++++++++++++++++ tools/virsh-domain-monitor.c | 7 ++++ tools/virsh.pod | 10 ++++- 5 files changed, 117 insertions(+), 1 deletion(-)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index a9d3efd..a283f93 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1723,6 +1723,7 @@ typedef enum { VIR_DOMAIN_STATS_VCPU = (1 << 3), /* return domain virtual CPU info */ VIR_DOMAIN_STATS_INTERFACE = (1 << 4), /* return domain interfaces info */ VIR_DOMAIN_STATS_BLOCK = (1 << 5), /* return domain block info */ + VIR_DOMAIN_STATS_TUNE_CPU = (1 << 6), /* return domain CPU tuning info */ } virDomainStatsTypes;
typedef enum { diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 89d1eab..b451299 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -11004,6 +11004,22 @@ virConnectGetDomainCapabilities(virConnectPtr conn, * "block.<num>.physical" - physical size in bytes of the container of the * backing image as unsigned long long. * + * VIR_DOMAIN_STATS_TUNE_CPU: Return CPU tuning statistics + * and usage information. + * The typed parameter keys are in this format: + * "tune.vcpu.quota" - max allowed bandwidth, in microseconds, as + * long long integer. -1 means 'infinite'. + * "tune.vcpu.period" - timeframe on which the virtual cpu quota is + * enforced, in microseconds, as unsigned long long. + * "tune.emu.quota" - max allowed bandwidth for emulator threads, + * in microseconds, as long long integer. + * -1 means 'infinite'. + * "tune.emu.period" - timeframe on which the emulator quota is + * enforced, in microseconds, as unsigned long long. + * "tune.cpu.shares" - weight of this domain. This value is meaningful + * only if compared with the other values of + * the running domains. Expressed as unsigned long long. + *
These options above represent configuration and not any statistic value, so they won't change unless libvirt is instructed to change them. I don't think they belong to the stats API. Additionally libvirt recently added an event to track change of the tunables. See virConnectDomainEventTunableCallback. http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainEventTun... http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_TUNABLE_CPU_C... http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_TUNABLE_CPU_E... and so on ... I think you want to use that event and leave this api for statistics only. NACK Peter
----- Original Message -----
From: "Peter Krempa" <pkrempa@redhat.com> To: "Francesco Romani" <fromani@redhat.com> Cc: libvir-list@redhat.com Sent: Monday, March 9, 2015 2:16:34 PM Subject: Re: [libvirt] [PATCH v2] qemu: bulk stats: implement (cpu) tune group.
[...]
+ * VIR_DOMAIN_STATS_TUNE_CPU: Return CPU tuning statistics + * and usage information. + * The typed parameter keys are in this format: + * "tune.vcpu.quota" - max allowed bandwidth, in microseconds, as + * long long integer. -1 means 'infinite'. + * "tune.vcpu.period" - timeframe on which the virtual cpu quota is + * enforced, in microseconds, as unsigned long long. + * "tune.emu.quota" - max allowed bandwidth for emulator threads, + * in microseconds, as long long integer. + * -1 means 'infinite'. + * "tune.emu.period" - timeframe on which the emulator quota is + * enforced, in microseconds, as unsigned long long. + * "tune.cpu.shares" - weight of this domain. This value is meaningful + * only if compared with the other values of + * the running domains. Expressed as unsigned long long. + *
These options above represent configuration and not any statistic value, so they won't change unless libvirt is instructed to change them. I don't think they belong to the stats API.
Additionally libvirt recently added an event to track change of the tunables. See virConnectDomainEventTunableCallback.
http://libvirt.org/html/libvirt-libvirt-domain.html#virConnectDomainEventTun... http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_TUNABLE_CPU_C... http://libvirt.org/html/libvirt-libvirt-domain.html#VIR_DOMAIN_TUNABLE_CPU_E... and so on ...
I think you want to use that event and leave this api for statistics only.
Yep, it seems that should serve us better. Thanks! -- Francesco Romani RedHat Engineering Virtualization R & D Phone: 8261328 IRC: fromani
participants (2)
-
Francesco Romani -
Peter Krempa