[PATCH Libvirt v3 00/10] Support dirty page rate upper limit
Hi, This is the latest version for the series, please review. Thanks, Yong v3: - rebase on master and fix conflicts - refactor the comments and update the version tag v2: The patch set introduce a new API virDomainSetVcpuDirtyLimit to allow upper Apps to set upper limits of dirty page rate for virtual CPUs, the corresponding virsh API as follows: limit-dirty-page-rate <domain> <rate> [--vcpu <number>] \ [--config] [--live] [--current] We put the dirty limit persistent info with the "vcpus" element in domain XML and extend dirtylimit statistics for domGetStats: <domain> ... <vcpu current='2'>3</vcpu> <vcpus> <vcpu id='0' hotpluggable='no' dirty_limit='10' order='1'.../> <vcpu id='1' hotpluggable='yes' dirty_limit='10' order='2'.../> </vcpus> ... </domain> If --vcpu option is not passed in the virsh command, set all virtual CPUs; if rate is set to zero, cancel the upper limit. Examples: To set the dirty page rate upper limit 10 MB/s for all virtual CPUs in c81_node1, use: # virsh limit-dirty-page-rate c81_node1 --rate 10 --live Set dirty page rate limit 10(MB/s) for all virtual CPUs successfully # virsh dumpxml c81_node1 | grep dirty_limit <vcpu id='0' enabled='yes' hotpluggable='no' order='1' dirty_limit='10'/> <vcpu id='1' enabled='yes' hotpluggable='no' order='2' dirty_limit='10'/> <vcpu id='2' enabled='yes' hotpluggable='no' order='3' dirty_limit='10'/> <vcpu id='3' enabled='no' hotpluggable='yes' dirty_limit='10'/> <vcpu id='4' enabled='no' hotpluggable='yes' dirty_limit='10'/> ...... Query the dirty limit info dynamically: # virsh domstats c81_node1 --dirtylimit Domain: 'c81_node1' dirtylimit.vcpu.0.limit=10 dirtylimit.vcpu.0.current=0 dirtylimit.vcpu.1.limit=10 dirtylimit.vcpu.1.current=0 dirtylimit.vcpu.2.limit=10 dirtylimit.vcpu.2.current=0 dirtylimit.vcpu.3.limit=10 dirtylimit.vcpu.3.current=0 dirtylimit.vcpu.4.limit=10 dirtylimit.vcpu.4.current=0 ...... To cancel the upper limit, use: # virsh limit-dirty-page-rate c81_node1 --rate 0 --live Cancel dirty page rate limit for all virtual CPUs successfully # virsh dumpxml c81_node1 | grep dirty_limit # virsh domstats c81_node1 --dirtylimit Domain: 'c81_node1' The dirty limit uses the QEMU dirty-limit feature introduced since 7.1.0, this feature allows CPU to be throttled as needed to keep their dirty page rate within the limit. It could, in some scenes, be used to provide quality-of-service in the aspect of the memory workload for virtual CPUs and QEMU itself use the feature to implement the dirty-limit throttle algorithm and apply it on the live migration, which improve responsiveness of large guests during live migration and can result in more stable read performance. The other application scenarios remain unexplored, before that, Libvirt could provide the basic API. Hyman Huang(黄勇) (10): qemu_capabilities: Introduce QEMU_CAPS_VCPU_DIRTY_LIMIT capability conf: Introduce XML for dirty limit configuration libvirt: Add virDomainSetVcpuDirtyLimit API qemu_driver: Implement qemuDomainSetVcpuDirtyLimit domain_validate: Export virDomainDefHasDirtyLimitStartupVcpus symbol qemu_process: Setup dirty limit after launching VM virsh: Introduce limit-dirty-page-rate api qemu_monitor: Implement qemuMonitorQueryVcpuDirtyLimit qemu_driver: Extend dirtlimit statistics for domGetStats virsh: Introduce command 'virsh domstats --dirtylimit' docs/formatdomain.rst | 7 +- docs/manpages/virsh.rst | 38 +++- include/libvirt/libvirt-domain.h | 5 + src/conf/domain_conf.c | 26 +++ src/conf/domain_conf.h | 8 + src/conf/domain_validate.c | 33 ++++ src/conf/domain_validate.h | 2 + src/conf/schemas/domaincommon.rng | 5 + src/driver-hypervisor.h | 7 + src/libvirt-domain.c | 68 +++++++ src/libvirt_private.syms | 1 + src/libvirt_public.syms | 1 + src/qemu/qemu_capabilities.c | 2 + src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_driver.c | 181 ++++++++++++++++++ src/qemu/qemu_monitor.c | 25 +++ src/qemu/qemu_monitor.h | 22 +++ src/qemu/qemu_monitor_json.c | 107 +++++++++++ src/qemu/qemu_monitor_json.h | 9 + src/qemu/qemu_process.c | 44 +++++ src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 18 +- src/remote_protocol-structs | 7 + .../qemucapabilitiesdata/caps_7.1.0_ppc64.xml | 1 + .../caps_7.1.0_x86_64.xml | 1 + tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml | 1 + .../caps_7.2.0_x86_64+hvf.xml | 1 + .../caps_7.2.0_x86_64.xml | 1 + .../caps_8.0.0_riscv64.xml | 1 + .../caps_8.0.0_x86_64.xml | 1 + .../qemucapabilitiesdata/caps_8.1.0_s390x.xml | 1 + .../caps_8.1.0_x86_64.xml | 1 + tools/virsh-domain-monitor.c | 7 + tools/virsh-domain.c | 109 +++++++++++ 34 files changed, 739 insertions(+), 4 deletions(-) -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> Introduce virDomainSetVcpuDirtyLimit API to set or cancel the dirty page rate upper limit. The API will throttle the virtual CPU as needed to keep their dirty page rate within the limit. Since it just throttles the virtual CPU, which dirties memory, read processes in the guest OS aren't penalized. The feature therefor could, in some scenes, be used to provide quality-of-service in the aspect of the memory workload for virtual CPUs. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- include/libvirt/libvirt-domain.h | 4 +++ src/driver-hypervisor.h | 7 ++++ src/libvirt-domain.c | 59 ++++++++++++++++++++++++++++++++ src/libvirt_public.syms | 1 + src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 18 +++++++++- src/remote_protocol-structs | 7 ++++ 7 files changed, 96 insertions(+), 1 deletion(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index a1902546bb..3d3c7cdcba 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -6506,4 +6506,8 @@ int virDomainFDAssociate(virDomainPtr domain, int *fds, unsigned int flags); +int virDomainSetVcpuDirtyLimit(virDomainPtr domain, + int vcpu, + unsigned long long rate, + unsigned int flags); #endif /* LIBVIRT_DOMAIN_H */ diff --git a/src/driver-hypervisor.h b/src/driver-hypervisor.h index 5219344b72..e61b9efca5 100644 --- a/src/driver-hypervisor.h +++ b/src/driver-hypervisor.h @@ -1448,6 +1448,12 @@ typedef int int *fds, unsigned int flags); +typedef int +(*virDrvDomainSetVcpuDirtyLimit)(virDomainPtr domain, + int vcpu, + unsigned long long rate, + unsigned int flags); + typedef struct _virHypervisorDriver virHypervisorDriver; /** @@ -1720,4 +1726,5 @@ struct _virHypervisorDriver { virDrvDomainGetMessages domainGetMessages; virDrvDomainStartDirtyRateCalc domainStartDirtyRateCalc; virDrvDomainFDAssociate domainFDAssociate; + virDrvDomainSetVcpuDirtyLimit domainSetVcpuDirtyLimit; }; diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 6616294fc1..5b505cc519 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -14041,3 +14041,62 @@ virDomainFDAssociate(virDomainPtr domain, virDispatchError(conn); return -1; } + +/** + * virDomainSetVcpuDirtyLimit: + * @domain: pointer to domain object + * @vcpu: index of the limited virtual CPU + * @rate: upper limit of dirty page rate (mebibyte/s) for virtual CPUs + * @flags: bitwise-OR of virDomainModificationImpact + * + * Dynamically set the dirty page rate upper limit for the virtual CPUs. + * + * @vcpu may be a positive value, zero, or equal to -1. If -1 is set, + * the change affects all virtual CPUs of VM; it affects the specified + * virtual CPU otherwise. + * @rate may be 0 to cancel the limit or a positive value to enable. The + * hypervisors are free to round it down to the nearest mebibyte/s. + * + * The API will throttle the virtual CPU as needed to keep their dirty + * page rate within the limit set by @rate. Since it just throttles the + * virtual CPU, which dirties memory, read processes in the guest OS + * aren't penalized. This could, in some scenes, be used to provide + * quality-of-service in the aspect of the memory workload for virtual + * CPUs. + * + * Returns 0 in case of success, -1 in case of failure. + * + * Since: 9.7.0 + */ +int +virDomainSetVcpuDirtyLimit(virDomainPtr domain, + int vcpu, + unsigned long long rate, + unsigned int flags) +{ + virConnectPtr conn; + + VIR_DOMAIN_DEBUG(domain, "vcpu=%d, rate=%llu, flags=0x%x", + vcpu, rate, flags); + + virResetLastError(); + + virCheckDomainReturn(domain, -1); + conn = domain->conn; + + virCheckReadOnlyGoto(conn->flags, error); + + if (conn->driver->domainSetVcpuDirtyLimit) { + int ret; + ret = conn->driver->domainSetVcpuDirtyLimit(domain, vcpu, rate, flags); + if (ret < 0) + goto error; + return ret; + } + + virReportUnsupportedError(); + + error: + virDispatchError(domain->conn); + return -1; +} diff --git a/src/libvirt_public.syms b/src/libvirt_public.syms index bd1e916d2a..602494935d 100644 --- a/src/libvirt_public.syms +++ b/src/libvirt_public.syms @@ -934,6 +934,7 @@ LIBVIRT_9.0.0 { LIBVIRT_9.7.0 { global: + virDomainSetVcpuDirtyLimit; virNetworkGetMetadata; virNetworkSetMetadata; } LIBVIRT_9.0.0; diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c index 0b925f8edc..15d023154b 100644 --- a/src/remote/remote_driver.c +++ b/src/remote/remote_driver.c @@ -8110,6 +8110,7 @@ static virHypervisorDriver hypervisor_driver = { .domainStartDirtyRateCalc = remoteDomainStartDirtyRateCalc, /* 7.2.0 */ .domainSetLaunchSecurityState = remoteDomainSetLaunchSecurityState, /* 8.0.0 */ .domainFDAssociate = remoteDomainFDAssociate, /* 9.0.0 */ + .domainSetVcpuDirtyLimit = remoteDomainSetVcpuDirtyLimit, /* 9.7.0 */ }; static virNetworkDriver network_driver = { diff --git a/src/remote/remote_protocol.x b/src/remote/remote_protocol.x index 7ff059e393..72b2684912 100644 --- a/src/remote/remote_protocol.x +++ b/src/remote/remote_protocol.x @@ -3955,6 +3955,14 @@ struct remote_domain_fd_associate_args { remote_nonnull_string name; unsigned int flags; }; + +struct remote_domain_set_vcpu_dirty_limit_args { + remote_nonnull_domain dom; + int vcpu; + unsigned hyper rate; + unsigned int flags; +}; + /*----- Protocol. -----*/ /* Define the program number, protocol version and procedure numbers here. */ @@ -7008,5 +7016,13 @@ enum remote_procedure { * @generate: both * @acl: network:read */ - REMOTE_PROC_NETWORK_GET_METADATA = 445 + REMOTE_PROC_NETWORK_GET_METADATA = 445, + + /** + * @generate: both + * @acl: domain:write + * @acl: domain:save:!VIR_DOMAIN_AFFECT_CONFIG|VIR_DOMAIN_AFFECT_LIVE + * @acl: domain:save:VIR_DOMAIN_AFFECT_CONFIG + */ + REMOTE_PROC_DOMAIN_SET_VCPU_DIRTY_LIMIT = 446 }; diff --git a/src/remote_protocol-structs b/src/remote_protocol-structs index c07e0af1e6..715a121f36 100644 --- a/src/remote_protocol-structs +++ b/src/remote_protocol-structs @@ -3290,6 +3290,12 @@ struct remote_domain_fd_associate_args { remote_nonnull_string name; u_int flags; }; +struct remote_domain_set_vcpu_dirty_limit_args { + remote_nonnull_domain dom; + int vcpu; + uint64_t rate; + u_int flags; +}; enum remote_procedure { REMOTE_PROC_CONNECT_OPEN = 1, REMOTE_PROC_CONNECT_CLOSE = 2, @@ -3736,4 +3742,5 @@ enum remote_procedure { REMOTE_PROC_DOMAIN_FD_ASSOCIATE = 443, REMOTE_PROC_NETWORK_SET_METADATA = 444, REMOTE_PROC_NETWORK_GET_METADATA = 445, + REMOTE_PROC_DOMAIN_SET_VCPU_DIRTY_LIMIT = 446, }; -- 2.38.5
On Tue, Aug 02, 2022 at 10:13:40PM +0800, ~hyman wrote:
From: Hyman Huang(黄勇) <yong.huang@smartx.com>
Introduce virDomainSetVcpuDirtyLimit API to set or cancel the dirty page rate upper limit.
The API will throttle the virtual CPU as needed to keep their dirty page rate within the limit. Since it just throttles the virtual CPU, which dirties memory, read processes in the guest OS aren't penalized.
The feature therefor could, in some scenes, be used to provide quality-of-service in the aspect of the memory workload for virtual CPUs.
Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- include/libvirt/libvirt-domain.h | 4 +++ src/driver-hypervisor.h | 7 ++++ src/libvirt-domain.c | 59 ++++++++++++++++++++++++++++++++ src/libvirt_public.syms | 1 + src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 18 +++++++++- src/remote_protocol-structs | 7 ++++ 7 files changed, 96 insertions(+), 1 deletion(-)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index a1902546bb..3d3c7cdcba 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -6506,4 +6506,8 @@ int virDomainFDAssociate(virDomainPtr domain, int *fds, unsigned int flags);
+int virDomainSetVcpuDirtyLimit(virDomainPtr domain, + int vcpu, + unsigned long long rate, + unsigned int flags);
We've generally tried to avoid adding single purpose APIs for tunable parameters, instead using APIs with virTypedParameter arrays to allow bulk updates. I note that we don't appear to have any mechanism currently to set the VCPU scheduler tunables either Perhaps we should have a more general virDomainSetVCPUTuneParameters(virDomainPtr domain, int vcpu, virTypedParameterPtr params, unsigned int params, unsigned int flags);
#endif /* LIBVIRT_DOMAIN_H */ diff --git a/src/driver-hypervisor.h b/src/driver-hypervisor.h index 5219344b72..e61b9efca5 100644 --- a/src/driver-hypervisor.h +++ b/src/driver-hypervisor.h @@ -1448,6 +1448,12 @@ typedef int int *fds, unsigned int flags);
+typedef int +(*virDrvDomainSetVcpuDirtyLimit)(virDomainPtr domain, + int vcpu, + unsigned long long rate, + unsigned int flags); + typedef struct _virHypervisorDriver virHypervisorDriver;
/** @@ -1720,4 +1726,5 @@ struct _virHypervisorDriver { virDrvDomainGetMessages domainGetMessages; virDrvDomainStartDirtyRateCalc domainStartDirtyRateCalc; virDrvDomainFDAssociate domainFDAssociate; + virDrvDomainSetVcpuDirtyLimit domainSetVcpuDirtyLimit; }; diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 6616294fc1..5b505cc519 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -14041,3 +14041,62 @@ virDomainFDAssociate(virDomainPtr domain, virDispatchError(conn); return -1; } + +/** + * virDomainSetVcpuDirtyLimit: + * @domain: pointer to domain object + * @vcpu: index of the limited virtual CPU + * @rate: upper limit of dirty page rate (mebibyte/s) for virtual CPUs + * @flags: bitwise-OR of virDomainModificationImpact + * + * Dynamically set the dirty page rate upper limit for the virtual CPUs. + * + * @vcpu may be a positive value, zero, or equal to -1. If -1 is set, + * the change affects all virtual CPUs of VM; it affects the specified + * virtual CPU otherwise. + * @rate may be 0 to cancel the limit or a positive value to enable. The + * hypervisors are free to round it down to the nearest mebibyte/s. + * + * The API will throttle the virtual CPU as needed to keep their dirty + * page rate within the limit set by @rate. Since it just throttles the + * virtual CPU, which dirties memory, read processes in the guest OS + * aren't penalized. This could, in some scenes, be used to provide + * quality-of-service in the aspect of the memory workload for virtual + * CPUs. + * + * Returns 0 in case of success, -1 in case of failure. + * + * Since: 9.7.0 + */ +int +virDomainSetVcpuDirtyLimit(virDomainPtr domain, + int vcpu, + unsigned long long rate, + unsigned int flags) +{ + virConnectPtr conn; + + VIR_DOMAIN_DEBUG(domain, "vcpu=%d, rate=%llu, flags=0x%x", + vcpu, rate, flags); + + virResetLastError(); + + virCheckDomainReturn(domain, -1); + conn = domain->conn; + + virCheckReadOnlyGoto(conn->flags, error); + + if (conn->driver->domainSetVcpuDirtyLimit) { + int ret; + ret = conn->driver->domainSetVcpuDirtyLimit(domain, vcpu, rate, flags); + if (ret < 0) + goto error; + return ret; + } + + virReportUnsupportedError(); + + error: + virDispatchError(domain->conn); + return -1; +} diff --git a/src/libvirt_public.syms b/src/libvirt_public.syms index bd1e916d2a..602494935d 100644 --- a/src/libvirt_public.syms +++ b/src/libvirt_public.syms @@ -934,6 +934,7 @@ LIBVIRT_9.0.0 {
LIBVIRT_9.7.0 { global: + virDomainSetVcpuDirtyLimit; virNetworkGetMetadata; virNetworkSetMetadata; } LIBVIRT_9.0.0; diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c index 0b925f8edc..15d023154b 100644 --- a/src/remote/remote_driver.c +++ b/src/remote/remote_driver.c @@ -8110,6 +8110,7 @@ static virHypervisorDriver hypervisor_driver = { .domainStartDirtyRateCalc = remoteDomainStartDirtyRateCalc, /* 7.2.0 */ .domainSetLaunchSecurityState = remoteDomainSetLaunchSecurityState, /* 8.0.0 */ .domainFDAssociate = remoteDomainFDAssociate, /* 9.0.0 */ + .domainSetVcpuDirtyLimit = remoteDomainSetVcpuDirtyLimit, /* 9.7.0 */ };
static virNetworkDriver network_driver = { diff --git a/src/remote/remote_protocol.x b/src/remote/remote_protocol.x index 7ff059e393..72b2684912 100644 --- a/src/remote/remote_protocol.x +++ b/src/remote/remote_protocol.x @@ -3955,6 +3955,14 @@ struct remote_domain_fd_associate_args { remote_nonnull_string name; unsigned int flags; }; + +struct remote_domain_set_vcpu_dirty_limit_args { + remote_nonnull_domain dom; + int vcpu; + unsigned hyper rate; + unsigned int flags; +}; + /*----- Protocol. -----*/
/* Define the program number, protocol version and procedure numbers here. */ @@ -7008,5 +7016,13 @@ enum remote_procedure { * @generate: both * @acl: network:read */ - REMOTE_PROC_NETWORK_GET_METADATA = 445 + REMOTE_PROC_NETWORK_GET_METADATA = 445, + + /** + * @generate: both + * @acl: domain:write + * @acl: domain:save:!VIR_DOMAIN_AFFECT_CONFIG|VIR_DOMAIN_AFFECT_LIVE + * @acl: domain:save:VIR_DOMAIN_AFFECT_CONFIG + */ + REMOTE_PROC_DOMAIN_SET_VCPU_DIRTY_LIMIT = 446 }; diff --git a/src/remote_protocol-structs b/src/remote_protocol-structs index c07e0af1e6..715a121f36 100644 --- a/src/remote_protocol-structs +++ b/src/remote_protocol-structs @@ -3290,6 +3290,12 @@ struct remote_domain_fd_associate_args { remote_nonnull_string name; u_int flags; }; +struct remote_domain_set_vcpu_dirty_limit_args { + remote_nonnull_domain dom; + int vcpu; + uint64_t rate; + u_int flags; +}; enum remote_procedure { REMOTE_PROC_CONNECT_OPEN = 1, REMOTE_PROC_CONNECT_CLOSE = 2, @@ -3736,4 +3742,5 @@ enum remote_procedure { REMOTE_PROC_DOMAIN_FD_ASSOCIATE = 443, REMOTE_PROC_NETWORK_SET_METADATA = 444, REMOTE_PROC_NETWORK_GET_METADATA = 445, + REMOTE_PROC_DOMAIN_SET_VCPU_DIRTY_LIMIT = 446, }; -- 2.38.5
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
From: Hyman Huang(黄勇) <yong.huang@smartx.com> set-vcpu-dirty-limit/cancel-vcpu-dirty-limit/query-vcpu-dirty-limit were introduced since qemu >=7.1.0. Introduce corresponding capability. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- src/qemu/qemu_capabilities.c | 2 ++ src/qemu/qemu_capabilities.h | 1 + tests/qemucapabilitiesdata/caps_7.1.0_ppc64.xml | 1 + tests/qemucapabilitiesdata/caps_7.1.0_x86_64.xml | 1 + tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml | 1 + tests/qemucapabilitiesdata/caps_7.2.0_x86_64+hvf.xml | 1 + tests/qemucapabilitiesdata/caps_7.2.0_x86_64.xml | 1 + tests/qemucapabilitiesdata/caps_8.0.0_riscv64.xml | 1 + tests/qemucapabilitiesdata/caps_8.0.0_x86_64.xml | 1 + tests/qemucapabilitiesdata/caps_8.1.0_s390x.xml | 1 + tests/qemucapabilitiesdata/caps_8.1.0_x86_64.xml | 1 + 11 files changed, 12 insertions(+) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index ea4e248349..cf3f7efc7a 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -697,6 +697,7 @@ VIR_ENUM_IMPL(virQEMUCaps, /* 450 */ "run-with.async-teardown", /* QEMU_CAPS_RUN_WITH_ASYNC_TEARDOWN */ + "set-vcpu-dirty-limit", /* QEMU_CAPS_VCPU_DIRTY_LIMIT */ ); @@ -1220,6 +1221,7 @@ struct virQEMUCapsStringFlags virQEMUCapsCommands[] = { { "calc-dirty-rate", QEMU_CAPS_CALC_DIRTY_RATE }, { "query-stats", QEMU_CAPS_QUERY_STATS }, { "query-stats-schemas", QEMU_CAPS_QUERY_STATS_SCHEMAS }, + { "set-vcpu-dirty-limit", QEMU_CAPS_VCPU_DIRTY_LIMIT }, }; struct virQEMUCapsStringFlags virQEMUCapsMigration[] = { diff --git a/src/qemu/qemu_capabilities.h b/src/qemu/qemu_capabilities.h index ba289ba6b9..623acc9fe7 100644 --- a/src/qemu/qemu_capabilities.h +++ b/src/qemu/qemu_capabilities.h @@ -676,6 +676,7 @@ typedef enum { /* virQEMUCapsFlags grouping marker for syntax-check */ /* 450 */ QEMU_CAPS_RUN_WITH_ASYNC_TEARDOWN, /* asynchronous teardown -run-with async-teardown=on|off */ + QEMU_CAPS_VCPU_DIRTY_LIMIT, /* 'set-vcpu-dirty-limit' QMP command present */ QEMU_CAPS_LAST /* this must always be the last item */ } virQEMUCapsFlags; diff --git a/tests/qemucapabilitiesdata/caps_7.1.0_ppc64.xml b/tests/qemucapabilitiesdata/caps_7.1.0_ppc64.xml index 310b8f9d75..3e386357b1 100644 --- a/tests/qemucapabilitiesdata/caps_7.1.0_ppc64.xml +++ b/tests/qemucapabilitiesdata/caps_7.1.0_ppc64.xml @@ -157,6 +157,7 @@ <flag name='virtio-crypto'/> <flag name='pvpanic-pci'/> <flag name='virtio-gpu.blob'/> + <flag name='set-vcpu-dirty-limit'/> <version>7001000</version> <microcodeVersion>42900244</microcodeVersion> <package>v7.1.0</package> diff --git a/tests/qemucapabilitiesdata/caps_7.1.0_x86_64.xml b/tests/qemucapabilitiesdata/caps_7.1.0_x86_64.xml index 07e8c57f27..dc7cdd2877 100644 --- a/tests/qemucapabilitiesdata/caps_7.1.0_x86_64.xml +++ b/tests/qemucapabilitiesdata/caps_7.1.0_x86_64.xml @@ -194,6 +194,7 @@ <flag name='virtio-crypto'/> <flag name='pvpanic-pci'/> <flag name='virtio-gpu.blob'/> + <flag name='set-vcpu-dirty-limit'/> <version>7001000</version> <microcodeVersion>43100244</microcodeVersion> <package>v7.1.0</package> diff --git a/tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml b/tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml index b8d4d4888c..4484a56e0a 100644 --- a/tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml +++ b/tests/qemucapabilitiesdata/caps_7.2.0_ppc.xml @@ -152,6 +152,7 @@ <flag name='virtio-crypto'/> <flag name='pvpanic-pci'/> <flag name='virtio-gpu.blob'/> + <flag name='set-vcpu-dirty-limit'/> <version>7002000</version> <microcodeVersion>0</microcodeVersion> <package>qemu-7.2.0-6.fc37</package> diff --git a/tests/qemucapabilitiesdata/caps_7.2.0_x86_64+hvf.xml b/tests/qemucapabilitiesdata/caps_7.2.0_x86_64+hvf.xml index 928270d287..f3fe6076ef 100644 --- a/tests/qemucapabilitiesdata/caps_7.2.0_x86_64+hvf.xml +++ b/tests/qemucapabilitiesdata/caps_7.2.0_x86_64+hvf.xml @@ -198,6 +198,7 @@ <flag name='cryptodev-backend-lkcf'/> <flag name='pvpanic-pci'/> <flag name='virtio-gpu.blob'/> + <flag name='set-vcpu-dirty-limit'/> <version>7002000</version> <microcodeVersion>43100245</microcodeVersion> <package>v7.2.0</package> diff --git a/tests/qemucapabilitiesdata/caps_7.2.0_x86_64.xml b/tests/qemucapabilitiesdata/caps_7.2.0_x86_64.xml index a889c1b034..7dc2a0ddd7 100644 --- a/tests/qemucapabilitiesdata/caps_7.2.0_x86_64.xml +++ b/tests/qemucapabilitiesdata/caps_7.2.0_x86_64.xml @@ -198,6 +198,7 @@ <flag name='cryptodev-backend-lkcf'/> <flag name='pvpanic-pci'/> <flag name='virtio-gpu.blob'/> + <flag name='set-vcpu-dirty-limit'/> <version>7002000</version> <microcodeVersion>43100245</microcodeVersion> <package>v7.2.0</package> diff --git a/tests/qemucapabilitiesdata/caps_8.0.0_riscv64.xml b/tests/qemucapabilitiesdata/caps_8.0.0_riscv64.xml index 4aeb210ca7..49075a9194 100644 --- a/tests/qemucapabilitiesdata/caps_8.0.0_riscv64.xml +++ b/tests/qemucapabilitiesdata/caps_8.0.0_riscv64.xml @@ -139,6 +139,7 @@ <flag name='virtio-crypto'/> <flag name='pvpanic-pci'/> <flag name='virtio-gpu.blob'/> + <flag name='set-vcpu-dirty-limit'/> <version>7002050</version> <microcodeVersion>0</microcodeVersion> <package>v7.2.0-333-g222059a0fc</package> diff --git a/tests/qemucapabilitiesdata/caps_8.0.0_x86_64.xml b/tests/qemucapabilitiesdata/caps_8.0.0_x86_64.xml index 6d105e78a9..6cae74ac72 100644 --- a/tests/qemucapabilitiesdata/caps_8.0.0_x86_64.xml +++ b/tests/qemucapabilitiesdata/caps_8.0.0_x86_64.xml @@ -202,6 +202,7 @@ <flag name='virtio-gpu.blob'/> <flag name='rbd-encryption-layering'/> <flag name='rbd-encryption-luks-any'/> + <flag name='set-vcpu-dirty-limit'/> <version>8000000</version> <microcodeVersion>43100244</microcodeVersion> <package>v8.0.0</package> diff --git a/tests/qemucapabilitiesdata/caps_8.1.0_s390x.xml b/tests/qemucapabilitiesdata/caps_8.1.0_s390x.xml index a7519021e2..b8774781eb 100644 --- a/tests/qemucapabilitiesdata/caps_8.1.0_s390x.xml +++ b/tests/qemucapabilitiesdata/caps_8.1.0_s390x.xml @@ -113,6 +113,7 @@ <flag name='rbd-encryption-layering'/> <flag name='rbd-encryption-luks-any'/> <flag name='run-with.async-teardown'/> + <flag name='set-vcpu-dirty-limit'/> <version>8000050</version> <microcodeVersion>39100245</microcodeVersion> <package>v8.0.0-1270-g1c12355b</package> diff --git a/tests/qemucapabilitiesdata/caps_8.1.0_x86_64.xml b/tests/qemucapabilitiesdata/caps_8.1.0_x86_64.xml index adcf594138..9cd5d2cea7 100644 --- a/tests/qemucapabilitiesdata/caps_8.1.0_x86_64.xml +++ b/tests/qemucapabilitiesdata/caps_8.1.0_x86_64.xml @@ -203,6 +203,7 @@ <flag name='rbd-encryption-luks-any'/> <flag name='qcow2-discard-no-unref'/> <flag name='run-with.async-teardown'/> + <flag name='set-vcpu-dirty-limit'/> <version>8001000</version> <microcodeVersion>43100245</microcodeVersion> <package>v8.1.0</package> -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> The upper limit (megabyte/s) of the dirty page rate configured by the user can be tracked by the XML. To allow this, add the following XML: <domain> ... <vcpu current='2'>3</vcpu> <vcpus> <vcpu id='0' hotpluggable='no' dirty_limit='10' order='1'.../> <vcpu id='1' hotpluggable='yes' dirty_limit='10' order='2'.../> </vcpus> ... The "dirty_limit" attribute in "vcpu" sub-element within "vcpus" element allows to set an upper limit for the individual vCPU. The value can be set dynamically by limit-dirty-page-rate API. Note that the dirty limit feature is based on the dirty-ring feature, so it requires dirty-ring size configuration in XML. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- docs/formatdomain.rst | 7 ++++++- src/conf/domain_conf.c | 26 ++++++++++++++++++++++++ src/conf/domain_conf.h | 8 ++++++++ src/conf/domain_validate.c | 33 +++++++++++++++++++++++++++++++ src/conf/schemas/domaincommon.rng | 5 +++++ 5 files changed, 78 insertions(+), 1 deletion(-) diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index bc469e5f9f..337b7ec9cc 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst @@ -649,7 +649,7 @@ CPU Allocation ... <vcpu placement='static' cpuset="1-4,^3,6" current="1">2</vcpu> <vcpus> - <vcpu id='0' enabled='yes' hotpluggable='no' order='1'/> + <vcpu id='0' enabled='yes' hotpluggable='no' order='1' dirty_limit='64'/> <vcpu id='1' enabled='no' hotpluggable='yes'/> </vcpus> ... @@ -715,6 +715,11 @@ CPU Allocation be enabled and non-hotpluggable. On PPC64 along with it vCPUs that are in the same core need to be enabled as well. All non-hotpluggable CPUs present at boot need to be grouped after vCPU 0. :since:`Since 2.2.0 (QEMU only)` + ``dirty_limit`` :since:`Since 9.7.0 (QEMU and KVM only)` + The optional attribute ``dirty_limit`` allows to set an upper limit (MB/s) + of the dirty page rate for the vCPU. User can change the upper limit value + dynamically by using ``limit-dirty-page-rate`` API. Require ``dirty-ring`` + size configured. IOThreads Allocation diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index 2c8727de54..aff0dba84a 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -17088,6 +17088,7 @@ virDomainVcpuParse(virDomainDef *def, virDomainXMLOption *xmlopt) { int n; + int rv; xmlNodePtr vcpuNode; size_t i; unsigned int maxvcpus; @@ -17175,6 +17176,13 @@ virDomainVcpuParse(virDomainDef *def, if (virXMLPropUInt(nodes[i], "order", 10, VIR_XML_PROP_NONE, &vcpu->order) < 0) return -1; + + if ((rv = virXMLPropULongLong(nodes[i], "dirty_limit", 10, VIR_XML_PROP_NONNEGATIVE, + &vcpu->dirty_limit)) < 0) { + return -1; + } else if (rv > 0) { + vcpu->dirtyLimitSet = true; + } } } else { if (virDomainDefSetVcpus(def, vcpus) < 0) @@ -21219,6 +21227,20 @@ virDomainDefVcpuCheckAbiStability(virDomainDef *src, i); return false; } + + if (svcpu->dirtyLimitSet != dvcpu->dirtyLimitSet) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("Dirty limit state of vCPU '%1$zu' differs between source and destination definitions"), + i); + return false; + } + + if (svcpu->dirty_limit != dvcpu->dirty_limit) { + virReportError(VIR_ERR_CONFIG_UNSUPPORTED, + _("Dirty limit of vCPU '%1$zu' differs between source and destination definitions"), + i); + return false; + } } return true; @@ -26822,6 +26844,10 @@ virDomainCpuDefFormat(virBuffer *buf, if (vcpu->order != 0) virBufferAsprintf(buf, " order='%d'", vcpu->order); + if (vcpu->dirtyLimitSet) { + virBufferAsprintf(buf, " dirty_limit='%llu'", vcpu->dirty_limit); + } + virBufferAddLit(buf, "/>\n"); } virBufferAdjustIndent(buf, -2); diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index ca195a52d2..0bca85bec6 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -2812,6 +2812,14 @@ struct _virDomainVcpuDef { virDomainThreadSchedParam sched; virObject *privateData; + + /* set to true if the dirty page rate upper limit for + * the virtual CPU is configured + * */ + bool dirtyLimitSet; + + /* dirty page rate upper limit */ + unsigned long long dirty_limit; }; struct _virDomainBlkiotune { diff --git a/src/conf/domain_validate.c b/src/conf/domain_validate.c index b050f21edb..2df9db3467 100644 --- a/src/conf/domain_validate.c +++ b/src/conf/domain_validate.c @@ -1785,6 +1785,36 @@ virDomainDefValidateIOThreads(const virDomainDef *def) return 0; } +static int +virDomainDefHasDirtyLimitStartupVcpus(const virDomainDef *def) +{ + size_t maxvcpus = virDomainDefGetVcpusMax(def); + virDomainVcpuDef *vcpu; + size_t i; + + for (i = 0; i < maxvcpus; i++) { + vcpu = def->vcpus[i]; + + if (vcpu->dirtyLimitSet && (vcpu->dirty_limit != 0)) + return true; + } + + return false; +} + +static int +virDomainDefDirtyLimitValidate(const virDomainDef *def) +{ + if (virDomainDefHasDirtyLimitStartupVcpus(def)) { + if (def->kvm_features->dirty_ring_size == 0) { + virReportError(VIR_ERR_XML_ERROR, "%s", + _("Dirty limit requires dirty-ring size configuration")); + return -1; + } + } + + return 0; +} static int virDomainDefValidateInternal(const virDomainDef *def, @@ -1841,6 +1871,9 @@ virDomainDefValidateInternal(const virDomainDef *def, if (virDomainDefValidateIOThreads(def) < 0) return -1; + if (virDomainDefDirtyLimitValidate(def) < 0) + return -1; + return 0; } diff --git a/src/conf/schemas/domaincommon.rng b/src/conf/schemas/domaincommon.rng index 2f9ba31c0a..2e6a744eb4 100644 --- a/src/conf/schemas/domaincommon.rng +++ b/src/conf/schemas/domaincommon.rng @@ -844,6 +844,11 @@ <ref name="unsignedInt"/> </attribute> </optional> + <optional> + <attribute name="dirty_limit"> + <ref name="unsignedLong"/> + </attribute> + </optional> </element> </zeroOrMore> </element> -- 2.38.5
On Tue, Aug 01, 2023 at 05:31:12PM +0800, ~hyman wrote:
From: Hyman Huang(黄勇) <yong.huang@smartx.com>
The upper limit (megabyte/s) of the dirty page rate configured by the user can be tracked by the XML. To allow this, add the following XML:
<domain> ... <vcpu current='2'>3</vcpu> <vcpus> <vcpu id='0' hotpluggable='no' dirty_limit='10' order='1'.../> <vcpu id='1' hotpluggable='yes' dirty_limit='10' order='2'.../> </vcpus> ...
The "dirty_limit" attribute in "vcpu" sub-element within "vcpus" element allows to set an upper limit for the individual vCPU. The value can be set dynamically by limit-dirty-page-rate API.
Note that the dirty limit feature is based on the dirty-ring feature, so it requires dirty-ring size configuration in XML.
Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- docs/formatdomain.rst | 7 ++++++- src/conf/domain_conf.c | 26 ++++++++++++++++++++++++ src/conf/domain_conf.h | 8 ++++++++ src/conf/domain_validate.c | 33 +++++++++++++++++++++++++++++++ src/conf/schemas/domaincommon.rng | 5 +++++ 5 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index bc469e5f9f..337b7ec9cc 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst
@@ -715,6 +715,11 @@ CPU Allocation be enabled and non-hotpluggable. On PPC64 along with it vCPUs that are in the same core need to be enabled as well. All non-hotpluggable CPUs present at boot need to be grouped after vCPU 0. :since:`Since 2.2.0 (QEMU only)` + ``dirty_limit`` :since:`Since 9.7.0 (QEMU and KVM only)` + The optional attribute ``dirty_limit`` allows to set an upper limit (MB/s) + of the dirty page rate for the vCPU. User can change the upper limit value + dynamically by using ``limit-dirty-page-rate`` API. Require ``dirty-ring`` + size configured.
What scenarios would you want to apply such a limit ? Is there to admins for sensible values to use when setting this limit ? What is the impact on the guest if it hits the limit ? Is the snigle vCPU blocked for the remainder of some timeslice, or are all vCPUs blocked ? Does it even make sense to control this with different values per-VCPU as opposed to a single value for the VM as a whole ? With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Thanks Daniel for the comments ! On Wed, Sep 6, 2023 at 4:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 01, 2023 at 05:31:12PM +0800, ~hyman wrote:
From: Hyman Huang(黄勇) <yong.huang@smartx.com>
The upper limit (megabyte/s) of the dirty page rate configured by the user can be tracked by the XML. To allow this, add the following XML:
<domain> ... <vcpu current='2'>3</vcpu> <vcpus> <vcpu id='0' hotpluggable='no' dirty_limit='10' order='1'.../> <vcpu id='1' hotpluggable='yes' dirty_limit='10' order='2'.../> </vcpus> ...
The "dirty_limit" attribute in "vcpu" sub-element within "vcpus" element allows to set an upper limit for the individual vCPU. The value can be set dynamically by limit-dirty-page-rate API.
Note that the dirty limit feature is based on the dirty-ring feature, so it requires dirty-ring size configuration in XML.
Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- docs/formatdomain.rst | 7 ++++++- src/conf/domain_conf.c | 26 ++++++++++++++++++++++++ src/conf/domain_conf.h | 8 ++++++++ src/conf/domain_validate.c | 33 +++++++++++++++++++++++++++++++ src/conf/schemas/domaincommon.rng | 5 +++++ 5 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index bc469e5f9f..337b7ec9cc 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst
@@ -715,6 +715,11 @@ CPU Allocation be enabled and non-hotpluggable. On PPC64 along with it vCPUs that are in the same core need to be enabled as well. All non-hotpluggable CPUs present at boot need to be grouped after vCPU 0. :since:`Since 2.2.0 (QEMU only)` + ``dirty_limit`` :since:`Since 9.7.0 (QEMU and KVM only)` + The optional attribute ``dirty_limit`` allows to set an upper limit (MB/s) + of the dirty page rate for the vCPU. User can change the upper limit value + dynamically by using ``limit-dirty-page-rate`` API. Require ``dirty-ring`` + size configured.
What scenarios would you want to apply such a limit ?
To be completely honest, I haven't given the scenarios anythought. This method has been utilized up till now by migration to throttle the guest within QEMU. The most likely situation, in my opinion, is that some virtual machines on a host are using up all of the memory bandwidth, either intentionally or accidentally, while other virtual machines perform worse when writing to memory.
Is there to admins for sensible values to use when setting this limit ?
Currently, the "calc-dirty-rate" API can be used by upper-level apps to peek at the dirty page rate of the vCPU. If more details regarding setting values were required, we would add them to the comments.
What is the impact on the guest if it hits the limit ?
Guest memory writes would take longer than usual to complete.
Is the snigle vCPU blocked for the remainder of some timeslice, or are all vCPUs blocked ?
Single vCPU.
Does it even make sense to control this with different values per-VCPU as opposed to a single value for the VM as a whole ?
Since this issue needs to be discussed, I'm inclined to offer a per-vCPU API and let the top APP make the final decision.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Thanks,
Yong -- Best regards
On Wed, Sep 06, 2023 at 06:54:29PM +0800, Yong Huang wrote:
Thanks Daniel for the comments !
On Wed, Sep 6, 2023 at 4:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 01, 2023 at 05:31:12PM +0800, ~hyman wrote:
From: Hyman Huang(黄勇) <yong.huang@smartx.com>
The upper limit (megabyte/s) of the dirty page rate configured by the user can be tracked by the XML. To allow this, add the following XML:
<domain> ... <vcpu current='2'>3</vcpu> <vcpus> <vcpu id='0' hotpluggable='no' dirty_limit='10' order='1'.../> <vcpu id='1' hotpluggable='yes' dirty_limit='10' order='2'.../> </vcpus> ...
The "dirty_limit" attribute in "vcpu" sub-element within "vcpus" element allows to set an upper limit for the individual vCPU. The value can be set dynamically by limit-dirty-page-rate API.
Note that the dirty limit feature is based on the dirty-ring feature, so it requires dirty-ring size configuration in XML.
Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- docs/formatdomain.rst | 7 ++++++- src/conf/domain_conf.c | 26 ++++++++++++++++++++++++ src/conf/domain_conf.h | 8 ++++++++ src/conf/domain_validate.c | 33 +++++++++++++++++++++++++++++++ src/conf/schemas/domaincommon.rng | 5 +++++ 5 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index bc469e5f9f..337b7ec9cc 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst
@@ -715,6 +715,11 @@ CPU Allocation be enabled and non-hotpluggable. On PPC64 along with it vCPUs that are in the same core need to be enabled as well. All non-hotpluggable CPUs present at boot need to be grouped after vCPU 0. :since:`Since 2.2.0 (QEMU only)` + ``dirty_limit`` :since:`Since 9.7.0 (QEMU and KVM only)` + The optional attribute ``dirty_limit`` allows to set an upper limit (MB/s) + of the dirty page rate for the vCPU. User can change the upper limit value + dynamically by using ``limit-dirty-page-rate`` API. Require ``dirty-ring`` + size configured.
What scenarios would you want to apply such a limit ?
To be completely honest, I haven't given the scenarios anythought. This method has been utilized up till now by migration to throttle the guest within QEMU.
Lets say the guest has 2 vcpus and current dirty rates are vcpu 0: 3 mb/s vcpu 1: 150 mb/s if we put a limit on vcpu 1 of 50 mb/s supposedly that would reduce rates sufficient to let us migrate. But at any point in time the guest OS scheduler could move the workload to vcpu 0 instead. So we would actually want the dirty limi to apply to both vCPUs, even though at this current snapshot in time, vcpu 1 has the bigger dirty rate.
The most likely situation, in my opinion, is that some virtual machines on a host are using up all of the memory bandwidth, either intentionally or accidentally, while other virtual machines perform worse when writing to memory.
If we're protecting against either an accidental or malicious guest workload, then again we need to apply the same limit across all vCPUs, otherwise the guest can just shift its workload to the least limited vCPU. IOW, in both these scenarios I don't see a reason to allow the rate limit to be set differently on each vCPU. A single value set across all vCPUs is needed to offer the protection desired, otherwise the guest trivially escapes the limit by moving workloads across vCPUs
Is there to admins for sensible values to use when setting this limit ?
Currently, the "calc-dirty-rate" API can be used by upper-level apps to peek at the dirty page rate of the vCPU. If more details regarding setting values were required, we would add them to the comments.
What is the impact on the guest if it hits the limit ?
Guest memory writes would take longer than usual to complete.
Is the snigle vCPU blocked for the remainder of some timeslice, or are all vCPUs blocked ?
Single vCPU.
Does it even make sense to control this with different values per-VCPU as opposed to a single value for the VM as a whole ?
Since this issue needs to be discussed, I'm inclined to offer a per-vCPU API and let the top APP make the final decision.
On the XML representation more generally, allow of our tuning related to CPUs is under the <cputune> element somewhere. We also usually allow units to be set explicitly. eg we might want <cputune> <vcpudirtyrate limit='23' units='mb'/> </cputune> If we follow our normal practice that would also imply allowing a 'vcpus' bitmask eg <cputune> <vcpudirtyrate vcpus='0-3^2' limit='23' units='mb'/> </cputune> which would also imply allowing multiple <cputune> <vcpudirtyrate vcpus='0-3^2' limit='23' units='mb'/> <vcpudirtyrate vcpus='2' limit='10' units='mb'/> </cputune> With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, Sep 6, 2023 at 7:24 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
Thanks Daniel for the comments !
On Wed, Sep 6, 2023 at 4:48 PM Daniel P. Berrangé <berrange@redhat.com> wrote:
On Tue, Aug 01, 2023 at 05:31:12PM +0800, ~hyman wrote:
From: Hyman Huang(黄勇) <yong.huang@smartx.com>
The upper limit (megabyte/s) of the dirty page rate configured by the user can be tracked by the XML. To allow this, add the following XML:
<domain> ... <vcpu current='2'>3</vcpu> <vcpus> <vcpu id='0' hotpluggable='no' dirty_limit='10' order='1'.../> <vcpu id='1' hotpluggable='yes' dirty_limit='10' order='2'.../> </vcpus> ...
The "dirty_limit" attribute in "vcpu" sub-element within "vcpus" element allows to set an upper limit for the individual vCPU. The value can be set dynamically by limit-dirty-page-rate API.
Note that the dirty limit feature is based on the dirty-ring feature, so it requires dirty-ring size configuration in XML.
Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- docs/formatdomain.rst | 7 ++++++- src/conf/domain_conf.c | 26 ++++++++++++++++++++++++ src/conf/domain_conf.h | 8 ++++++++ src/conf/domain_validate.c | 33 +++++++++++++++++++++++++++++++ src/conf/schemas/domaincommon.rng | 5 +++++ 5 files changed, 78 insertions(+), 1 deletion(-)
diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst index bc469e5f9f..337b7ec9cc 100644 --- a/docs/formatdomain.rst +++ b/docs/formatdomain.rst
@@ -715,6 +715,11 @@ CPU Allocation be enabled and non-hotpluggable. On PPC64 along with it vCPUs
are in the
same core need to be enabled as well. All non-hotpluggable CPUs
present at
boot need to be grouped after vCPU 0. :since:`Since 2.2.0 (QEMU
only)`
+ ``dirty_limit`` :since:`Since 9.7.0 (QEMU and KVM only)` + The optional attribute ``dirty_limit`` allows to set an upper
(MB/s)
+ of the dirty page rate for the vCPU. User can change the upper
On Wed, Sep 06, 2023 at 06:54:29PM +0800, Yong Huang wrote: that limit limit
value
+ dynamically by using ``limit-dirty-page-rate`` API. Require ``dirty-ring`` + size configured.
What scenarios would you want to apply such a limit ?
To be completely honest, I haven't given the scenarios anythought. This method has been utilized up till now by migration to throttle the guest within QEMU.
Lets say the guest has 2 vcpus and current dirty rates are
vcpu 0: 3 mb/s vcpu 1: 150 mb/s
if we put a limit on vcpu 1 of 50 mb/s supposedly that would reduce rates sufficient to let us migrate. But at any point in time the guest OS scheduler could move the workload to vcpu 0 instead. So we would actually want the dirty limi to apply to both vCPUs, even though at this current snapshot in time, vcpu 1 has the bigger dirty rate.
The most likely situation, in my opinion, is that some virtual machines on a host are using up all of the memory bandwidth, either intentionally or accidentally, while other virtual machines perform worse when writing to memory.
If we're protecting against either an accidental or malicious guest workload, then again we need to apply the same limit across all vCPUs, otherwise the guest can just shift its workload to the least limited vCPU.
IOW, in both these scenarios I don't see a reason to allow the rate limit to be set differently on each vCPU. A single value set across all vCPUs is needed to offer the protection desired, otherwise the guest trivially escapes the limit by moving workloads across vCPUs
Ok, that works for me.
Is there to admins for sensible values to use when setting this limit ?
Currently, the "calc-dirty-rate" API can be used by upper-level apps to peek at the dirty page rate of the vCPU. If more details regarding setting values were required, we would add them to the comments.
What is the impact on the guest if it hits the limit ?
Guest memory writes would take longer than usual to complete.
Is the snigle vCPU blocked for the remainder of some timeslice, or are all vCPUs blocked ?
Single vCPU.
Does it even make sense to control this with different values per-VCPU as opposed to a single value for the VM as a whole ?
Since this issue needs to be discussed, I'm inclined to offer a per-vCPU API and let the top APP make the final decision.
On the XML representation more generally, allow of our tuning related to CPUs is under the <cputune> element somewhere. We also usually allow units to be set explicitly. eg we might want
<cputune> <vcpudirtyrate limit='23' units='mb'/> </cputune>
If we follow our normal practice that would also imply allowing a 'vcpus' bitmask eg
<cputune> <vcpudirtyrate vcpus='0-3^2' limit='23' units='mb'/> </cputune>
which would also imply allowing multiple
<cputune> <vcpudirtyrate vcpus='0-3^2' limit='23' units='mb'/> <vcpudirtyrate vcpus='2' limit='10' units='mb'/> </cputune>
Thanks to the comment, this api is more graceful and I'll try it in the next version.
With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
Thanks, Yong -- Best regards
From: Hyman Huang(黄勇) <yong.huang@smartx.com> Implement qemuMonitorQueryVcpuDirtyLimit which query vcpu dirty limit info by calling qmp 'query-vcpu-dirty-limit'. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- src/qemu/qemu_monitor.c | 12 +++++++ src/qemu/qemu_monitor.h | 17 ++++++++++ src/qemu/qemu_monitor_json.c | 64 ++++++++++++++++++++++++++++++++++++ src/qemu/qemu_monitor_json.h | 4 +++ 4 files changed, 97 insertions(+) diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 90bc0e62c9..9b18b4e46b 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -4513,3 +4513,15 @@ qemuMonitorSetVcpuDirtyLimit(qemuMonitor *mon, return qemuMonitorJSONSetVcpuDirtyLimit(mon, vcpu, rate); } + + +int +qemuMonitorQueryVcpuDirtyLimit(qemuMonitor *mon, + qemuMonitorVcpuDirtyLimitInfo *info) +{ + VIR_DEBUG("info=%p", info); + + QEMU_CHECK_MONITOR(mon); + + return qemuMonitorJSONQueryVcpuDirtyLimit(mon, info); +} diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 07a05365cf..1828bb202a 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -1584,3 +1584,20 @@ int qemuMonitorSetVcpuDirtyLimit(qemuMonitor *mon, int vcpu, unsigned long long rate); + +typedef struct _qemuMonitorVcpuDirtyLimit qemuMonitorVcpuDirtyLimit; +struct _qemuMonitorVcpuDirtyLimit { + int idx; /* virtual cpu index */ + unsigned long long limit; /* virtual cpu dirty page rate limit in MB/s */ + unsigned long long current; /* virtual cpu dirty page rate in MB/s */ +}; + +typedef struct _qemuMonitorVcpuDirtyLimitInfo qemuMonitorVcpuDirtyLimitInfo; +struct _qemuMonitorVcpuDirtyLimitInfo { + size_t nvcpus; /* number of virtual cpu */ + qemuMonitorVcpuDirtyLimit *limits; /* array of dirty page rate limit */ +}; + +int +qemuMonitorQueryVcpuDirtyLimit(qemuMonitor *mon, + qemuMonitorVcpuDirtyLimitInfo *info); diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index c8f6069566..8a72abf28d 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -8895,3 +8895,67 @@ qemuMonitorJSONSetVcpuDirtyLimit(qemuMonitor *mon, return 0; } + +static int +qemuMonitorJSONExtractVcpuDirtyLimitInfo(virJSONValue *data, + qemuMonitorVcpuDirtyLimitInfo *info) +{ + size_t nvcpus; + size_t i; + + nvcpus = virJSONValueArraySize(data); + info->nvcpus = nvcpus; + info->limits = g_new0(qemuMonitorVcpuDirtyLimit, nvcpus); + + for (i = 0; i < nvcpus; i++) { + virJSONValue *entry = virJSONValueArrayGet(data, i); + if (virJSONValueObjectGetNumberInt(entry, "cpu-index", + &info->limits[i].idx) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("query-vcpu-dirty-limit reply was missing 'cpu-index' data")); + return -1; + } + + if (virJSONValueObjectGetNumberUlong(entry, "limit-rate", + &info->limits[i].limit) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("query-vcpu-dirty-limit reply was missing 'limit-rate' data")); + return -1; + } + + if (virJSONValueObjectGetNumberUlong(entry, "current-rate", + &info->limits[i].current) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("query-vcpu-dirty-limit reply was missing 'current-rate' data")); + return -1; + } + } + + return 0; +} + +int +qemuMonitorJSONQueryVcpuDirtyLimit(qemuMonitor *mon, + qemuMonitorVcpuDirtyLimitInfo *info) +{ + g_autoptr(virJSONValue) cmd = NULL; + g_autoptr(virJSONValue) reply = NULL; + virJSONValue *data = NULL; + + if (!(cmd = qemuMonitorJSONMakeCommand("query-vcpu-dirty-limit", NULL))) + return -1; + + if (qemuMonitorJSONCommand(mon, cmd, &reply) < 0) + return -1; + + if (qemuMonitorJSONCheckError(cmd, reply) < 0) + return -1; + + if (!(data = virJSONValueObjectGetArray(reply, "return"))) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("query-vcpu-dirty-limit reply was missing 'return' data")); + return -1; + } + + return qemuMonitorJSONExtractVcpuDirtyLimitInfo(data, info); +} diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h index 89f61b3052..bd8131508b 100644 --- a/src/qemu/qemu_monitor_json.h +++ b/src/qemu/qemu_monitor_json.h @@ -830,3 +830,7 @@ int qemuMonitorJSONSetVcpuDirtyLimit(qemuMonitor *mon, int vcpu, unsigned long long rate); + +int +qemuMonitorJSONQueryVcpuDirtyLimit(qemuMonitor *mon, + qemuMonitorVcpuDirtyLimitInfo *info); -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> Implement qemuDomainSetVcpuDirtyLimit, which can be used to set or cancel the upper limit of the dirty page rate for virtual CPUs. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- src/qemu/qemu_driver.c | 131 +++++++++++++++++++++++++++++++++++ src/qemu/qemu_monitor.c | 13 ++++ src/qemu/qemu_monitor.h | 5 ++ src/qemu/qemu_monitor_json.c | 43 ++++++++++++ src/qemu/qemu_monitor_json.h | 5 ++ 5 files changed, 197 insertions(+) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 0d4da937b0..64d97c0fba 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -19876,6 +19876,136 @@ qemuDomainFDAssociate(virDomainPtr domain, return ret; } +static void +qemuDomainSetDirtyLimit(virDomainVcpuDef *vcpu, + unsigned long long rate) +{ + if (rate > 0) { + vcpu->dirtyLimitSet = true; + vcpu->dirty_limit = rate; + } else { + vcpu->dirtyLimitSet = false; + vcpu->dirty_limit = 0; + } +} + +static void +qemuDomainSetVcpuDirtyLimitConfig(virDomainDef *def, + int vcpu, + unsigned long long rate) +{ + def->individualvcpus = true; + + if (vcpu == -1) { + size_t maxvcpus = virDomainDefGetVcpusMax(def); + size_t i; + for (i = 0; i < maxvcpus; i++) { + qemuDomainSetDirtyLimit(virDomainDefGetVcpu(def, i), rate); + } + } else { + qemuDomainSetDirtyLimit(virDomainDefGetVcpu(def, vcpu), rate); + } +} + +static int +qemuDomainSetVcpuDirtyLimitInternal(virQEMUDriver *driver, + virDomainObj *vm, + virDomainDef *def, + virDomainDef *persistentDef, + int vcpu, + unsigned long long rate) +{ + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver); + qemuDomainObjPrivate *priv = vm->privateData; + + VIR_DEBUG("vcpu %d, rate %llu", vcpu, rate); + if (def) { + qemuDomainObjEnterMonitor(vm); + if (qemuMonitorSetVcpuDirtyLimit(priv->mon, vcpu, rate) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Failed to set dirty page rate limit")); + qemuDomainObjExitMonitor(vm); + return -1; + } + qemuDomainObjExitMonitor(vm); + qemuDomainSetVcpuDirtyLimitConfig(def, vcpu, rate); + } + + if (persistentDef) { + qemuDomainSetVcpuDirtyLimitConfig(persistentDef, vcpu, rate); + if (virDomainDefSave(persistentDef, driver->xmlopt, cfg->configDir) < 0) + return -1; + } + + return 0; +} + +static int +qemuDomainSetVcpuDirtyLimit(virDomainPtr domain, + int vcpu, + unsigned long long rate, + unsigned int flags) +{ + virQEMUDriver *driver = domain->conn->privateData; + virDomainObj *vm = NULL; + qemuDomainObjPrivate *priv; + virDomainDef *def = NULL; + virDomainDef *persistentDef = NULL; + int ret = -1; + + virCheckFlags(VIR_DOMAIN_AFFECT_LIVE | + VIR_DOMAIN_AFFECT_CONFIG, -1); + + if (!(vm = qemuDomainObjFromDomain(domain))) + return -1; + + if (virDomainSetVcpuDirtyLimitEnsureACL(domain->conn, vm->def, flags) < 0) + goto cleanup; + + if (virDomainObjBeginJob(vm, VIR_JOB_MODIFY) < 0) + goto cleanup; + + if (virDomainObjGetDefs(vm, flags, &def, &persistentDef) < 0) + goto endjob; + + if (persistentDef) { + if (vcpu >= 0 && vcpu >= (int)virDomainDefGetVcpusMax(persistentDef)) { + virReportError(VIR_ERR_INVALID_ARG, + _("vcpu %1$d is not present in persistent config"), + vcpu); + goto endjob; + } + } + + if (def) { + if (virDomainObjCheckActive(vm) < 0) + goto endjob; + + priv = vm->privateData; + if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_VCPU_DIRTY_LIMIT)) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s", + _("QEMU does not support setting dirty page rate limit")); + goto endjob; + } + + if (vcpu >= 0 && vcpu >= (int)virDomainDefGetVcpusMax(def)) { + virReportError(VIR_ERR_INVALID_ARG, + _("vcpu %1$d is not present in live config"), + vcpu); + goto endjob; + } + } + + ret = qemuDomainSetVcpuDirtyLimitInternal(driver, vm, def, persistentDef, + vcpu, rate); + + endjob: + virDomainObjEndJob(vm); + + cleanup: + virDomainObjEndAPI(&vm); + return ret; +} static virHypervisorDriver qemuHypervisorDriver = { .name = QEMU_DRIVER_NAME, @@ -20126,6 +20256,7 @@ static virHypervisorDriver qemuHypervisorDriver = { .domainStartDirtyRateCalc = qemuDomainStartDirtyRateCalc, /* 7.2.0 */ .domainSetLaunchSecurityState = qemuDomainSetLaunchSecurityState, /* 8.0.0 */ .domainFDAssociate = qemuDomainFDAssociate, /* 9.0.0 */ + .domainSetVcpuDirtyLimit = qemuDomainSetVcpuDirtyLimit, /* 9.7.0 */ }; diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 7053539c7d..90bc0e62c9 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -4500,3 +4500,16 @@ qemuMonitorGetStatsByQOMPath(virJSONValue *arr, return NULL; } + + +int +qemuMonitorSetVcpuDirtyLimit(qemuMonitor *mon, + int vcpu, + unsigned long long rate) +{ + VIR_DEBUG("set vcpu %d dirty page rate limit %llu", vcpu, rate); + + QEMU_CHECK_MONITOR(mon); + + return qemuMonitorJSONSetVcpuDirtyLimit(mon, vcpu, rate); +} diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 6c590933aa..07a05365cf 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -1579,3 +1579,8 @@ qemuMonitorExtractQueryStats(virJSONValue *info); virJSONValue * qemuMonitorGetStatsByQOMPath(virJSONValue *arr, char *qom_path); + +int +qemuMonitorSetVcpuDirtyLimit(qemuMonitor *mon, + int vcpu, + unsigned long long rate); diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 5b9edadcf7..c8f6069566 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -8852,3 +8852,46 @@ qemuMonitorJSONQueryStats(qemuMonitor *mon, return virJSONValueObjectStealArray(reply, "return"); } + +/** + * qemuMonitorJSONSetVcpuDirtyLimit: + * @mon: monitor object + * @vcpu: virtual cpu index to be set, -1 affects all virtual CPUs + * @rate: dirty page rate upper limit to be set, use 0 to disable + * and a positive value to enable + * + * Returns -1 on failure. + */ +int +qemuMonitorJSONSetVcpuDirtyLimit(qemuMonitor *mon, + int vcpu, + unsigned long long rate) +{ + g_autoptr(virJSONValue) cmd = NULL; + g_autoptr(virJSONValue) reply = NULL; + + if (rate != 0) { + /* set the vcpu dirty page rate limit */ + if (!(cmd = qemuMonitorJSONMakeCommand("set-vcpu-dirty-limit", + "k:cpu-index", vcpu, + "U:dirty-rate", rate, + NULL))) { + return -1; + } + } else { + /* cancel the vcpu dirty page rate limit */ + if (!(cmd = qemuMonitorJSONMakeCommand("cancel-vcpu-dirty-limit", + "k:cpu-index", vcpu, + NULL))) { + return -1; + } + } + + if (qemuMonitorJSONCommand(mon, cmd, &reply) < 0) + return -1; + + if (qemuMonitorJSONCheckError(cmd, reply) < 0) + return -1; + + return 0; +} diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h index 06023b98ea..89f61b3052 100644 --- a/src/qemu/qemu_monitor_json.h +++ b/src/qemu/qemu_monitor_json.h @@ -825,3 +825,8 @@ qemuMonitorJSONQueryStats(qemuMonitor *mon, qemuMonitorQueryStatsTargetType target, char **vcpus, GPtrArray *providers); + +int +qemuMonitorJSONSetVcpuDirtyLimit(qemuMonitor *mon, + int vcpu, + unsigned long long rate); -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> Introduce limit-dirty-page-rate virsh api to set or cancel dirty page rate upper limit for virtual CPUs. Usage is below: $ virsh limit-dirty-page-rate <domain> --rate <number> \ [--vcpu <number>] [--config] [--live] [--current] Set the dirty page rate upper limit for the given vcpu specified by the "vcpu"; set for all virtual CPUs if vcpu option is not passed in. Cancel the dirty page rate upper limit if the "rate" option is set to zero. Note that the API requires dirty-ring size configured. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- docs/manpages/virsh.rst | 26 ++++++++++ tools/virsh-domain.c | 109 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 135 insertions(+) diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst index 91e1d5de37..33a8f3b9fe 100644 --- a/docs/manpages/virsh.rst +++ b/docs/manpages/virsh.rst @@ -5273,6 +5273,32 @@ use to avoid keeping them open unnecessarily. Best-effort security label restore may be requested by using the *--seclabel-restore* flag. +limit-dirty-page-rate +--------------------- + +**Syntax:** + +:: + + limit-dirty-page-rate <domain> --rate <number> [--vcpu <number>] + [--config] [--live] [--current] + +Set or cancel a domain's dirty page rate upper limit for the given virtual CPU +specified by *--vcpu*; set for all virtual CPUs if *--vcpu* is not specified; +and cancel the domain's dirty page rate upper limit if *--rate* is set to zero. + +Virtual CPUs will be throttled as needed to keep their dirty page rate within +the limit if the feature enabled. This could, in some scenes, be used to provide +quality-of-service in the aspect of the memory workload for virtual CPUs. + +If *--live* is specified, affect a running domain. +If *--config* is specified, affect the next startup of a persistent domain. +If *--current* is specified, it is equivalent to either *--live* or +*--config*, depending on the current state of the domain. +Both *--live* and *--config* flags may be given, but *--current* is +exclusive. Not specifying any flag is the same as specifying *--current*. + + NODEDEV COMMANDS ================ diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index 7abafe2ba3..ae51d9fdfb 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -13732,6 +13732,109 @@ cmdDomDirtyRateCalc(vshControl *ctl, const vshCmd *cmd) return true; } +/* + * "limit-dirty-page-rate" command + */ +static const vshCmdInfo info_limit_dirty_page_rate[] = { + {.name = "help", + .data = N_("Set or cancel dirty page rate upper limit") + }, + {.name = "desc", + .data = N_("Set or cancel dirty page rate upper limit, " + "require dirty-ring size configured") + }, + {.name = NULL} +}; + +static const vshCmdOptDef opts_limit_dirty_page_rate[] = { + VIRSH_COMMON_OPT_DOMAIN_FULL(0), + {.name = "rate", + .type = VSH_OT_INT, + .flags = VSH_OFLAG_REQ, + .help = N_("Upper limit of dirty page rate (MB/s) for " + "virtual CPUs, use 0 to cancel") + }, + {.name = "vcpu", + .type = VSH_OT_INT, + .help = N_("Index of a virtual CPU") + }, + VIRSH_COMMON_OPT_DOMAIN_PERSISTENT, + VIRSH_COMMON_OPT_DOMAIN_CONFIG, + VIRSH_COMMON_OPT_DOMAIN_LIVE, + VIRSH_COMMON_OPT_DOMAIN_CURRENT, + {.name = NULL} +}; + +static bool +cmdLimitDirtyPageRate(vshControl *ctl, const vshCmd *cmd) +{ + g_autoptr(virshDomain) dom = NULL; + int vcpu_idx = -1; + unsigned long long rate = 0; + unsigned int flags = VIR_DOMAIN_AFFECT_CURRENT; + bool vcpu = vshCommandOptBool(cmd, "vcpu"); + bool current = vshCommandOptBool(cmd, "current"); + bool config = vshCommandOptBool(cmd, "config"); + bool live = vshCommandOptBool(cmd, "live"); + + VSH_EXCLUSIVE_OPTIONS_VAR(current, live); + VSH_EXCLUSIVE_OPTIONS_VAR(current, config); + + if (config) + flags |= VIR_DOMAIN_AFFECT_CONFIG; + if (live) + flags |= VIR_DOMAIN_AFFECT_LIVE; + + if (!(dom = virshCommandOptDomain(ctl, cmd, NULL))) + return false; + + if (vshCommandOptULongLong(ctl, cmd, "rate", &rate) < 0) + return false; + + if (vcpu) { + if (vshCommandOptInt(ctl, cmd, "vcpu", &vcpu_idx) < 0) + return false; + + if (vcpu_idx < 0) { + vshError(ctl, "%s", + _("Invalid vcpu index, using --vcpu to specify cpu index")); + return false; + } + } + + if (vcpu) { + /* Set the dirty page rate upper limit for the specified + * virtual CPU in the given VM; cancel it if rate is set + * to zero. + */ + if (virDomainSetVcpuDirtyLimit(dom, vcpu_idx, + rate, flags) < 0) + return false; + if (rate == 0) + vshPrintExtra(ctl, + _("Cancel vcpu[%1$d] dirty page rate upper limit successfully\n"), + vcpu_idx); + else + vshPrintExtra(ctl, + _("Set vcpu[%1$d] dirty page rate upper limit %2$lld(MB/s) successfully\n"), + vcpu_idx, rate); + } else { + /* Set all dirty page rate upper limits for virtual CPUs in + * the given VM; cancel it if the rate is set to zero. + */ + if (virDomainSetVcpuDirtyLimit(dom, -1, rate, flags) < 0) + return false; + if (rate == 0) + vshPrintExtra(ctl, "%s", + _("Cancel dirty page rate limit for all virtual CPUs successfully\n")); + else + vshPrintExtra(ctl, + _("Set dirty page rate limit %1$lld(MB/s) for all virtual CPUs successfully\n"), + rate); + } + + return true; +} const vshCmdDef domManagementCmds[] = { {.name = "attach-device", @@ -14396,5 +14499,11 @@ const vshCmdDef domManagementCmds[] = { .info = info_dom_fd_associate, .flags = 0 }, + {.name = "limit-dirty-page-rate", + .handler = cmdLimitDirtyPageRate, + .opts = opts_limit_dirty_page_rate, + .info = info_limit_dirty_page_rate, + .flags = 0 + }, {.name = NULL} }; -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> Extend dirtylimit statistics for domGetStats to display the information of the upper limit of dirty page rate for virtual CPUs. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- include/libvirt/libvirt-domain.h | 1 + src/libvirt-domain.c | 9 ++++++ src/qemu/qemu_driver.c | 50 ++++++++++++++++++++++++++++++++ 3 files changed, 60 insertions(+) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 3d3c7cdcba..14fc5ff82e 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -2739,6 +2739,7 @@ typedef enum { VIR_DOMAIN_STATS_MEMORY = (1 << 8), /* return domain memory info (Since: 6.0.0) */ VIR_DOMAIN_STATS_DIRTYRATE = (1 << 9), /* return domain dirty rate info (Since: 7.2.0) */ VIR_DOMAIN_STATS_VM = (1 << 10), /* return vm info (Since: 8.9.0) */ + VIR_DOMAIN_STATS_DIRTYLIMIT = (1 << 11), /* return domain dirty limit info (Since: 9.6.0) */ } virDomainStatsTypes; /** diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 5b505cc519..08c354617c 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -12514,6 +12514,15 @@ virConnectGetDomainCapabilities(virConnectPtr conn, * naming or meaning will stay consistent. Changes to existing fields, * however, are expected to be rare. * + * VIR_DOMAIN_STATS_DIRTYLIMIT: + * Return virtual CPU dirty limit information. The typed parameter keys are in + * this format: + * + * "dirtylimit.vcpu.<num>.limit" - The dirty page rate upper limit for the + * virtual CPU, in MB/s. + * "dirtylimit.vcpu.<num>.current" - The dirty page rate for the virtual CPU + * currently, in MB/s. + * * Note that entire stats groups or individual stat fields may be missing from * the output in case they are not supported by the given hypervisor, are not * applicable for the current state of the guest domain, or their retrieval diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 64d97c0fba..a23c86180d 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -17649,6 +17649,50 @@ qemuDomainGetStatsVm(virQEMUDriver *driver G_GNUC_UNUSED, return 0; } + +static int +qemuDomainGetStatsDirtyLimitMon(virDomainObj *vm, + qemuMonitorVcpuDirtyLimitInfo *info) +{ + qemuDomainObjPrivate *priv = vm->privateData; + int ret; + + qemuDomainObjEnterMonitor(vm); + ret = qemuMonitorQueryVcpuDirtyLimit(priv->mon, info); + qemuDomainObjExitMonitor(vm); + + return ret; +} + + +static int +qemuDomainGetStatsDirtyLimit(virQEMUDriver *driver G_GNUC_UNUSED, + virDomainObj *dom, + virTypedParamList *params, + unsigned int privflags) +{ + qemuMonitorVcpuDirtyLimitInfo info; + size_t i; + + if (!HAVE_JOB(privflags) || !virDomainObjIsActive(dom)) + return 0; + + if (qemuDomainGetStatsDirtyLimitMon(dom, &info) < 0) + return -1; + + for (i = 0; i < info.nvcpus; i++) { + virTypedParamListAddULLong(params, info.limits[i].limit, + "dirtylimit.vcpu.%d.limit", + info.limits[i].idx); + virTypedParamListAddULLong(params, info.limits[i].current, + "dirtylimit.vcpu.%d.current", + info.limits[i].idx); + } + + return 0; +} + + typedef int (*qemuDomainGetStatsFunc)(virQEMUDriver *driver, virDomainObj *dom, @@ -17673,6 +17717,11 @@ static virQEMUCapsFlags queryVmRequired[] = { QEMU_CAPS_LAST }; +static virQEMUCapsFlags queryDirtyLimitRequired[] = { + QEMU_CAPS_VCPU_DIRTY_LIMIT, + QEMU_CAPS_LAST +}; + static struct qemuDomainGetStatsWorker qemuDomainGetStatsWorkers[] = { { qemuDomainGetStatsState, VIR_DOMAIN_STATS_STATE, false, NULL }, { qemuDomainGetStatsCpu, VIR_DOMAIN_STATS_CPU_TOTAL, true, NULL }, @@ -17685,6 +17734,7 @@ static struct qemuDomainGetStatsWorker qemuDomainGetStatsWorkers[] = { { qemuDomainGetStatsMemory, VIR_DOMAIN_STATS_MEMORY, false, NULL }, { qemuDomainGetStatsDirtyRate, VIR_DOMAIN_STATS_DIRTYRATE, true, queryDirtyRateRequired }, { qemuDomainGetStatsVm, VIR_DOMAIN_STATS_VM, true, queryVmRequired }, + { qemuDomainGetStatsDirtyLimit, VIR_DOMAIN_STATS_DIRTYLIMIT, true, queryDirtyLimitRequired }, { NULL, 0, false, NULL } }; -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> Export virDomainDefHasDirtyLimitStartupVcpus as a util function, which could be used in qemu_process.c file for the next commit. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- src/conf/domain_validate.c | 2 +- src/conf/domain_validate.h | 2 ++ src/libvirt_private.syms | 1 + 3 files changed, 4 insertions(+), 1 deletion(-) diff --git a/src/conf/domain_validate.c b/src/conf/domain_validate.c index 2df9db3467..04c18462da 100644 --- a/src/conf/domain_validate.c +++ b/src/conf/domain_validate.c @@ -1785,7 +1785,7 @@ virDomainDefValidateIOThreads(const virDomainDef *def) return 0; } -static int +int virDomainDefHasDirtyLimitStartupVcpus(const virDomainDef *def) { size_t maxvcpus = virDomainDefGetVcpusMax(def); diff --git a/src/conf/domain_validate.h b/src/conf/domain_validate.h index fc441cef5b..ccec3663cc 100644 --- a/src/conf/domain_validate.h +++ b/src/conf/domain_validate.h @@ -47,3 +47,5 @@ int virDomainDiskDefSourceLUNValidate(const virStorageSource *src); int virDomainDefOSValidate(const virDomainDef *def, virDomainXMLOption *xmlopt); + +int virDomainDefHasDirtyLimitStartupVcpus(const virDomainDef *def); diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index f8cda2f773..32656dea7a 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -793,6 +793,7 @@ virDomainDefPostParse; # conf/domain_validate.h virDomainActualNetDefValidate; +virDomainDefHasDirtyLimitStartupVcpus; virDomainDefOSValidate; virDomainDefValidate; virDomainDeviceValidateAliasForHotplug; -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> The dirty_limit attribute in XML requires setting up the upper limit of dirty page rate once after launching the VM, so add the implementation. Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- src/qemu/qemu_process.c | 44 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 7a1cdb0302..560051bcaf 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -6156,6 +6156,46 @@ qemuDomainHasHotpluggableStartupVcpus(virDomainDef *def) } +static int +qemuProcessSetupDirtyLimit(virDomainObj *vm, + virDomainAsyncJob asyncJob) +{ + qemuDomainObjPrivate *priv = vm->privateData; + virDomainDef *def = vm->def; + int ret = -1; + + /* Dirty limit capability is not present, skip the setup */ + if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_VCPU_DIRTY_LIMIT)) + return 0; + + if (virDomainDefHasDirtyLimitStartupVcpus(def)) { + size_t maxvcpus = virDomainDefGetVcpusMax(def); + virDomainVcpuDef *vcpu; + size_t i; + + if (qemuDomainObjEnterMonitorAsync(vm, asyncJob) < 0) + return -1; + + for (i = 0; i < maxvcpus; i++) { + vcpu = virDomainDefGetVcpu(def, i); + + if (vcpu->dirtyLimitSet && (vcpu->dirty_limit != 0)) { + if ((ret = qemuMonitorSetVcpuDirtyLimit(priv->mon, i, vcpu->dirty_limit)) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, + _("Failed to set dirty page rate limit of vcpu[%1$zu]"), i); + qemuDomainObjExitMonitor(vm); + return ret; + } + VIR_DEBUG("Set vcpu[%zu] dirty page rate limit %lld", i, vcpu->dirty_limit); + } + } + qemuDomainObjExitMonitor(vm); + } + + return 0; +} + + static int qemuProcessVcpusSortOrder(const void *a, const void *b) @@ -7833,6 +7873,10 @@ qemuProcessLaunch(virConnectPtr conn, if (qemuProcessUpdateAndVerifyCPU(vm, asyncJob) < 0) goto cleanup; + VIR_DEBUG("Setting Dirty Limit for virtual CPUs"); + if (qemuProcessSetupDirtyLimit(vm, asyncJob) < 0) + goto cleanup; + VIR_DEBUG("Detecting IOThread PIDs"); if (qemuProcessDetectIOThreadPIDs(vm, asyncJob) < 0) goto cleanup; -- 2.38.5
From: Hyman Huang(黄勇) <yong.huang@smartx.com> Introduce command 'virsh domstats --dirtylimit' for reporting dirty page rate upper limit (megabyte/s) infomation. The info is listed as follows: Domain: 'vm' dirtylimit.vcpu.0.limit=10 dirtylimit.vcpu.0.current=16 dirtylimit.vcpu.1.limit=10 dirtylimit.vcpu.1.current=0 ... Signed-off-by: Hyman Huang(黄勇) <yong.huang@smartx.com> --- docs/manpages/virsh.rst | 12 ++++++++++-- tools/virsh-domain-monitor.c | 7 +++++++ 2 files changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst index 33a8f3b9fe..5bd06cb3a5 100644 --- a/docs/manpages/virsh.rst +++ b/docs/manpages/virsh.rst @@ -2317,7 +2317,7 @@ domstats domstats [--raw] [--enforce] [--backing] [--nowait] [--state] [--cpu-total] [--balloon] [--vcpu] [--interface] [--block] [--perf] [--iothread] [--memory] [--dirtyrate] [--vm] - [[--list-active] [--list-inactive] + [--dirtylimit] [[--list-active] [--list-inactive] [--list-persistent] [--list-transient] [--list-running]y [--list-paused] [--list-shutoff] [--list-other]] | [domain ...] @@ -2336,7 +2336,7 @@ The individual statistics groups are selectable via specific flags. By default all supported statistics groups are returned. Supported statistics groups flags are: *--state*, *--cpu-total*, *--balloon*, *--vcpu*, *--interface*, *--block*, *--perf*, *--iothread*, *--memory*, -*--dirtyrate*, *--vm*. +*--dirtyrate*, *--vm*, *--dirtylimit*. Note that - depending on the hypervisor type and version or the domain state - not all of the following statistics may be returned. @@ -2579,6 +2579,14 @@ not available for statistical purposes. The *--vm* option enables reporting of hypervisor-specific statistics. Naming and meaning of the fields is entirely hypervisor dependent. + +*--dirtylimit* returns: + +* ``dirtylimit.vcpu.<num>.limit`` - the upper limit of dirty page rate for a + virtual CPU in megabyte/s +* ``dirtylimit.vcpu.<num>.current`` - the dirty page rate for a virtual CPU + currently in megabyte/s + The statistics in this group have the following naming scheme: ``vm.$NAME.$TYPE`` diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index 89fdc7a050..efa2609719 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -2063,6 +2063,10 @@ static const vshCmdOptDef opts_domstats[] = { .type = VSH_OT_BOOL, .help = N_("report hypervisor-specific statistics"), }, + {.name = "dirtylimit", + .type = VSH_OT_BOOL, + .help = N_("report domain dirty page rate upper limit infomation"), + }, {.name = "list-active", .type = VSH_OT_BOOL, .help = N_("list only active domains"), @@ -2187,6 +2191,9 @@ cmdDomstats(vshControl *ctl, const vshCmd *cmd) if (vshCommandOptBool(cmd, "vm")) stats |= VIR_DOMAIN_STATS_VM; + if (vshCommandOptBool(cmd, "dirtylimit")) + stats |= VIR_DOMAIN_STATS_DIRTYLIMIT; + if (vshCommandOptBool(cmd, "list-active")) flags |= VIR_CONNECT_GET_ALL_DOMAINS_STATS_ACTIVE; -- 2.38.5
On Wed, Sep 06, 2023 at 03:27:10AM +0000, ~hyman wrote:
Hyman Huang(黄勇) (10): qemu_capabilities: Introduce QEMU_CAPS_VCPU_DIRTY_LIMIT capability conf: Introduce XML for dirty limit configuration libvirt: Add virDomainSetVcpuDirtyLimit API qemu_driver: Implement qemuDomainSetVcpuDirtyLimit domain_validate: Export virDomainDefHasDirtyLimitStartupVcpus symbol qemu_process: Setup dirty limit after launching VM virsh: Introduce limit-dirty-page-rate api qemu_monitor: Implement qemuMonitorQueryVcpuDirtyLimit qemu_driver: Extend dirtlimit statistics for domGetStats virsh: Introduce command 'virsh domstats --dirtylimit'
I'm not sure how you do this, but it seems like the e-mail timestamps (when they are being sent) are weird. Like they were set to the time when the commit was authored or something. Just letting you know that there might be something off somewhere.
On Wed, Sep 6, 2023 at 9:16 PM Martin Kletzander <mkletzan@redhat.com> wrote:
On Wed, Sep 06, 2023 at 03:27:10AM +0000, ~hyman wrote:
Hyman Huang(黄勇) (10): qemu_capabilities: Introduce QEMU_CAPS_VCPU_DIRTY_LIMIT capability conf: Introduce XML for dirty limit configuration libvirt: Add virDomainSetVcpuDirtyLimit API qemu_driver: Implement qemuDomainSetVcpuDirtyLimit domain_validate: Export virDomainDefHasDirtyLimitStartupVcpus symbol qemu_process: Setup dirty limit after launching VM virsh: Introduce limit-dirty-page-rate api qemu_monitor: Implement qemuMonitorQueryVcpuDirtyLimit qemu_driver: Extend dirtlimit statistics for domGetStats virsh: Introduce command 'virsh domstats --dirtylimit'
I'm not sure how you do this, but it seems like the e-mail timestamps (when they are being sent) are weird. Like they were set to the time when the commit was authored or something. Just letting you know that there might be something off somewhere.
I'm using the sourcehut(refer to https://sr.ht) to send the email that QEMU recommends as sending properly formatted patch emails is not convenient for me, maybe that's the reason. I'll send the series directly next time. Thanks, Yong -- Best regards
participants (4)
-
Daniel P. Berrangé -
Martin Kletzander -
Yong Huang -
~hyman