9:39 p.m.
In qemu>=8.1, the dirty-limit functionality for live
migration was included. In the live migration
scenario, it implements the force convergence
using the dirty-limit approach, which results in
better reliable read performance. A straightforward
dirty-limit capability for live migration is added by
this patchset. Users might not care about other
dirty-limit arguments like "x-vcpu-dirty-limit-period"
or "vcpu-dirty-limit," thus do not expose them to
Libvirt and leave the default settings in place.
values as default. For more details about dirty-limit.
please see the following reference:
https://lore.kernel.org/qemu-
devel/169024923116.19090.10825599068950039132-0(a)git.sr.ht/
Please review,
thanks, Yong
Hyman Huang (4):
Add VIR_MIGRATE_DIRTY_LIMIT flag
qemu_migration: Implement VIR_MIGRATE_DIRTY_LIMIT flag
virsh: Add support for VIR_MIGRATE_DIRTY_LIMIT flag
NEWS: document support for dirty-limit live migration
NEWS.rst | 8 ++++++++
docs/manpages/virsh.rst | 10 +++++++++-
include/libvirt/libvirt-domain.h | 5 +++++
src/libvirt-domain.c | 8 ++++++++
src/qemu/qemu_migration.c | 8 ++++++++
src/qemu/qemu_migration.h | 1 +
src/qemu/qemu_migration_params.c | 6 ++++++
src/qemu/qemu_migration_params.h | 1 +
tools/virsh-domain.c | 10 ++++++++++
9 files changed, 56 insertions(+), 1 deletion(-)
--
2.38.5
9:09 p.m.
New subject: [PATCH Libvirt 1/4] Add VIR_MIGRATE_DIRTY_LIMIT flag
From: Hyman Huang <yong.huang(a)smartx.com>
The flag can be used to enable dirty-limit capability for
live migration.
Signed-off-by: Hyman Huang <yong.huang(a)smartx.com>
---
include/libvirt/libvirt-domain.h | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index a1902546bb..4a4209c01a 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -1100,6 +1100,11 @@ typedef enum {
* Since: 8.5.0
*/
VIR_MIGRATE_ZEROCOPY = (1 << 20),
+
+ /* Enable dirty-limit capability to make migration convergent.
+ * Since: 9.9.0
+ */
+ VIR_MIGRATE_DIRTY_LIMIT = (1 << 21),
} virDomainMigrateFlags;
--
2.38.5
9:26 p.m.
New subject: [PATCH Libvirt 2/4] qemu_migration: Implement VIR_MIGRATE_DIRTY_LIMIT flag
From: Hyman Huang <yong.huang(a)smartx.com>
Signed-off-by: Hyman Huang <yong.huang(a)smartx.com>
---
src/qemu/qemu_migration.c | 8 ++++++++
src/qemu/qemu_migration.h | 1 +
src/qemu/qemu_migration_params.c | 6 ++++++
src/qemu/qemu_migration_params.h | 1 +
4 files changed, 16 insertions(+)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index 011482c2b5..15cd80df9f 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -2560,6 +2560,14 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver,
if (!qemuMigrationSrcIsAllowed(vm, true, vm->job->asyncJob, flags))
return NULL;
+ if (flags & VIR_MIGRATE_DIRTY_LIMIT &&
+ (!(vm->def->features[VIR_DOMAIN_FEATURE_KVM] == VIR_TRISTATE_SWITCH_ON
&&
+ vm->def->kvm_features->features[VIR_DOMAIN_KVM_DIRTY_RING] ==
VIR_TRISTATE_SWITCH_ON))) {
+ virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s",
+ _("dirtylimit migration requires dirty-ring features
enabled"));
+ return NULL;
+ }
+
if (!(flags & (VIR_MIGRATE_UNSAFE | VIR_MIGRATE_OFFLINE)) &&
!qemuMigrationSrcIsSafe(vm->def, priv->qemuCaps,
nmigrate_disks, migrate_disks, flags))
diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h
index ed62fd4a91..a2a598fe28 100644
--- a/src/qemu/qemu_migration.h
+++ b/src/qemu/qemu_migration.h
@@ -62,6 +62,7 @@
VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES | \
VIR_MIGRATE_POSTCOPY_RESUME | \
VIR_MIGRATE_ZEROCOPY | \
+ VIR_MIGRATE_DIRTY_LIMIT | \
0)
/* All supported migration parameters and their types. */
diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c
index 79fe6e97c8..797293d8e7 100644
--- a/src/qemu/qemu_migration_params.c
+++ b/src/qemu/qemu_migration_params.c
@@ -99,6 +99,7 @@ VIR_ENUM_IMPL(qemuMigrationCapability,
"dirty-bitmaps",
"return-path",
"zero-copy-send",
+ "dirty-limit",
);
@@ -194,6 +195,11 @@ static const qemuMigrationParamsFlagMapItem
qemuMigrationParamsFlagMap[] = {
VIR_MIGRATE_ZEROCOPY,
QEMU_MIGRATION_CAP_ZERO_COPY_SEND,
QEMU_MIGRATION_SOURCE},
+
+ {QEMU_MIGRATION_FLAG_REQUIRED,
+ VIR_MIGRATE_DIRTY_LIMIT,
+ QEMU_MIGRATION_CAP_DIRTY_LIMIT,
+ QEMU_MIGRATION_SOURCE},
};
/* Translation from VIR_MIGRATE_PARAM_* typed parameters to
diff --git a/src/qemu/qemu_migration_params.h b/src/qemu/qemu_migration_params.h
index 5857673227..44a44aae07 100644
--- a/src/qemu/qemu_migration_params.h
+++ b/src/qemu/qemu_migration_params.h
@@ -40,6 +40,7 @@ typedef enum {
QEMU_MIGRATION_CAP_BLOCK_DIRTY_BITMAPS,
QEMU_MIGRATION_CAP_RETURN_PATH,
QEMU_MIGRATION_CAP_ZERO_COPY_SEND,
+ QEMU_MIGRATION_CAP_DIRTY_LIMIT,
QEMU_MIGRATION_CAP_LAST
} qemuMigrationCapability;
--
2.38.5
8:15 p.m.
New subject: [PATCH Libvirt 3/4] virsh: Add support for VIR_MIGRATE_DIRTY_LIMIT flag
From: Hyman Huang <yong.huang(a)smartx.com>
Introudce option to enable dirty-limit convergence algorithim
during live migration.
Signed-off-by: Hyman Huang <yong.huang(a)smartx.com>
---
docs/manpages/virsh.rst | 10 +++++++++-
src/libvirt-domain.c | 8 ++++++++
tools/virsh-domain.c | 10 ++++++++++
3 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst
index c425974912..9342dd4cfd 100644
--- a/docs/manpages/virsh.rst
+++ b/docs/manpages/virsh.rst
@@ -3373,7 +3373,7 @@ migrate
[--compressed] [--comp-methods method-list]
[--comp-mt-level] [--comp-mt-threads] [--comp-mt-dthreads]
[--comp-xbzrle-cache] [--comp-zlib-level] [--comp-zstd-level]
- [--auto-converge] [auto-converge-initial]
+ [--dirty-limit] [--auto-converge] [auto-converge-initial]
[auto-converge-increment] [--persistent-xml file] [--tls]
[--postcopy-bandwidth bandwidth]
[--parallel [--parallel-connections connections]]
@@ -3429,6 +3429,14 @@ source or destination host and the ``migrate`` command will report
an error
leaving the domain active on both hosts. To recover from such situation repeat
the original ``migrate`` command with an additional *--postcopy-resume* flag.
+*--dirty-limit* forces convergence using dirty-limit algorithms during live
+migration. For QEMU/KVM, this means migration will throttle vCPUs as needed to
+keep their dirty page rate within the migration parameter ``vcpu-dirty-limit``
+(1 megabits/s by default). This can improve the responsiveness of large guests
+during live migration and result in more stable read performance. It requires
+dirty-ring size configuration and conflicts with the traditional
+``auto-converge`` algorithm.
+
*--auto-converge* forces convergence during live migration. The initial
guest CPU throttling rate can be set with *auto-converge-initial*. If the
initial throttling rate is not enough to ensure convergence, the rate is
diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c
index 6616294fc1..f1a7681da5 100644
--- a/src/libvirt-domain.c
+++ b/src/libvirt-domain.c
@@ -4213,6 +4213,10 @@ virDomainMigrate3(virDomainPtr domain,
/* Now checkout the destination */
virCheckReadOnlyGoto(dconn->flags, error);
+ VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_AUTO_CONVERGE,
+ VIR_MIGRATE_DIRTY_LIMIT,
+ error);
+
VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_NON_SHARED_DISK,
VIR_MIGRATE_NON_SHARED_INC,
error);
@@ -4637,6 +4641,10 @@ virDomainMigrateToURI3(virDomainPtr domain,
virCheckDomainReturn(domain, -1);
virCheckReadOnlyGoto(domain->conn->flags, error);
+ VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_AUTO_CONVERGE,
+ VIR_MIGRATE_DIRTY_LIMIT,
+ error);
+
VIR_EXCLUSIVE_FLAGS_GOTO(VIR_MIGRATE_TUNNELLED,
VIR_MIGRATE_PARALLEL,
error);
diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index 7abafe2ba3..31ee31b394 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -10922,6 +10922,10 @@ static const vshCmdOptDef opts_migrate[] = {
.type = VSH_OT_BOOL,
.help = N_("compress repeated pages during live migration")
},
+ {.name = "dirty-limit",
+ .type = VSH_OT_BOOL,
+ .help = N_("force convergence with dirty-limit algorithim during live
migration")
+ },
{.name = "auto-converge",
.type = VSH_OT_BOOL,
.help = N_("force convergence during live migration")
@@ -11108,6 +11112,7 @@ doMigrate(void *opaque)
{ "change-protection", VIR_MIGRATE_CHANGE_PROTECTION },
{ "unsafe", VIR_MIGRATE_UNSAFE },
{ "compressed", VIR_MIGRATE_COMPRESSED },
+ { "dirty-limit", VIR_MIGRATE_DIRTY_LIMIT },
{ "auto-converge", VIR_MIGRATE_AUTO_CONVERGE },
{ "rdma-pin-all", VIR_MIGRATE_RDMA_PIN_ALL },
{ "offline", VIR_MIGRATE_OFFLINE },
@@ -11134,6 +11139,11 @@ doMigrate(void *opaque)
flags |= flagmap[i].migflag;
}
+ if (flags & VIR_MIGRATE_DIRTY_LIMIT && flags &
VIR_MIGRATE_AUTO_CONVERGE) {
+ vshError(ctl, "'--dirty-limit' conflicts with
'--auto-converge'");
+ goto out;
+ }
+
if (flags & VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES &&
!(flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_DISK))) {
vshError(ctl, "'--copy-storage-synchronous-writes' requires one of
'--copy-storage-all', '--copy-storage-inc'");
--
2.38.5
9:13 p.m.
New subject: [PATCH Libvirt 4/4] NEWS: document support for dirty-limit live migration
From: Hyman Huang <yong.huang(a)smartx.com>
Signed-off-by: Hyman Huang <yong.huang(a)smartx.com>
---
NEWS.rst | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/NEWS.rst b/NEWS.rst
index fbdd9674a3..dd0dda923e 100644
--- a/NEWS.rst
+++ b/NEWS.rst
@@ -25,6 +25,14 @@ v9.9.0 (unreleased)
guest features to see if the current libvirt supports both deleting
and reverting external snapshots.
+ * QEMU: Support dirty-limit live migration
+
+ Introduce ``VIR_MIGRATE_DIRTY_LIMIT`` flag for live migration API,
+ which can enable the dirty-limit convergence algorithm during live
+ migration and reap more stable read performance for guest vCPUs.
+
+ Also, add the corresponding ``--dirty-limit`` option to ``virsh migrate``.
+
* **Improvements**
* **Bug fixes**
--
2.38.5
4:05 a.m.
ping1
On Thu, Oct 12, 2023 at 10:39 AM ~hyman <hyman(a)git.sr.ht> wrote:
In qemu>=8.1, the dirty-limit functionality for live
migration was included. In the live migration
scenario, it implements the force convergence
using the dirty-limit approach, which results in
better reliable read performance. A straightforward
dirty-limit capability for live migration is added by
this patchset. Users might not care about other
dirty-limit arguments like "x-vcpu-dirty-limit-period"
or "vcpu-dirty-limit," thus do not expose them to
Libvirt and leave the default settings in place.
values as default. For more details about dirty-limit.
please see the following reference:
https://lore.kernel.org/qemu-
devel/169024923116.19090.10825599068950039132-0(a)git.sr.ht/
Please review,
thanks, Yong
Hyman Huang (4):
Add VIR_MIGRATE_DIRTY_LIMIT flag
qemu_migration: Implement VIR_MIGRATE_DIRTY_LIMIT flag
virsh: Add support for VIR_MIGRATE_DIRTY_LIMIT flag
NEWS: document support for dirty-limit live migration
NEWS.rst | 8 ++++++++
docs/manpages/virsh.rst | 10 +++++++++-
include/libvirt/libvirt-domain.h | 5 +++++
src/libvirt-domain.c | 8 ++++++++
src/qemu/qemu_migration.c | 8 ++++++++
src/qemu/qemu_migration.h | 1 +
src/qemu/qemu_migration_params.c | 6 ++++++
src/qemu/qemu_migration_params.h | 1 +
tools/virsh-domain.c | 10 ++++++++++
9 files changed, 56 insertions(+), 1 deletion(-)
--
2.38.5
--
Best regards
403
days inactive
408
days old
5 comments
2 participants
participants (2)
-
Yong Huang -
~hyman