[libvirt PATCH 0/3] Introduce VIR_MIGRATE_POSTCOPY_PREEMPT flag
See 1/3 for details. https://issues.redhat.com/browse/RHEL-7100 Jiri Denemark (3): Introduce VIR_MIGRATE_POSTCOPY_PREEMPT flag virsh migrate: Add --postcopy-preempt option qemu_migration: Implement VIR_MIGRATE_POSTCOPY_PREEMPT docs/manpages/virsh.rst | 9 ++++++--- include/libvirt/libvirt-domain.h | 8 ++++++++ src/qemu/qemu_migration.c | 21 ++++++++++++++++++--- src/qemu/qemu_migration.h | 1 + src/qemu/qemu_migration_params.c | 6 ++++++ src/qemu/qemu_migration_params.h | 1 + tools/virsh-domain.c | 6 ++++++ 7 files changed, 46 insertions(+), 6 deletions(-) -- 2.43.0
During post-copy migration (once it actually switches to post-copy mode) dirty memory pages are continued to be migrated iteratively, while the destination can explicitly request a specific page to be migrated before the iterative process gets to it (which happens when a guest wants to read a page that was not migrated yet). Without this flag explicitly requested pages need to wait until all other pages already queued are transferred. Enabling this flag will instruct the hypervisor to create a separate migration channel for explicitly requested pages so that they can preempt the queue. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 30cce85b29..ee23ae97a5 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1100,6 +1100,14 @@ typedef enum { * Since: 8.5.0 */ VIR_MIGRATE_ZEROCOPY = (1 << 20), + + /* Allow post-copy requests to preempt normal migration stream. This way + * page faults on the destination hosts caused by accessing a memory page + * that was not migrated yet should be handled faster. + * + * Since: 10.0.0 + */ + VIR_MIGRATE_POSTCOPY_PREEMPT = (1 << 21), } virDomainMigrateFlags; -- 2.43.0
During post-copy migration (once it actually switches to post-copy mode) dirty memory pages are continued to be migrated iteratively, while the destination can explicitly request a specific page to be migrated before the iterative process gets to it (which happens when a guest wants to read a page that was not migrated yet). Without this flag explicitly requested pages need to wait until all other pages already queued are transferred. Enabling this flag will instruct the hypervisor to create a separate migration channel for explicitly requested pages so that they can preempt the queue.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 8 ++++++++ 1 file changed, 8 insertions(+)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 30cce85b29..ee23ae97a5 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1100,6 +1100,14 @@ typedef enum { * Since: 8.5.0 */ VIR_MIGRATE_ZEROCOPY = (1 << 20), + + /* Allow post-copy requests to preempt normal migration stream. This way + * page faults on the destination hosts caused by accessing a memory page + * that was not migrated yet should be handled faster.
Based on this description and the commit message there doesn't seem to be any reason why the user should not enable that. And in that case I think
On Fri, Jan 5, 2024 at 4:55 PM Jiri Denemark <jdenemar@redhat.com> wrote: the users will be better off when libvirt auto-enables this.
On Fri, Jan 05, 2024 at 17:11:57 +0100, Peter Krempa wrote:
On Fri, Jan 5, 2024 at 4:55 PM Jiri Denemark <jdenemar@redhat.com> wrote:
During post-copy migration (once it actually switches to post-copy mode) dirty memory pages are continued to be migrated iteratively, while the destination can explicitly request a specific page to be migrated before the iterative process gets to it (which happens when a guest wants to read a page that was not migrated yet). Without this flag explicitly requested pages need to wait until all other pages already queued are transferred. Enabling this flag will instruct the hypervisor to create a separate migration channel for explicitly requested pages so that they can preempt the queue.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 8 ++++++++ 1 file changed, 8 insertions(+)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 30cce85b29..ee23ae97a5 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1100,6 +1100,14 @@ typedef enum { * Since: 8.5.0 */ VIR_MIGRATE_ZEROCOPY = (1 << 20), + + /* Allow post-copy requests to preempt normal migration stream. This way + * page faults on the destination hosts caused by accessing a memory page + * that was not migrated yet should be handled faster.
Based on this description and the commit message there doesn't seem to be any reason why the user should not enable that. And in that case I think the users will be better off when libvirt auto-enables this.
Well yes, ideally you want this enabled all the time. But Peter Xu did not sound like it was ready for being always enabled. I guess I'll check with him if it's still the case or not. Jirka
On Fri, Jan 05, 2024 at 17:31:50 +0100, Jiri Denemark wrote:
On Fri, Jan 05, 2024 at 17:11:57 +0100, Peter Krempa wrote:
On Fri, Jan 5, 2024 at 4:55 PM Jiri Denemark <jdenemar@redhat.com> wrote:
During post-copy migration (once it actually switches to post-copy mode) dirty memory pages are continued to be migrated iteratively, while the destination can explicitly request a specific page to be migrated before the iterative process gets to it (which happens when a guest wants to read a page that was not migrated yet). Without this flag explicitly requested pages need to wait until all other pages already queued are transferred. Enabling this flag will instruct the hypervisor to create a separate migration channel for explicitly requested pages so that they can preempt the queue.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 8 ++++++++ 1 file changed, 8 insertions(+)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 30cce85b29..ee23ae97a5 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1100,6 +1100,14 @@ typedef enum { * Since: 8.5.0 */ VIR_MIGRATE_ZEROCOPY = (1 << 20), + + /* Allow post-copy requests to preempt normal migration stream. This way + * page faults on the destination hosts caused by accessing a memory page + * that was not migrated yet should be handled faster.
Based on this description and the commit message there doesn't seem to be any reason why the user should not enable that. And in that case I think the users will be better off when libvirt auto-enables this.
Well yes, ideally you want this enabled all the time. But Peter Xu did not sound like it was ready for being always enabled. I guess I'll check with him if it's still the case or not.
So Peter confirmed we may enable postcopy-preempt for all post-copy migrations (as long as the capability is supported). Reworked patch series is coming in a few moments. Jirka
This new option will enable VIR_MIGRATE_POSTCOPY_PREEMPT flag. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- docs/manpages/virsh.rst | 9 ++++++--- tools/virsh-domain.c | 6 ++++++ 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst index ed1027e133..4d55665ac1 100644 --- a/docs/manpages/virsh.rst +++ b/docs/manpages/virsh.rst @@ -3373,9 +3373,9 @@ migrate [--persistent] [--undefinesource] [--suspend] [--copy-storage-all] [--copy-storage-inc] [--change-protection] [--unsafe] [--verbose] [--rdma-pin-all] [--abort-on-error] [--postcopy] - [--postcopy-after-precopy] [--postcopy-resume] [--zerocopy] - domain desturi [migrateuri] [graphicsuri] [listen-address] [dname] - [--timeout seconds [--timeout-suspend | --timeout-postcopy]] + [--postcopy-after-precopy] [--postcopy-resume] [--postcopy-preempt] + [--zerocopy] domain desturi [migrateuri] [graphicsuri] [listen-address] + [dname] [--timeout seconds [--timeout-suspend | --timeout-postcopy]] [--xml file] [--migrate-disks disk-list] [--disks-port port] [--compressed] [--comp-methods method-list] [--comp-mt-level] [--comp-mt-threads] [--comp-mt-dthreads] @@ -3428,6 +3428,9 @@ Once migration is running, the user may switch to post-copy using the ``migrate-postcopy`` command sent from another virsh instance or use *--postcopy-after-precopy* along with *--postcopy* to let libvirt automatically switch to post-copy after the first pass of pre-copy is finished. +The *--postcopy-preempt* option can be used to speed up handling page faults on +the destination host caused by accessing a memory page that was not migrated yet +by letting such pages preempt migration of other memory pages. The maximum bandwidth consumed during the post-copy phase may be limited using *--postcopy-bandwidth*. The maximum bandwidth consumed during the pre-copy phase may be limited using *--bandwidth*. In case connection between the hosts breaks diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index 58d5a4ab57..25b41cab22 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -11014,6 +11014,10 @@ static const vshCmdOptDef opts_migrate[] = { .type = VSH_OT_BOOL, .help = N_("resume failed post-copy migration") }, + {.name = "postcopy-preempt", + .type = VSH_OT_BOOL, + .help = N_("let post-copy requests preempt normal migration stream") + }, {.name = "zerocopy", .type = VSH_OT_BOOL, .help = N_("use zero-copy mechanism for migrating memory pages") @@ -11182,6 +11186,7 @@ doMigrate(void *opaque) { "abort-on-error", VIR_MIGRATE_ABORT_ON_ERROR }, { "postcopy", VIR_MIGRATE_POSTCOPY }, { "postcopy-resume", VIR_MIGRATE_POSTCOPY_RESUME }, + { "postcopy-preempt", VIR_MIGRATE_POSTCOPY_PREEMPT }, { "zerocopy", VIR_MIGRATE_ZEROCOPY }, { "tls", VIR_MIGRATE_TLS }, { "parallel", VIR_MIGRATE_PARALLEL }, @@ -11530,6 +11535,7 @@ cmdMigrate(vshControl *ctl, const vshCmd *cmd) VSH_EXCLUSIVE_OPTIONS("copy-storage-all", "copy-storage-inc"); VSH_REQUIRE_OPTION("postcopy-after-precopy", "postcopy"); VSH_REQUIRE_OPTION("postcopy-resume", "postcopy"); + VSH_REQUIRE_OPTION("postcopy-preempt", "postcopy"); VSH_REQUIRE_OPTION("timeout-postcopy", "postcopy"); VSH_REQUIRE_OPTION("persistent-xml", "persistent"); VSH_REQUIRE_OPTION("tls-destination", "tls"); -- 2.43.0
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 21 ++++++++++++++++++--- src/qemu/qemu_migration.h | 1 + src/qemu/qemu_migration_params.c | 6 ++++++ src/qemu/qemu_migration_params.h | 1 + 4 files changed, 26 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 3ba0aa502b..09445354a9 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2611,6 +2611,13 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, return NULL; } + if (flags & VIR_MIGRATE_POSTCOPY_PREEMPT && + !(flags & VIR_MIGRATE_POSTCOPY)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("post-copy preemption can only be enabled for post-copy migration")); + return NULL; + } + if (flags & VIR_MIGRATE_OFFLINE) { if (flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC)) { @@ -3605,6 +3612,13 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, return -1; } + if (flags & VIR_MIGRATE_POSTCOPY_PREEMPT && + !(flags & VIR_MIGRATE_POSTCOPY)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("post-copy preemption can only be enabled for post-copy migration")); + return -1; + } + if (cfg->migrateTLSForce && !(flags & VIR_MIGRATE_TUNNELLED) && !(flags & VIR_MIGRATE_TLS)) { @@ -5210,10 +5224,11 @@ qemuMigrationSrcPerformNative(virQEMUDriver *driver, spec.dest.socket.path = uribits->path; } else { - /* RDMA and multi-fd migration requires QEMU to connect to the destination - * itself. + /* RDMA, multi-fd, and postcopy-preempt migration requires QEMU to + * connect to the destination itself. */ - if (STREQ(uribits->scheme, "rdma") || (flags & VIR_MIGRATE_PARALLEL)) + if (STREQ(uribits->scheme, "rdma") || + flags & (VIR_MIGRATE_PARALLEL | VIR_MIGRATE_POSTCOPY_PREEMPT)) spec.destType = MIGRATION_DEST_HOST; else spec.destType = MIGRATION_DEST_CONNECT_HOST; diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index ed62fd4a91..27fb783359 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -62,6 +62,7 @@ VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES | \ VIR_MIGRATE_POSTCOPY_RESUME | \ VIR_MIGRATE_ZEROCOPY | \ + VIR_MIGRATE_POSTCOPY_PREEMPT | \ 0) /* All supported migration parameters and their types. */ diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c index 79fe6e97c8..35eafd8f0d 100644 --- a/src/qemu/qemu_migration_params.c +++ b/src/qemu/qemu_migration_params.c @@ -99,6 +99,7 @@ VIR_ENUM_IMPL(qemuMigrationCapability, "dirty-bitmaps", "return-path", "zero-copy-send", + "postcopy-preempt", ); @@ -194,6 +195,11 @@ static const qemuMigrationParamsFlagMapItem qemuMigrationParamsFlagMap[] = { VIR_MIGRATE_ZEROCOPY, QEMU_MIGRATION_CAP_ZERO_COPY_SEND, QEMU_MIGRATION_SOURCE}, + + {QEMU_MIGRATION_FLAG_REQUIRED, + VIR_MIGRATE_POSTCOPY_PREEMPT, + QEMU_MIGRATION_CAP_POSTCOPY_PREEMPT, + QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION}, }; /* Translation from VIR_MIGRATE_PARAM_* typed parameters to diff --git a/src/qemu/qemu_migration_params.h b/src/qemu/qemu_migration_params.h index 5857673227..74f1680e9e 100644 --- a/src/qemu/qemu_migration_params.h +++ b/src/qemu/qemu_migration_params.h @@ -40,6 +40,7 @@ typedef enum { QEMU_MIGRATION_CAP_BLOCK_DIRTY_BITMAPS, QEMU_MIGRATION_CAP_RETURN_PATH, QEMU_MIGRATION_CAP_ZERO_COPY_SEND, + QEMU_MIGRATION_CAP_POSTCOPY_PREEMPT, QEMU_MIGRATION_CAP_LAST } qemuMigrationCapability; -- 2.43.0
On a Friday in 2024, Jiri Denemark wrote:
See 1/3 for details.
https://issues.redhat.com/browse/RHEL-7100
Jiri Denemark (3): Introduce VIR_MIGRATE_POSTCOPY_PREEMPT flag virsh migrate: Add --postcopy-preempt option qemu_migration: Implement VIR_MIGRATE_POSTCOPY_PREEMPT
docs/manpages/virsh.rst | 9 ++++++--- include/libvirt/libvirt-domain.h | 8 ++++++++ src/qemu/qemu_migration.c | 21 ++++++++++++++++++--- src/qemu/qemu_migration.h | 1 + src/qemu/qemu_migration_params.c | 6 ++++++ src/qemu/qemu_migration_params.h | 1 + tools/virsh-domain.c | 6 ++++++ 7 files changed, 46 insertions(+), 6 deletions(-)
Reviewed-by: Ján Tomko <jtomko@redhat.com> Jano
participants (3)
-
Jiri Denemark -
Ján Tomko -
Peter Krempa