9:53 a.m.
See 1/3 for details.
https://issues.redhat.com/browse/RHEL-7100
Jiri Denemark (3):
Introduce VIR_MIGRATE_POSTCOPY_PREEMPT flag
virsh migrate: Add --postcopy-preempt option
qemu_migration: Implement VIR_MIGRATE_POSTCOPY_PREEMPT
docs/manpages/virsh.rst | 9 ++++++---
include/libvirt/libvirt-domain.h | 8 ++++++++
src/qemu/qemu_migration.c | 21 ++++++++++++++++++---
src/qemu/qemu_migration.h | 1 +
src/qemu/qemu_migration_params.c | 6 ++++++
src/qemu/qemu_migration_params.h | 1 +
tools/virsh-domain.c | 6 ++++++
7 files changed, 46 insertions(+), 6 deletions(-)
--
2.43.0
9:53 a.m.
New subject: [libvirt PATCH 1/3] Introduce VIR_MIGRATE_POSTCOPY_PREEMPT flag
During post-copy migration (once it actually switches to post-copy mode)
dirty memory pages are continued to be migrated iteratively, while the
destination can explicitly request a specific page to be migrated before
the iterative process gets to it (which happens when a guest wants to
read a page that was not migrated yet). Without this flag explicitly
requested pages need to wait until all other pages already queued are
transferred. Enabling this flag will instruct the hypervisor to create a
separate migration channel for explicitly requested pages so that they
can preempt the queue.
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
include/libvirt/libvirt-domain.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index 30cce85b29..ee23ae97a5 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -1100,6 +1100,14 @@ typedef enum {
* Since: 8.5.0
*/
VIR_MIGRATE_ZEROCOPY = (1 << 20),
+
+ /* Allow post-copy requests to preempt normal migration stream. This way
+ * page faults on the destination hosts caused by accessing a memory page
+ * that was not migrated yet should be handled faster.
+ *
+ * Since: 10.0.0
+ */
+ VIR_MIGRATE_POSTCOPY_PREEMPT = (1 << 21),
} virDomainMigrateFlags;
--
2.43.0
10:11 a.m.
New subject: [libvirt PATCH 1/3] Introduce VIR_MIGRATE_POSTCOPY_PREEMPT flag
On Fri, Jan 5, 2024 at 4:55 PM Jiri Denemark <jdenemar(a)redhat.com> wrote:
During post-copy migration (once it actually switches to post-copy
mode)
dirty memory pages are continued to be migrated iteratively, while the
destination can explicitly request a specific page to be migrated before
the iterative process gets to it (which happens when a guest wants to
read a page that was not migrated yet). Without this flag explicitly
requested pages need to wait until all other pages already queued are
transferred. Enabling this flag will instruct the hypervisor to create a
separate migration channel for explicitly requested pages so that they
can preempt the queue.
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
include/libvirt/libvirt-domain.h | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/include/libvirt/libvirt-domain.h
b/include/libvirt/libvirt-domain.h
index 30cce85b29..ee23ae97a5 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -1100,6 +1100,14 @@ typedef enum {
* Since: 8.5.0
*/
VIR_MIGRATE_ZEROCOPY = (1 << 20),
+
+ /* Allow post-copy requests to preempt normal migration stream. This
way
+ * page faults on the destination hosts caused by accessing a memory
page
+ * that was not migrated yet should be handled faster.
Based on this description and the commit message there doesn't seem to be
any
reason why the user should not enable that. And in that case I think
the users will be better off when libvirt auto-enables this.
10:31 a.m.
New subject: [libvirt PATCH 1/3] Introduce VIR_MIGRATE_POSTCOPY_PREEMPT
flag
On Fri, Jan 05, 2024 at 17:11:57 +0100, Peter Krempa wrote:
On Fri, Jan 5, 2024 at 4:55 PM Jiri Denemark
<jdenemar(a)redhat.com> wrote:
> During post-copy migration (once it actually switches to post-copy mode)
> dirty memory pages are continued to be migrated iteratively, while the
> destination can explicitly request a specific page to be migrated before
> the iterative process gets to it (which happens when a guest wants to
> read a page that was not migrated yet). Without this flag explicitly
> requested pages need to wait until all other pages already queued are
> transferred. Enabling this flag will instruct the hypervisor to create a
> separate migration channel for explicitly requested pages so that they
> can preempt the queue.
>
> Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
> ---
> include/libvirt/libvirt-domain.h | 8 ++++++++
> 1 file changed, 8 insertions(+)
>
> diff --git a/include/libvirt/libvirt-domain.h
> b/include/libvirt/libvirt-domain.h
> index 30cce85b29..ee23ae97a5 100644
> --- a/include/libvirt/libvirt-domain.h
> +++ b/include/libvirt/libvirt-domain.h
> @@ -1100,6 +1100,14 @@ typedef enum {
> * Since: 8.5.0
> */
> VIR_MIGRATE_ZEROCOPY = (1 << 20),
> +
> + /* Allow post-copy requests to preempt normal migration stream. This
> way
> + * page faults on the destination hosts caused by accessing a memory
> page
> + * that was not migrated yet should be handled faster.
>
> Based on this description and the commit message there doesn't seem to be
any reason why the user should not enable that. And in that case I think
the users will be better off when libvirt auto-enables this.
Well yes, ideally you want this enabled all the time. But Peter Xu did
not sound like it was ready for being always enabled. I guess I'll check
with him if it's still the case or not.
Jirka
11:01 a.m.
New subject: [libvirt PATCH 1/3] Introduce
VIR_MIGRATE_POSTCOPY_PREEMPT flag
On Fri, Jan 05, 2024 at 17:31:50 +0100, Jiri Denemark wrote:
On Fri, Jan 05, 2024 at 17:11:57 +0100, Peter Krempa wrote:
> On Fri, Jan 5, 2024 at 4:55 PM Jiri Denemark <jdenemar(a)redhat.com> wrote:
>
> > During post-copy migration (once it actually switches to post-copy mode)
> > dirty memory pages are continued to be migrated iteratively, while the
> > destination can explicitly request a specific page to be migrated before
> > the iterative process gets to it (which happens when a guest wants to
> > read a page that was not migrated yet). Without this flag explicitly
> > requested pages need to wait until all other pages already queued are
> > transferred. Enabling this flag will instruct the hypervisor to create a
> > separate migration channel for explicitly requested pages so that they
> > can preempt the queue.
> >
> > Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
> > ---
> > include/libvirt/libvirt-domain.h | 8 ++++++++
> > 1 file changed, 8 insertions(+)
> >
> > diff --git a/include/libvirt/libvirt-domain.h
> > b/include/libvirt/libvirt-domain.h
> > index 30cce85b29..ee23ae97a5 100644
> > --- a/include/libvirt/libvirt-domain.h
> > +++ b/include/libvirt/libvirt-domain.h
> > @@ -1100,6 +1100,14 @@ typedef enum {
> > * Since: 8.5.0
> > */
> > VIR_MIGRATE_ZEROCOPY = (1 << 20),
> > +
> > + /* Allow post-copy requests to preempt normal migration stream. This
> > way
> > + * page faults on the destination hosts caused by accessing a memory
> > page
> > + * that was not migrated yet should be handled faster.
> >
> > Based on this description and the commit message there doesn't seem to be
> any reason why the user should not enable that. And in that case I think
> the users will be better off when libvirt auto-enables this.
Well yes, ideally you want this enabled all the time. But Peter Xu did
not sound like it was ready for being always enabled. I guess I'll check
with him if it's still the case or not.
So Peter confirmed we may enable postcopy-preempt for all post-copy
migrations (as long as the capability is supported). Reworked patch
series is coming in a few moments.
Jirka
9:53 a.m.
New subject: [libvirt PATCH 2/3] virsh migrate: Add --postcopy-preempt option
This new option will enable VIR_MIGRATE_POSTCOPY_PREEMPT flag.
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
docs/manpages/virsh.rst | 9 ++++++---
tools/virsh-domain.c | 6 ++++++
2 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst
index ed1027e133..4d55665ac1 100644
--- a/docs/manpages/virsh.rst
+++ b/docs/manpages/virsh.rst
@@ -3373,9 +3373,9 @@ migrate
[--persistent] [--undefinesource] [--suspend] [--copy-storage-all]
[--copy-storage-inc] [--change-protection] [--unsafe] [--verbose]
[--rdma-pin-all] [--abort-on-error] [--postcopy]
- [--postcopy-after-precopy] [--postcopy-resume] [--zerocopy]
- domain desturi [migrateuri] [graphicsuri] [listen-address] [dname]
- [--timeout seconds [--timeout-suspend | --timeout-postcopy]]
+ [--postcopy-after-precopy] [--postcopy-resume] [--postcopy-preempt]
+ [--zerocopy] domain desturi [migrateuri] [graphicsuri] [listen-address]
+ [dname] [--timeout seconds [--timeout-suspend | --timeout-postcopy]]
[--xml file] [--migrate-disks disk-list] [--disks-port port]
[--compressed] [--comp-methods method-list]
[--comp-mt-level] [--comp-mt-threads] [--comp-mt-dthreads]
@@ -3428,6 +3428,9 @@ Once migration is running, the user may switch to post-copy using
the
``migrate-postcopy`` command sent from another virsh instance or use
*--postcopy-after-precopy* along with *--postcopy* to let libvirt
automatically switch to post-copy after the first pass of pre-copy is finished.
+The *--postcopy-preempt* option can be used to speed up handling page faults on
+the destination host caused by accessing a memory page that was not migrated yet
+by letting such pages preempt migration of other memory pages.
The maximum bandwidth consumed during the post-copy phase may be limited using
*--postcopy-bandwidth*. The maximum bandwidth consumed during the pre-copy phase
may be limited using *--bandwidth*. In case connection between the hosts breaks
diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index 58d5a4ab57..25b41cab22 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -11014,6 +11014,10 @@ static const vshCmdOptDef opts_migrate[] = {
.type = VSH_OT_BOOL,
.help = N_("resume failed post-copy migration")
},
+ {.name = "postcopy-preempt",
+ .type = VSH_OT_BOOL,
+ .help = N_("let post-copy requests preempt normal migration stream")
+ },
{.name = "zerocopy",
.type = VSH_OT_BOOL,
.help = N_("use zero-copy mechanism for migrating memory pages")
@@ -11182,6 +11186,7 @@ doMigrate(void *opaque)
{ "abort-on-error", VIR_MIGRATE_ABORT_ON_ERROR },
{ "postcopy", VIR_MIGRATE_POSTCOPY },
{ "postcopy-resume", VIR_MIGRATE_POSTCOPY_RESUME },
+ { "postcopy-preempt", VIR_MIGRATE_POSTCOPY_PREEMPT },
{ "zerocopy", VIR_MIGRATE_ZEROCOPY },
{ "tls", VIR_MIGRATE_TLS },
{ "parallel", VIR_MIGRATE_PARALLEL },
@@ -11530,6 +11535,7 @@ cmdMigrate(vshControl *ctl, const vshCmd *cmd)
VSH_EXCLUSIVE_OPTIONS("copy-storage-all", "copy-storage-inc");
VSH_REQUIRE_OPTION("postcopy-after-precopy", "postcopy");
VSH_REQUIRE_OPTION("postcopy-resume", "postcopy");
+ VSH_REQUIRE_OPTION("postcopy-preempt", "postcopy");
VSH_REQUIRE_OPTION("timeout-postcopy", "postcopy");
VSH_REQUIRE_OPTION("persistent-xml", "persistent");
VSH_REQUIRE_OPTION("tls-destination", "tls");
--
2.43.0
9:53 a.m.
New subject: [libvirt PATCH 3/3] qemu_migration: Implement VIR_MIGRATE_POSTCOPY_PREEMPT
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
src/qemu/qemu_migration.c | 21 ++++++++++++++++++---
src/qemu/qemu_migration.h | 1 +
src/qemu/qemu_migration_params.c | 6 ++++++
src/qemu/qemu_migration_params.h | 1 +
4 files changed, 26 insertions(+), 3 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index 3ba0aa502b..09445354a9 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -2611,6 +2611,13 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver,
return NULL;
}
+ if (flags & VIR_MIGRATE_POSTCOPY_PREEMPT &&
+ !(flags & VIR_MIGRATE_POSTCOPY)) {
+ virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s",
+ _("post-copy preemption can only be enabled for post-copy
migration"));
+ return NULL;
+ }
+
if (flags & VIR_MIGRATE_OFFLINE) {
if (flags & (VIR_MIGRATE_NON_SHARED_DISK |
VIR_MIGRATE_NON_SHARED_INC)) {
@@ -3605,6 +3612,13 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver,
return -1;
}
+ if (flags & VIR_MIGRATE_POSTCOPY_PREEMPT &&
+ !(flags & VIR_MIGRATE_POSTCOPY)) {
+ virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s",
+ _("post-copy preemption can only be enabled for post-copy
migration"));
+ return -1;
+ }
+
if (cfg->migrateTLSForce &&
!(flags & VIR_MIGRATE_TUNNELLED) &&
!(flags & VIR_MIGRATE_TLS)) {
@@ -5210,10 +5224,11 @@ qemuMigrationSrcPerformNative(virQEMUDriver *driver,
spec.dest.socket.path = uribits->path;
} else {
- /* RDMA and multi-fd migration requires QEMU to connect to the destination
- * itself.
+ /* RDMA, multi-fd, and postcopy-preempt migration requires QEMU to
+ * connect to the destination itself.
*/
- if (STREQ(uribits->scheme, "rdma") || (flags &
VIR_MIGRATE_PARALLEL))
+ if (STREQ(uribits->scheme, "rdma") ||
+ flags & (VIR_MIGRATE_PARALLEL | VIR_MIGRATE_POSTCOPY_PREEMPT))
spec.destType = MIGRATION_DEST_HOST;
else
spec.destType = MIGRATION_DEST_CONNECT_HOST;
diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h
index ed62fd4a91..27fb783359 100644
--- a/src/qemu/qemu_migration.h
+++ b/src/qemu/qemu_migration.h
@@ -62,6 +62,7 @@
VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES | \
VIR_MIGRATE_POSTCOPY_RESUME | \
VIR_MIGRATE_ZEROCOPY | \
+ VIR_MIGRATE_POSTCOPY_PREEMPT | \
0)
/* All supported migration parameters and their types. */
diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c
index 79fe6e97c8..35eafd8f0d 100644
--- a/src/qemu/qemu_migration_params.c
+++ b/src/qemu/qemu_migration_params.c
@@ -99,6 +99,7 @@ VIR_ENUM_IMPL(qemuMigrationCapability,
"dirty-bitmaps",
"return-path",
"zero-copy-send",
+ "postcopy-preempt",
);
@@ -194,6 +195,11 @@ static const qemuMigrationParamsFlagMapItem
qemuMigrationParamsFlagMap[] = {
VIR_MIGRATE_ZEROCOPY,
QEMU_MIGRATION_CAP_ZERO_COPY_SEND,
QEMU_MIGRATION_SOURCE},
+
+ {QEMU_MIGRATION_FLAG_REQUIRED,
+ VIR_MIGRATE_POSTCOPY_PREEMPT,
+ QEMU_MIGRATION_CAP_POSTCOPY_PREEMPT,
+ QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION},
};
/* Translation from VIR_MIGRATE_PARAM_* typed parameters to
diff --git a/src/qemu/qemu_migration_params.h b/src/qemu/qemu_migration_params.h
index 5857673227..74f1680e9e 100644
--- a/src/qemu/qemu_migration_params.h
+++ b/src/qemu/qemu_migration_params.h
@@ -40,6 +40,7 @@ typedef enum {
QEMU_MIGRATION_CAP_BLOCK_DIRTY_BITMAPS,
QEMU_MIGRATION_CAP_RETURN_PATH,
QEMU_MIGRATION_CAP_ZERO_COPY_SEND,
+ QEMU_MIGRATION_CAP_POSTCOPY_PREEMPT,
QEMU_MIGRATION_CAP_LAST
} qemuMigrationCapability;
--
2.43.0
10:03 a.m.
On a Friday in 2024, Jiri Denemark wrote:
See 1/3 for details.
https://issues.redhat.com/browse/RHEL-7100
Jiri Denemark (3):
Introduce VIR_MIGRATE_POSTCOPY_PREEMPT flag
virsh migrate: Add --postcopy-preempt option
qemu_migration: Implement VIR_MIGRATE_POSTCOPY_PREEMPT
docs/manpages/virsh.rst | 9 ++++++---
include/libvirt/libvirt-domain.h | 8 ++++++++
src/qemu/qemu_migration.c | 21 ++++++++++++++++++---
src/qemu/qemu_migration.h | 1 +
src/qemu/qemu_migration_params.c | 6 ++++++
src/qemu/qemu_migration_params.h | 1 +
tools/virsh-domain.c | 6 ++++++
7 files changed, 46 insertions(+), 6 deletions(-)
Reviewed-by: Ján Tomko <jtomko(a)redhat.com>
Jano
319
days inactive
322
days old
7 comments
3 participants
participants (3)
-
Jiri Denemark -
Ján Tomko -
Peter Krempa