9:08 a.m.
This series adds a new VIR_MIGRATE_PARALLEL flag for migration APIs
which will ask the hypervisor to use multiple parallel connections for
migrating a domain. The number of parallel connections can be set using
VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS typed parameter.
Version 2:
- rebased
Jiri Denemark (3):
Public API for parallel migration
qemu: Add support for parallel migration
virsh: Add options for parallel migration
include/libvirt/libvirt-domain.h | 14 ++++++++++++++
src/qemu/qemu_migration.c | 9 ++++++---
src/qemu/qemu_migration.h | 5 ++++-
src/qemu/qemu_migration_params.c | 18 ++++++++++++++++++
src/qemu/qemu_migration_params.h | 2 ++
tools/virsh-domain.c | 19 +++++++++++++++++++
tools/virsh.pod | 7 +++++++
7 files changed, 70 insertions(+), 4 deletions(-)
--
2.20.1
9:08 a.m.
New subject: [libvirt] [PATCH v2 1/3] Public API for parallel migration
This patch adds a new VIR_MIGRATE_PARALLEL flag for migration APIs which
will ask the hypervisor to use multiple parallel connections for
migrating a domain. The number of parallel connections can be set using
VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS typed parameter.
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
include/libvirt/libvirt-domain.h | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h
index d82c75a9d9..b2d82d8152 100644
--- a/include/libvirt/libvirt-domain.h
+++ b/include/libvirt/libvirt-domain.h
@@ -831,6 +831,12 @@ typedef enum {
*/
VIR_MIGRATE_TLS = (1 << 16),
+ /* Send memory pages to the destination host through several network
+ * connections. See VIR_MIGRATE_PARAM_PARALLEL_* parameters for
+ * configuring the parallel migration.
+ */
+ VIR_MIGRATE_PARALLEL = (1 << 17),
+
} virDomainMigrateFlags;
@@ -1025,6 +1031,14 @@ typedef enum {
*/
# define VIR_MIGRATE_PARAM_AUTO_CONVERGE_INCREMENT "auto_converge.increment"
+/**
+ * VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS:
+ *
+ * virDomainMigrate* params field: number of connections used during parallel
+ * migration. As VIR_TYPED_PARAM_INT.
+ */
+# define VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS "parallel.connections"
+
/* Domain migration. */
virDomainPtr virDomainMigrate (virDomainPtr domain, virConnectPtr dconn,
unsigned long flags, const char *dname,
--
2.20.1
2:03 p.m.
New subject: [libvirt] [PATCH v2 1/3] Public API for parallel migration
On 2/8/19 10:08 AM, Jiri Denemark wrote:
This patch adds a new VIR_MIGRATE_PARALLEL flag for migration APIs
which
will ask the hypervisor to use multiple parallel connections for
migrating a domain. The number of parallel connections can be set using
VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS typed parameter.
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
include/libvirt/libvirt-domain.h | 14 ++++++++++++++
1 file changed, 14 insertions(+)
Shall I assume the multifd-page-count is not going to be implemented? or
something for the future?
Reviewed-by: John Ferlan <jferlan(a)redhat.com>
John
3:35 a.m.
New subject: [libvirt] [PATCH v2 1/3] Public API for parallel migration
On Wed, Feb 13, 2019 at 15:03:44 -0500, John Ferlan wrote:
On 2/8/19 10:08 AM, Jiri Denemark wrote:
> This patch adds a new VIR_MIGRATE_PARALLEL flag for migration APIs which
> will ask the hypervisor to use multiple parallel connections for
> migrating a domain. The number of parallel connections can be set using
> VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS typed parameter.
>
> Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
> ---
> include/libvirt/libvirt-domain.h | 14 ++++++++++++++
> 1 file changed, 14 insertions(+)
>
Shall I assume the multifd-page-count is not going to be implemented?
Right, this parameter was dropped in QEMU.
Jirka
9:08 a.m.
New subject: [libvirt] [PATCH v2 2/3] qemu: Add support for parallel migration
The VIR_MIGRATE_PARALLEL flag is implemented using QEMU's multifd
migration capability and the corresponding multifd-channels migration
parameter.
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
Notes:
QEMU still uses the x- prefix for multifd capability and multifd-channels,
but Juan already sent a series of patches to make multifd migration fully
supported by dropping the experimental prefix.
src/qemu/qemu_migration.c | 9 ++++++---
src/qemu/qemu_migration.h | 5 ++++-
src/qemu/qemu_migration_params.c | 18 ++++++++++++++++++
src/qemu/qemu_migration_params.h | 2 ++
4 files changed, 30 insertions(+), 4 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c
index 3107a279dd..22f31680c9 100644
--- a/src/qemu/qemu_migration.c
+++ b/src/qemu/qemu_migration.c
@@ -3822,10 +3822,13 @@ qemuMigrationSrcPerformNative(virQEMUDriverPtr driver,
}
}
- if (STRNEQ(uribits->scheme, "rdma"))
- spec.destType = MIGRATION_DEST_CONNECT_HOST;
- else
+ /* RDMA and multi-fd migration requires QEMU to connect to the destination
+ * itself.
+ */
+ if (STREQ(uribits->scheme, "rdma") || (flags &
VIR_MIGRATE_PARALLEL))
spec.destType = MIGRATION_DEST_HOST;
+ else
+ spec.destType = MIGRATION_DEST_CONNECT_HOST;
spec.dest.host.protocol = uribits->scheme;
spec.dest.host.name = uribits->server;
spec.dest.host.port = uribits->port;
diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h
index ca73d3e467..fea5000a4d 100644
--- a/src/qemu/qemu_migration.h
+++ b/src/qemu/qemu_migration.h
@@ -57,7 +57,9 @@
VIR_MIGRATE_AUTO_CONVERGE | \
VIR_MIGRATE_RDMA_PIN_ALL | \
VIR_MIGRATE_POSTCOPY | \
- VIR_MIGRATE_TLS)
+ VIR_MIGRATE_TLS | \
+ VIR_MIGRATE_PARALLEL | \
+ 0)
/* All supported migration parameters and their types. */
# define QEMU_MIGRATION_PARAMETERS \
@@ -80,6 +82,7 @@
VIR_MIGRATE_PARAM_AUTO_CONVERGE_INITIAL, VIR_TYPED_PARAM_INT, \
VIR_MIGRATE_PARAM_AUTO_CONVERGE_INCREMENT, VIR_TYPED_PARAM_INT, \
VIR_MIGRATE_PARAM_BANDWIDTH_POSTCOPY, VIR_TYPED_PARAM_ULLONG, \
+ VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS, VIR_TYPED_PARAM_INT, \
NULL
diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c
index 28f2c75ea1..82416667a6 100644
--- a/src/qemu/qemu_migration_params.c
+++ b/src/qemu/qemu_migration_params.c
@@ -86,6 +86,7 @@ VIR_ENUM_IMPL(qemuMigrationCapability, QEMU_MIGRATION_CAP_LAST,
"compress",
"pause-before-switchover",
"late-block-activate",
+ "multifd",
);
@@ -103,6 +104,7 @@ VIR_ENUM_IMPL(qemuMigrationParam, QEMU_MIGRATION_PARAM_LAST,
"block-incremental",
"xbzrle-cache-size",
"max-postcopy-bandwidth",
+ "multifd-channels",
);
typedef struct _qemuMigrationParamsAlwaysOnItem qemuMigrationParamsAlwaysOnItem;
@@ -151,6 +153,10 @@ static const qemuMigrationParamsFlagMapItem
qemuMigrationParamsFlagMap[] = {
{VIR_MIGRATE_POSTCOPY,
QEMU_MIGRATION_CAP_POSTCOPY,
QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION},
+
+ {VIR_MIGRATE_PARALLEL,
+ QEMU_MIGRATION_CAP_MULTIFD,
+ QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION},
};
/* Translation from VIR_MIGRATE_PARAM_* typed parameters to
@@ -184,6 +190,10 @@ static const qemuMigrationParamsTPMapItem qemuMigrationParamsTPMap[]
= {
.unit = 1024 * 1024, /* MiB/s */
.param = QEMU_MIGRATION_PARAM_MAX_POSTCOPY_BANDWIDTH,
.party = QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION},
+
+ {.typedParam = VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS,
+ .param = QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS,
+ .party = QEMU_MIGRATION_SOURCE | QEMU_MIGRATION_DESTINATION},
};
static const qemuMigrationParamType qemuMigrationParamTypes[] = {
@@ -199,6 +209,7 @@ static const qemuMigrationParamType qemuMigrationParamTypes[] = {
[QEMU_MIGRATION_PARAM_BLOCK_INCREMENTAL] = QEMU_MIGRATION_PARAM_TYPE_BOOL,
[QEMU_MIGRATION_PARAM_XBZRLE_CACHE_SIZE] = QEMU_MIGRATION_PARAM_TYPE_ULL,
[QEMU_MIGRATION_PARAM_MAX_POSTCOPY_BANDWIDTH] = QEMU_MIGRATION_PARAM_TYPE_ULL,
+ [QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS] = QEMU_MIGRATION_PARAM_TYPE_INT,
};
verify(ARRAY_CARDINALITY(qemuMigrationParamTypes) == QEMU_MIGRATION_PARAM_LAST);
@@ -535,6 +546,13 @@ qemuMigrationParamsFromFlags(virTypedParameterPtr params,
goto error;
}
+ if (migParams->params[QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS].set &&
+ !(flags & VIR_MIGRATE_PARALLEL)) {
+ virReportError(VIR_ERR_INVALID_ARG, "%s",
+ _("Turn parallel migration on to tune it"));
+ goto error;
+ }
+
if (qemuMigrationParamsSetCompression(params, nparams, flags, migParams) < 0)
goto error;
diff --git a/src/qemu/qemu_migration_params.h b/src/qemu/qemu_migration_params.h
index 233ebe772b..0515339d91 100644
--- a/src/qemu/qemu_migration_params.h
+++ b/src/qemu/qemu_migration_params.h
@@ -38,6 +38,7 @@ typedef enum {
QEMU_MIGRATION_CAP_COMPRESS,
QEMU_MIGRATION_CAP_PAUSE_BEFORE_SWITCHOVER,
QEMU_MIGRATION_CAP_LATE_BLOCK_ACTIVATE,
+ QEMU_MIGRATION_CAP_MULTIFD,
QEMU_MIGRATION_CAP_LAST
} qemuMigrationCapability;
@@ -56,6 +57,7 @@ typedef enum {
QEMU_MIGRATION_PARAM_BLOCK_INCREMENTAL,
QEMU_MIGRATION_PARAM_XBZRLE_CACHE_SIZE,
QEMU_MIGRATION_PARAM_MAX_POSTCOPY_BANDWIDTH,
+ QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS,
QEMU_MIGRATION_PARAM_LAST
} qemuMigrationParam;
--
2.20.1
2:03 p.m.
New subject: [libvirt] [PATCH v2 2/3] qemu: Add support for parallel migration
On 2/8/19 10:08 AM, Jiri Denemark wrote:
The VIR_MIGRATE_PARALLEL flag is implemented using QEMU's
multifd
migration capability and the corresponding multifd-channels migration
parameter.
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
Notes:
QEMU still uses the x- prefix for multifd capability and multifd-channels,
but Juan already sent a series of patches to make multifd migration fully
supported by dropping the experimental prefix.
src/qemu/qemu_migration.c | 9 ++++++---
src/qemu/qemu_migration.h | 5 ++++-
src/qemu/qemu_migration_params.c | 18 ++++++++++++++++++
src/qemu/qemu_migration_params.h | 2 ++
4 files changed, 30 insertions(+), 4 deletions(-)
Standard practice is to wait for the x-* to hit the qemu.git tree before
we push this...
Reviewed-by: John Ferlan <jferlan(a)redhat.com>
John
3:42 a.m.
New subject: [libvirt] [PATCH v2 2/3] qemu: Add support for parallel migration
On Wed, Feb 13, 2019 at 15:03:58 -0500, John Ferlan wrote:
On 2/8/19 10:08 AM, Jiri Denemark wrote:
> The VIR_MIGRATE_PARALLEL flag is implemented using QEMU's multifd
> migration capability and the corresponding multifd-channels migration
> parameter.
>
> Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
> ---
>
> Notes:
> QEMU still uses the x- prefix for multifd capability and multifd-channels,
> but Juan already sent a series of patches to make multifd migration fully
> supported by dropping the experimental prefix.
>
> src/qemu/qemu_migration.c | 9 ++++++---
> src/qemu/qemu_migration.h | 5 ++++-
> src/qemu/qemu_migration_params.c | 18 ++++++++++++++++++
> src/qemu/qemu_migration_params.h | 2 ++
> 4 files changed, 30 insertions(+), 4 deletions(-)
>
Standard practice is to wait for the x-* to hit the qemu.git tree before
we push this...
Of course. The QEMU patches were finally applied yesterday, which is
just in time for this series to go in the upcoming 5.2.
Jirka
9:08 a.m.
New subject: [libvirt] [PATCH v2 3/3] virsh: Add options for parallel migration
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
tools/virsh-domain.c | 19 +++++++++++++++++++
tools/virsh.pod | 7 +++++++
2 files changed, 26 insertions(+)
diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index 8b20059335..c704faf7e1 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -10561,6 +10561,14 @@ static const vshCmdOptDef opts_migrate[] = {
.type = VSH_OT_INT,
.help = N_("post-copy migration bandwidth limit in MiB/s")
},
+ {.name = "parallel",
+ .type = VSH_OT_BOOL,
+ .help = N_("enable parallel migration")
+ },
+ {.name = "parallel-connections",
+ .type = VSH_OT_INT,
+ .help = N_("number of connections for parallel migration")
+ },
{.name = NULL}
};
@@ -10766,6 +10774,14 @@ doMigrate(void *opaque)
goto save_error;
}
+ if (vshCommandOptInt(ctl, cmd, "parallel-connections", &intOpt) <
0)
+ goto out;
+ if (intOpt &&
+ virTypedParamsAddInt(¶ms, &nparams, &maxparams,
+ VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS,
+ intOpt) < 0)
+ goto save_error;
+
if (vshCommandOptBool(cmd, "live"))
flags |= VIR_MIGRATE_LIVE;
if (vshCommandOptBool(cmd, "p2p"))
@@ -10814,6 +10830,9 @@ doMigrate(void *opaque)
if (vshCommandOptBool(cmd, "tls"))
flags |= VIR_MIGRATE_TLS;
+ if (vshCommandOptBool(cmd, "parallel"))
+ flags |= VIR_MIGRATE_PARALLEL;
+
if (flags & VIR_MIGRATE_PEER2PEER || vshCommandOptBool(cmd, "direct"))
{
if (virDomainMigrateToURI3(dom, desturi, params, nparams, flags) == 0)
ret = '0';
diff --git a/tools/virsh.pod b/tools/virsh.pod
index 67edb57b14..7604c7536e 100644
--- a/tools/virsh.pod
+++ b/tools/virsh.pod
@@ -1904,6 +1904,7 @@ I<domain> I<desturi> [I<migrateuri>]
[I<graphicsuri>] [I<listen-address>] [I<dna
[I<--comp-xbzrle-cache>] [I<--auto-converge>]
[I<auto-converge-initial>]
[I<auto-converge-increment>] [I<--persistent-xml> B<file>]
[I<--tls>]
[I<--postcopy-bandwidth> B<bandwidth>]
+[I<--parallel> [I<parallel-connections> B<connections>]]
Migrate domain to another host. Add I<--live> for live migration; <--p2p>
for peer-2-peer migration; I<--direct> for direct migration; or
I<--tunnelled>
@@ -1997,6 +1998,12 @@ Providing I<--tls> causes the migration to use the host
configured TLS setup
the migration of the domain. Usage requires proper TLS setup for both source
and target.
+I<--parallel> option will cause migration data to be sent over multiple
+parallel connections. The number of such connections can be set using
+I<parallel-connections>. Parallel connections may help with saturating the
+network link between the source and the target and thus speeding up the
+migration.
+
Running migration can be canceled by interrupting virsh (usually using
C<Ctrl-C>) or by B<domjobabort> command sent from another virsh instance.
--
2.20.1
2:05 p.m.
New subject: [libvirt] [PATCH v2 3/3] virsh: Add options for parallel migration
On 2/8/19 10:08 AM, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com>
---
tools/virsh-domain.c | 19 +++++++++++++++++++
tools/virsh.pod | 7 +++++++
2 files changed, 26 insertions(+)
Should supplying parallel-connections imply parallel since the previous
patch would fail:
+ if (migParams->params[QEMU_MIGRATION_PARAM_MULTIFD_CHANNELS].set &&
+ !(flags & VIR_MIGRATE_PARALLEL)) {
+ virReportError(VIR_ERR_INVALID_ARG, "%s",
+ _("Turn parallel migration on to tune it"));
+ goto error;
+ }
+
either that or make parallel-connections be an optional parameter to
--parallel using VSH_OFLAG_EMPTY_OK and converting the optionally read
string into an int.
diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c
index 8b20059335..c704faf7e1 100644
--- a/tools/virsh-domain.c
+++ b/tools/virsh-domain.c
@@ -10561,6 +10561,14 @@ static const vshCmdOptDef opts_migrate[] = {
.type = VSH_OT_INT,
.help = N_("post-copy migration bandwidth limit in MiB/s")
},
+ {.name = "parallel",
+ .type = VSH_OT_BOOL,
+ .help = N_("enable parallel migration")
+ },
+ {.name = "parallel-connections",
+ .type = VSH_OT_INT,
+ .help = N_("number of connections for parallel migration")
+ },
{.name = NULL}
};
@@ -10766,6 +10774,14 @@ doMigrate(void *opaque)
goto save_error;
}
+ if (vshCommandOptInt(ctl, cmd, "parallel-connections", &intOpt) <
0)
+ goto out;
+ if (intOpt &&
+ virTypedParamsAddInt(¶ms, &nparams, &maxparams,
+ VIR_MIGRATE_PARAM_PARALLEL_CONNECTIONS,
+ intOpt) < 0)
+ goto save_error;
+
if (vshCommandOptBool(cmd, "live"))
flags |= VIR_MIGRATE_LIVE;
if (vshCommandOptBool(cmd, "p2p"))
@@ -10814,6 +10830,9 @@ doMigrate(void *opaque)
if (vshCommandOptBool(cmd, "tls"))
flags |= VIR_MIGRATE_TLS;
+ if (vshCommandOptBool(cmd, "parallel"))
+ flags |= VIR_MIGRATE_PARALLEL;
+
if (flags & VIR_MIGRATE_PEER2PEER || vshCommandOptBool(cmd, "direct"))
{
if (virDomainMigrateToURI3(dom, desturi, params, nparams, flags) == 0)
ret = '0';
diff --git a/tools/virsh.pod b/tools/virsh.pod
index 67edb57b14..7604c7536e 100644
--- a/tools/virsh.pod
+++ b/tools/virsh.pod
@@ -1904,6 +1904,7 @@ I<domain> I<desturi> [I<migrateuri>]
[I<graphicsuri>] [I<listen-address>] [I<dna
[I<--comp-xbzrle-cache>] [I<--auto-converge>]
[I<auto-converge-initial>]
[I<auto-converge-increment>] [I<--persistent-xml> B<file>]
[I<--tls>]
[I<--postcopy-bandwidth> B<bandwidth>]
+[I<--parallel> [I<parallel-connections> B<connections>]]
--parallel-connections
Migrate domain to another host. Add I<--live> for live migration; <--p2p>
for peer-2-peer migration; I<--direct> for direct migration; or
I<--tunnelled>
@@ -1997,6 +1998,12 @@ Providing I<--tls> causes the migration to use the host
configured TLS setup
the migration of the domain. Usage requires proper TLS setup for both source
and target.
+I<--parallel> option will cause migration data to be sent over multiple
+parallel connections. The number of such connections can be set using
+I<parallel-connections>. Parallel connections may help with saturating the
--parallel-connections
+network link between the source and the target and thus speeding up
the
+migration.
+
[although too many could really be a problem ;-)]
What's here is fine.... If you want to alter to have one option with an
optional parameter, then just repost this patch.
Reviewed-by: John Ferlan <jferlan(a)redhat.com>
John
Running migration can be canceled by interrupting virsh (usually
using
C<Ctrl-C>) or by B<domjobabort> command sent from another virsh instance.
2059
days inactive
2106
days old
8 comments
2 participants
participants (2)
-
Jiri Denemark -
John Ferlan