[PATCH v2 0/2] Adding support for client-based FD migrations.
Libvirt supports FD-based live migrations in the form of tunnelled migrations. However, this isn't as performant as of now, since it involves extra copies.· This patchset proposes adding support for externally provided file descriptors for live migration. By allowing the client to supply the migration FD directly, the extra copy can be avoided, improving overall migration performance. The intent is to gather early feedback on the approach and its feasibility. Comments, concerns, and suggestions are very welcome :) V2: 1. Removed the approach of introducing a new FD passing API and instead reuse the virDomainFDAssociate API along with a dummy VM on the destination host. 2. Introduced a new virDomainMigrate parameter, VIR_MIGRATE_PARAM_FD_SET. Tejus GK (2): qemu: support incoming "fd" uri qemu: add support for fd based live migrations include/libvirt/libvirt-domain.h | 14 +++ src/qemu/qemu_driver.c | 24 +++-- src/qemu/qemu_migration.c | 174 +++++++++++++++++++++++++------ src/qemu/qemu_migration.h | 3 + 4 files changed, 178 insertions(+), 37 deletions(-) -- 2.43.7
If "fd" is the chosen protocol for the live migration, the destination VM should start up with -incoming as "fd", this patch adds support for the same. Signed-off-by: Tejus GK <tejus.gk@nutanix.com> --- src/qemu/qemu_migration.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 6dd022163b..e6c6be38c0 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2427,7 +2427,8 @@ qemuMigrationDstGetURI(const char *migrateFrom, { char *uri = NULL; - if (STREQ(migrateFrom, "stdio")) + if (STREQ(migrateFrom, "stdio") || + STREQ(migrateFrom, "fd")) uri = g_strdup_printf("fd:%d", migrateFd); else uri = g_strdup(migrateFrom); @@ -3136,6 +3137,8 @@ qemuMigrationDstPrepare(virQEMUDriver *driver, if (tunnel) { migrateFrom = g_strdup("stdio"); + } else if (g_strcmp0(protocol, "fd") == 0) { + migrateFrom = g_strdup("fd"); } else if (g_strcmp0(protocol, "unix") == 0) { migrateFrom = g_strdup_printf("%s:%s", protocol, listenAddress); } else { -- 2.43.7
QEMU already supports migration using a file descriptor passed directly to QEMU. Currently, libvirt uses this mechanism via the tunnelled migration path, where libvirt creates a set of file descriptors using pipes, passes them to QEMU, and performs the migration. This patch introduces support for FD-based live migration in libvirt, where the file descriptors are opened by the client and provided to libvirt. Clients are expected to first define a dummy domain on the destination host with the same UUID as the source domain, and then associate the file descriptors with libvirt on both the source and destination using the virDomainFDAssociate API. Since the client may assign an arbitrary name to the file descriptor used for migration, this patch introduces a new virDomainMigrate parameter, VIR_MIGRATE_PARAM_FD_SET, which instructs libvirt which FD name to look up and pass through to QEMU for the migration. Signed-off-by: Tejus GK <tejus.gk@nutanix.com> --- include/libvirt/libvirt-domain.h | 14 +++ src/qemu/qemu_driver.c | 24 +++-- src/qemu/qemu_migration.c | 169 ++++++++++++++++++++++++++----- src/qemu/qemu_migration.h | 3 + 4 files changed, 174 insertions(+), 36 deletions(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index b0acd0083b..6d2a2a0c35 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1438,6 +1438,20 @@ typedef enum { */ # define VIR_MIGRATE_PARAM_TLS_DESTINATION "tls.destination" +/** + * VIR_MIGRATE_PARAM_FD_SET: + * + * virDomainMigrate* params field: the name of the file descriptor set + * previously associated with the domain via virDomainFDAssociate() to be used + * for FD-based migration. As VIR_TYPED_PARAM_STRING. + * + * When using "fd://" as the migration URI, this parameter specifies which + * FD set should be used for the migration data stream. + * + * Since: 12.0.0 + */ +# define VIR_MIGRATE_PARAM_FD_SET "fd.set" + /* Domain migration. */ virDomainPtr virDomainMigrate (virDomainPtr domain, virConnectPtr dconn, unsigned long flags, const char *dname, diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index cdd333c882..076b3e8a74 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -10851,7 +10851,7 @@ qemuDomainMigratePrepare2(virConnectPtr dconn, NULL, 0, NULL, NULL, /* No cookies */ uri_in, uri_out, &def, origname, NULL, NULL, 0, NULL, - migParams, flags); + NULL, migParams, flags); } @@ -10901,7 +10901,7 @@ qemuDomainMigratePerform(virDomainPtr dom, */ ret = qemuMigrationSrcPerform(driver, dom->conn, vm, NULL, NULL, dconnuri, uri, NULL, NULL, NULL, NULL, 0, - NULL, + NULL, NULL, migParams, cookie, cookielen, NULL, NULL, /* No output cookies in v2 */ flags, dname, bandwidth, false); @@ -11070,7 +11070,7 @@ qemuDomainMigratePrepare3(virConnectPtr dconn, cookieout, cookieoutlen, uri_in, uri_out, &def, origname, NULL, NULL, 0, - NULL, migParams, flags); + NULL, NULL, migParams, flags); } static int @@ -11091,6 +11091,7 @@ qemuDomainMigratePrepare3Params(virConnectPtr dconn, const char *dname = NULL; const char *uri_in = NULL; const char *listenAddress = NULL; + const char *fdset = NULL; int nbdPort = 0; g_autofree const char **migrate_disks = NULL; g_autofree char *origname = NULL; @@ -11118,7 +11119,10 @@ qemuDomainMigratePrepare3Params(virConnectPtr dconn, &nbdURI) < 0 || virTypedParamsGetInt(params, nparams, VIR_MIGRATE_PARAM_DISKS_PORT, - &nbdPort) < 0) + &nbdPort) < 0 || + virTypedParamsGetString(params, nparams, + VIR_MIGRATE_PARAM_FD_SET, + &fdset) < 0) return -1; virTypedParamsGetStringList(params, nparams, VIR_MIGRATE_PARAM_MIGRATE_DISKS, @@ -11173,7 +11177,7 @@ qemuDomainMigratePrepare3Params(virConnectPtr dconn, uri_in, uri_out, &def, origname, listenAddress, migrate_disks, nbdPort, - nbdURI, migParams, flags); + nbdURI, fdset, migParams, flags); } @@ -11301,7 +11305,7 @@ qemuDomainMigratePerform3(virDomainPtr dom, ret = qemuMigrationSrcPerform(driver, dom->conn, vm, xmlin, NULL, dconnuri, uri, NULL, NULL, NULL, NULL, 0, - NULL, migParams, + NULL, NULL, migParams, cookiein, cookieinlen, cookieout, cookieoutlen, flags, dname, bandwidth, true); @@ -11336,6 +11340,7 @@ qemuDomainMigratePerform3Params(virDomainPtr dom, int nbdPort = 0; g_autoptr(qemuMigrationParams) migParams = NULL; const char *nbdURI = NULL; + const char *fdset = NULL; int ret = -1; virCheckFlags(QEMU_MIGRATION_FLAGS, -1); @@ -11368,7 +11373,10 @@ qemuDomainMigratePerform3Params(virDomainPtr dom, &nbdURI) < 0 || virTypedParamsGetString(params, nparams, VIR_MIGRATE_PARAM_PERSIST_XML, - &persist_xml) < 0) + &persist_xml) < 0 || + virTypedParamsGetString(params, nparams, + VIR_MIGRATE_PARAM_FD_SET, + &fdset) < 0) goto cleanup; @@ -11414,7 +11422,7 @@ qemuDomainMigratePerform3Params(virDomainPtr dom, ret = qemuMigrationSrcPerform(driver, dom->conn, vm, dom_xml, persist_xml, dconnuri, uri, graphicsuri, listenAddress, migrate_disks, migrate_disks_detect_zeroes, nbdPort, - nbdURI, migParams, + nbdURI, fdset, migParams, cookiein, cookieinlen, cookieout, cookieoutlen, flags, dname, bandwidth, true); cleanup: diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index e6c6be38c0..2c63242acd 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3289,6 +3289,7 @@ qemuMigrationDstPrepareActive(virQEMUDriver *driver, const char **migrate_disks, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, unsigned int flags) { @@ -3303,6 +3304,7 @@ qemuMigrationDstPrepareActive(virQEMUDriver *driver, unsigned int startFlags; bool relabel = false; bool tunnel = !!st; + int migrationFD = -1; int ret = -1; int rv; @@ -3321,6 +3323,31 @@ qemuMigrationDstPrepareActive(virQEMUDriver *driver, virPipe(dataFD) < 0) goto error; + if (fdset) { + virStorageSourceFDTuple *fdtuple = NULL; + + if (!(fdtuple = virHashLookup(priv->fds, fdset))) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("no file descriptor associated with name '%1$s' for migration of domain '%2$s'"), + fdset, vm->def->name); + goto error; + } + + if (fdtuple->nfds != 1) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("expected a single file descriptor for migration of domain '%1$s'"), + vm->def->name); + goto error; + } + + if ((migrationFD = dup(fdtuple->fds[0])) < 0) { + virReportSystemError(errno, + _("failed to duplicate migration FD for domain '%1$s'"), + vm->def->name); + goto error; + } + } + startFlags = VIR_QEMU_PROCESS_START_AUTODESTROY; if (qemuProcessInit(driver, vm, mig->cpu, VIR_ASYNC_JOB_MIGRATION_IN, @@ -3330,7 +3357,7 @@ qemuMigrationDstPrepareActive(virQEMUDriver *driver, if (!(incoming = qemuMigrationDstPrepare(driver, vm, tunnel, protocol, listenAddress, port, - &dataFD[0]))) + migrationFD ? &migrationFD : &dataFD[0]))) goto error; qemuMigrationDstPrepareDiskSeclabels(vm, migrate_disks, flags); @@ -3439,6 +3466,7 @@ qemuMigrationDstPrepareActive(virQEMUDriver *driver, cleanup: qemuProcessIncomingDefFree(incoming); + VIR_FORCE_CLOSE(migrationFD); VIR_FORCE_CLOSE(dataFD[0]); VIR_FORCE_CLOSE(dataFD[1]); virObjectEventStateQueue(driver->domainEventState, event); @@ -3481,6 +3509,7 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, const char **migrate_disks, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, unsigned int flags) { @@ -3494,9 +3523,9 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, bool taint_hook = false; VIR_DEBUG("name=%s, origname=%s, protocol=%s, port=%hu, " - "listenAddress=%s, nbdPort=%d, nbdURI=%s, flags=0x%x", + "listenAddress=%s, nbdPort=%d, nbdURI=%s, fdset=%s, flags=0x%x", (*def)->name, NULLSTR(origname), protocol, port, - listenAddress, nbdPort, NULLSTR(nbdURI), flags); + listenAddress, nbdPort, NULLSTR(nbdURI), NULLSTR(fdset), flags); if (!(flags & VIR_MIGRATE_OFFLINE)) { cookieFlags = QEMU_MIGRATION_COOKIE_GRAPHICS | @@ -3588,7 +3617,7 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, if (qemuMigrationDstPrepareActive(driver, vm, dconn, mig, st, protocol, port, listenAddress, migrate_disks, - nbdPort, nbdURI, + nbdPort, nbdURI, fdset, migParams, flags) < 0) { goto stopjob; } @@ -3754,6 +3783,7 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, const char **migrate_disks, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, unsigned int flags) { @@ -3814,7 +3844,7 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, def, origname, st, protocol, port, autoPort, listenAddress, migrate_disks, - nbdPort, nbdURI, + nbdPort, nbdURI, fdset, migParams, flags); } @@ -3851,7 +3881,7 @@ qemuMigrationDstPrepareTunnel(virQEMUDriver *driver, return qemuMigrationDstPrepareAny(driver, dconn, cookiein, cookieinlen, cookieout, cookieoutlen, def, origname, st, NULL, 0, false, NULL, NULL, 0, - NULL, migParams, flags); + NULL, NULL, migParams, flags); } @@ -3892,6 +3922,7 @@ qemuMigrationDstPrepareDirect(virQEMUDriver *driver, const char **migrate_disks, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, unsigned int flags) { @@ -3907,11 +3938,12 @@ qemuMigrationDstPrepareDirect(virQEMUDriver *driver, "cookieout=%p, cookieoutlen=%p, uri_in=%s, uri_out=%p, " "def=%p, origname=%s, listenAddress=%s, " "migrate_disks=%p, nbdPort=%d, " - "nbdURI=%s, flags=0x%x", + "nbdURI=%s, fdset=%s, flags=0x%x", driver, dconn, NULLSTR(cookiein), cookieinlen, cookieout, cookieoutlen, NULLSTR(uri_in), uri_out, *def, origname, NULLSTR(listenAddress), migrate_disks, nbdPort, NULLSTR(nbdURI), + NULLSTR(fdset), flags); *uri_out = NULL; @@ -3976,7 +4008,8 @@ qemuMigrationDstPrepareDirect(virQEMUDriver *driver, if (STRNEQ(uri->scheme, "tcp") && STRNEQ(uri->scheme, "rdma") && - STRNEQ(uri->scheme, "unix")) { + STRNEQ(uri->scheme, "unix") && + STRNEQ(uri->scheme, "fd")) { virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, _("unsupported scheme %1$s in migration URI %2$s"), uri->scheme, uri_in); @@ -3986,6 +4019,9 @@ qemuMigrationDstPrepareDirect(virQEMUDriver *driver, if (STREQ(uri->scheme, "unix")) { autoPort = false; listenAddress = uri->path; + } else if (STREQ(uri->scheme, "fd")) { + autoPort = false; + listenAddress = NULL; } else { if (uri->server == NULL) { virReportError(VIR_ERR_INVALID_ARG, @@ -4020,7 +4056,7 @@ qemuMigrationDstPrepareDirect(virQEMUDriver *driver, NULL, uri ? uri->scheme : "tcp", port, autoPort, listenAddress, migrate_disks, nbdPort, - nbdURI, migParams, flags); + nbdURI, fdset, migParams, flags); cleanup: if (ret != 0) { VIR_FREE(*uri_out); @@ -5364,19 +5400,22 @@ qemuMigrationSrcPerformNative(virQEMUDriver *driver, const char *graphicsuri, const char **migrate_disks, const char **migrate_disks_detect_zeroes, + const char *fdset, qemuMigrationParams *migParams, const char *nbdURI) { + qemuDomainObjPrivate *priv = vm->privateData; g_autoptr(virURI) uribits = NULL; int ret = -1; qemuMigrationSpec spec; VIR_DEBUG("driver=%p, vm=%p, uri=%s, cookiein=%s, cookieinlen=%d, " "cookieout=%p, cookieoutlen=%p, flags=0x%x, bandwidth=%lu, " - "graphicsuri=%s, migrate_disks=%p, migrate_disks_detect_zeroes=%p", + "graphicsuri=%s, migrate_disks=%p, migrate_disks_detect_zeroes=%p, fdset=%s", driver, vm, uri, NULLSTR(cookiein), cookieinlen, cookieout, cookieoutlen, flags, bandwidth, - NULLSTR(graphicsuri), migrate_disks, migrate_disks_detect_zeroes); + NULLSTR(graphicsuri), migrate_disks, migrate_disks_detect_zeroes, + NULLSTR(fdset)); if (!(uribits = qemuMigrationAnyParseURI(uri, NULL))) return -1; @@ -5396,7 +5435,41 @@ qemuMigrationSrcPerformNative(virQEMUDriver *driver, } } - if (STREQ(uribits->scheme, "unix")) { + if (STREQ(uribits->scheme, "fd")) { + virStorageSourceFDTuple *fdtuple = NULL; + int migrationFD = -1; + + if (!(fdtuple = virHashLookup(priv->fds, fdset))) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("no file descriptor associated with name '%1$s' for migration of domain '%2$s'"), + fdset, vm->def->name); + return -1; + } + + if (fdtuple->nfds != 1) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("expected a single file descriptor for migration of domain '%1$s'"), + vm->def->name); + return -1; + } + + if ((migrationFD = dup(fdtuple->fds[0])) < 0) { + virReportSystemError(errno, + _("failed to duplicate migration FD for domain '%1$s'"), + vm->def->name); + return -1; + } + + if (qemuSecuritySetImageFDLabel(driver->securityManager, vm->def, migrationFD) < 0) { + VIR_FORCE_CLOSE(migrationFD); + return -1; + } + + spec.destType = MIGRATION_DEST_FD; + spec.dest.fd.qemu = migrationFD; + spec.dest.fd.local = -1; + + } else if (STREQ(uribits->scheme, "unix")) { if ((flags & VIR_MIGRATE_TLS) && !(flags & VIR_MIGRATE_POSTCOPY_RESUME) && !qemuMigrationParamsTLSHostnameIsSet(migParams)) { @@ -5541,7 +5614,7 @@ qemuMigrationSrcPerformResume(virQEMUDriver *driver, ret = qemuMigrationSrcPerformNative(driver, vm, NULL, uri, cookiein, cookieinlen, cookieout, cookieoutlen, flags, - 0, NULL, NULL, NULL, NULL, migParams, NULL); + 0, NULL, NULL, NULL, NULL, NULL, migParams, NULL); virCloseCallbacksDomainAdd(vm, conn, qemuMigrationAnyConnectionClosed); @@ -5649,7 +5722,7 @@ qemuMigrationSrcPerformPeer2Peer2(virQEMUDriver *driver, cookie, cookielen, NULL, NULL, /* No out cookie with v2 migration */ flags, bandwidth, dconn, NULL, NULL, - NULL, migParams, NULL); + NULL, NULL, migParams, NULL); /* Perform failed. Make sure Finish doesn't overwrite the error */ if (ret < 0) @@ -5715,6 +5788,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, const char **migrate_disks_detect_zeroes, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, unsigned long long bandwidth, bool useParams, @@ -5740,11 +5814,11 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, VIR_DEBUG("driver=%p, sconn=%p, dconn=%p, dconnuri=%s, vm=%p, xmlin=%s, " "dname=%s, uri=%s, graphicsuri=%s, listenAddress=%s, " "migrate_disks=%p, migrate_disks_detect_zeroes=%p, nbdPort=%d, nbdURI=%s, " - "bandwidth=%llu, useParams=%d, flags=0x%x", + "fdset=%s, bandwidth=%llu, useParams=%d, flags=0x%x", driver, sconn, dconn, NULLSTR(dconnuri), vm, NULLSTR(xmlin), NULLSTR(dname), NULLSTR(uri), NULLSTR(graphicsuri), NULLSTR(listenAddress), migrate_disks, migrate_disks_detect_zeroes, nbdPort, - NULLSTR(nbdURI), bandwidth, useParams, flags); + NULLSTR(nbdURI), NULLSTR(fdset), bandwidth, useParams, flags); /* Unlike the virDomainMigrateVersion3 counterpart, we don't need * to worry about auto-setting the VIR_MIGRATE_CHANGE_PROTECTION @@ -5827,6 +5901,11 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, if (qemuMigrationParamsDump(migParams, ¶ms, &nparams, &maxparams, &flags) < 0) goto cleanup; + if (fdset && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_FD_SET, + fdset) < 0) + goto cleanup; } if (!(flags & VIR_MIGRATE_POSTCOPY_RESUME) && @@ -5927,7 +6006,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, &cookieout, &cookieoutlen, flags, bandwidth, dconn, graphicsuri, migrate_disks, migrate_disks_detect_zeroes, - migParams, nbdURI); + fdset, migParams, nbdURI); } if (ret == 0) @@ -6103,6 +6182,7 @@ qemuMigrationSrcPerformPeer2Peer(virQEMUDriver *driver, const char **migrate_disks_detect_zeroes, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, unsigned int flags, const char *dname, @@ -6121,11 +6201,11 @@ qemuMigrationSrcPerformPeer2Peer(virQEMUDriver *driver, VIR_DEBUG("driver=%p, sconn=%p, vm=%p, xmlin=%s, dconnuri=%s, uri=%s, " "graphicsuri=%s, listenAddress=%s, " - "migrate_disks=%p, nbdPort=%d, nbdURI=%s, flags=0x%x, " + "migrate_disks=%p, nbdPort=%d, nbdURI=%s, fdset=%s, flags=0x%x, " "dname=%s, bandwidth=%lu", driver, sconn, vm, NULLSTR(xmlin), NULLSTR(dconnuri), NULLSTR(uri), NULLSTR(graphicsuri), NULLSTR(listenAddress), - migrate_disks, nbdPort, NULLSTR(nbdURI), + migrate_disks, nbdPort, NULLSTR(nbdURI), NULLSTR(fdset), flags, NULLSTR(dname), bandwidth); if (flags & VIR_MIGRATE_TUNNELLED && uri) { @@ -6146,6 +6226,36 @@ qemuMigrationSrcPerformPeer2Peer(virQEMUDriver *driver, goto cleanup; } + if (flags & VIR_MIGRATE_PARALLEL && fdset) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("FD-based migration does not support parallel migration")); + goto cleanup; + } + + if (flags & VIR_MIGRATE_POSTCOPY && fdset) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("FD-based migration does not support post-copy")); + goto cleanup; + } + + if (flags & VIR_MIGRATE_POSTCOPY_RESUME && fdset) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("FD-based migration does not support post-copy resume")); + goto cleanup; + } + + if (flags & VIR_MIGRATE_ZEROCOPY && fdset) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("FD-based migration does not support zero-copy")); + goto cleanup; + } + + if (flags & VIR_MIGRATE_TLS && fdset) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("FD-based migration does not support TLS")); + goto cleanup; + } + /* the order of operations is important here; we make sure the * destination side is completely setup before we touch the source */ @@ -6226,7 +6336,7 @@ qemuMigrationSrcPerformPeer2Peer(virQEMUDriver *driver, ret = qemuMigrationSrcPerformPeer2Peer3(driver, sconn, dconn, dconnuri, vm, xmlin, persist_xml, dname, uri, graphicsuri, listenAddress, migrate_disks, migrate_disks_detect_zeroes, - nbdPort, nbdURI, migParams, bandwidth, + nbdPort, nbdURI, fdset, migParams, bandwidth, !!useParams, flags); } else { ret = qemuMigrationSrcPerformPeer2Peer2(driver, sconn, dconn, vm, @@ -6265,6 +6375,7 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, const char **migrate_disks_detect_zeroes, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, const char *cookiein, int cookieinlen, @@ -6313,7 +6424,7 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, ret = qemuMigrationSrcPerformPeer2Peer(driver, conn, vm, xmlin, persist_xml, dconnuri, uri, graphicsuri, listenAddress, migrate_disks, migrate_disks_detect_zeroes, nbdPort, - nbdURI, + nbdURI, fdset, migParams, flags, dname, bandwidth, &v3proto); } else { @@ -6323,7 +6434,7 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, ret = qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, cookieout, cookieoutlen, flags, bandwidth, NULL, NULL, NULL, NULL, - migParams, nbdURI); + fdset, migParams, nbdURI); } if (ret < 0) goto endjob; @@ -6390,6 +6501,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, const char *graphicsuri, const char **migrate_disks, const char **migrate_disks_detect_zeroes, + const char *fdset, qemuMigrationParams *migParams, const char *cookiein, int cookieinlen, @@ -6426,7 +6538,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, cookieout, cookieoutlen, flags, bandwidth, NULL, graphicsuri, migrate_disks, migrate_disks_detect_zeroes, - migParams, nbdURI) < 0) + fdset, migParams, nbdURI) < 0) goto cleanup; virCloseCallbacksDomainAdd(vm, conn, qemuMigrationAnyConnectionClosed); @@ -6471,6 +6583,7 @@ qemuMigrationSrcPerform(virQEMUDriver *driver, const char **migrate_disks_detect_zeroes, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, const char *cookiein, int cookieinlen, @@ -6486,12 +6599,12 @@ qemuMigrationSrcPerform(virQEMUDriver *driver, VIR_DEBUG("driver=%p, conn=%p, vm=%p, xmlin=%s, dconnuri=%s, " "uri=%s, graphicsuri=%s, listenAddress=%s, " "migrate_disks=%p, nbdPort=%d, " - "nbdURI=%s, " + "nbdURI=%s, fdset=%s, " "cookiein=%s, cookieinlen=%d, cookieout=%p, cookieoutlen=%p, " "flags=0x%x, dname=%s, bandwidth=%lu, v3proto=%d", driver, conn, vm, NULLSTR(xmlin), NULLSTR(dconnuri), NULLSTR(uri), NULLSTR(graphicsuri), NULLSTR(listenAddress), - migrate_disks, nbdPort, NULLSTR(nbdURI), + migrate_disks, nbdPort, NULLSTR(nbdURI), NULLSTR(fdset), NULLSTR(cookiein), cookieinlen, cookieout, cookieoutlen, flags, NULLSTR(dname), bandwidth, v3proto); @@ -6513,7 +6626,7 @@ qemuMigrationSrcPerform(virQEMUDriver *driver, return qemuMigrationSrcPerformJob(driver, conn, vm, xmlin, persist_xml, dconnuri, uri, graphicsuri, listenAddress, migrate_disks, migrate_disks_detect_zeroes, nbdPort, - nbdURI, migParams, + nbdURI, fdset, migParams, cookiein, cookieinlen, cookieout, cookieoutlen, flags, dname, bandwidth, v3proto); @@ -6529,7 +6642,7 @@ qemuMigrationSrcPerform(virQEMUDriver *driver, return qemuMigrationSrcPerformPhase(driver, conn, vm, persist_xml, uri, graphicsuri, migrate_disks, migrate_disks_detect_zeroes, - migParams, + fdset, migParams, cookiein, cookieinlen, cookieout, cookieoutlen, flags, bandwidth, nbdURI); @@ -6538,7 +6651,7 @@ qemuMigrationSrcPerform(virQEMUDriver *driver, return qemuMigrationSrcPerformJob(driver, conn, vm, xmlin, persist_xml, NULL, uri, graphicsuri, listenAddress, migrate_disks, migrate_disks_detect_zeroes, nbdPort, - nbdURI, migParams, + nbdURI, fdset, migParams, cookiein, cookieinlen, cookieout, cookieoutlen, flags, dname, bandwidth, v3proto); diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index db03144207..2bfc06aaef 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -93,6 +93,7 @@ VIR_MIGRATE_PARAM_TLS_DESTINATION, VIR_TYPED_PARAM_STRING, \ VIR_MIGRATE_PARAM_DISKS_URI, VIR_TYPED_PARAM_STRING, \ VIR_MIGRATE_PARAM_BANDWIDTH_AVAIL_SWITCHOVER, VIR_TYPED_PARAM_ULLONG, \ + VIR_MIGRATE_PARAM_FD_SET, VIR_TYPED_PARAM_STRING, \ NULL @@ -166,6 +167,7 @@ qemuMigrationDstPrepareDirect(virQEMUDriver *driver, const char **migrate_disks, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, unsigned int flags); @@ -183,6 +185,7 @@ qemuMigrationSrcPerform(virQEMUDriver *driver, const char **migrate_disks_detect_zeroes, int nbdPort, const char *nbdURI, + const char *fdset, qemuMigrationParams *migParams, const char *cookiein, int cookieinlen, -- 2.43.7
On 2 Feb 2026, at 3:29 PM, Tejus GK <tejus.gk@nutanix.com> wrote:
Libvirt supports FD-based live migrations in the form of tunnelled migrations. However, this isn't as performant as of now, since it involves extra copies.·
This patchset proposes adding support for externally provided file descriptors for live migration. By allowing the client to supply the migration FD directly, the extra copy can be avoided, improving overall migration performance.
The intent is to gather early feedback on the approach and its feasibility. Comments, concerns, and suggestions are very welcome :)
V2: 1. Removed the approach of introducing a new FD passing API and instead reuse the virDomainFDAssociate API along with a dummy VM on the destination host. 2. Introduced a new virDomainMigrate parameter, VIR_MIGRATE_PARAM_FD_SET.
Tejus GK (2): qemu: support incoming "fd" uri qemu: add support for fd based live migrations
include/libvirt/libvirt-domain.h | 14 +++ src/qemu/qemu_driver.c | 24 +++-- src/qemu/qemu_migration.c | 174 +++++++++++++++++++++++++------ src/qemu/qemu_migration.h | 3 + 4 files changed, 178 insertions(+), 37 deletions(-)
-- 2.43.7
Hi all, A gentle ping on this patch-series :) Regards, Tejus
On Mon, Feb 09, 2026 at 08:23:29 +0000, Tejus GK wrote:
On 2 Feb 2026, at 3:29 PM, Tejus GK <tejus.gk@nutanix.com> wrote:
Libvirt supports FD-based live migrations in the form of tunnelled migrations. However, this isn't as performant as of now, since it involves extra copies.·
This patchset proposes adding support for externally provided file descriptors for live migration. By allowing the client to supply the migration FD directly, the extra copy can be avoided, improving overall migration performance.
The intent is to gather early feedback on the approach and its feasibility. Comments, concerns, and suggestions are very welcome :)
V2: 1. Removed the approach of introducing a new FD passing API and instead reuse the virDomainFDAssociate API along with a dummy VM on the destination host. 2. Introduced a new virDomainMigrate parameter, VIR_MIGRATE_PARAM_FD_SET.
Tejus GK (2): qemu: support incoming "fd" uri qemu: add support for fd based live migrations
include/libvirt/libvirt-domain.h | 14 +++ src/qemu/qemu_driver.c | 24 +++-- src/qemu/qemu_migration.c | 174 +++++++++++++++++++++++++------ src/qemu/qemu_migration.h | 3 + 4 files changed, 178 insertions(+), 37 deletions(-)
-- 2.43.7
Hi all,
A gentle ping on this patch-series :)
The use of FD-passing in this series looks mostly sane to me. It's weird that the FD is labelled with the image label though. But as I've seen today while reviewing a different series that may make sense to me. This series will need to get rebased though as the virStorageSourceFDTuple was renamed today. It's just the type name, so nothing complex should be needed. In addition I've noticed that VIR_MIGRATE_PARAM_FD_SET is not implemented in virsh. You should add that. For the implementation into migration code I'll have to defer to Jiri as it's his domain. I think the best approach will be to resend the fixed version.
participants (2)
-
Peter Krempa -
Tejus GK