[libvirt] [PATCH v4 1/2] libxl: add p2p migration
Introduce support for VIR_MIGRATE_PEER2PEER in libvirt migration. Most of the changes occur at the source and no modifications at the receiver. In P2P mode there is only the Perform phase so we must handle the connection with the destination and actually perform the migration. libxlDomainPerformP2P implements the connection to the destination and libxlDoMigrateP2P implements the actual migration logic with virConnectPtr. In this function we take care of doing all phases of migration in the destination similar to virDomainMigrateVersion3Full. We appropriately save the last error reported in each of the phases to provide proper reporting. We don't yet support VIR_MIGRATE_TUNNELED and we always use V3 with extensible params, thus it also makes the implementation simpler. It is worth noting that the receiver didn't have any changes, and since it's still the v3 sequence thus it is possible to migrate from a P2P to non-P2P host. Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- Note: v4 series requires the patch ("remote: set VIR_TYPED_PARAM_STRING_OKAY on migration") [0] to make migration work again, but it's not reviewed yet. [https://www.redhat.com/archives/libvir-list/2016-February/msg00317.html] Changes since v3: - Put keep alive support into a separate patch and do it with proper configuration. - Fixed spelling errors on commit message. Changes since v2: - Remove Connect Close callback Changes since v1: - Move Begin step to libxlDoMigrateP2P to have all 4 steps together. - Remove if before VIR_FREE(dom_xml) --- src/libxl/libxl_driver.c | 13 ++- src/libxl/libxl_migration.c | 197 ++++++++++++++++++++++++++++++++++++++++++++ src/libxl/libxl_migration.h | 11 +++ 3 files changed, 218 insertions(+), 3 deletions(-) diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index 2a6c2de..9e3f568 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -5011,6 +5011,7 @@ libxlConnectSupportsFeature(virConnectPtr conn, int feature) switch (feature) { case VIR_DRV_FEATURE_TYPED_PARAM_STRING: case VIR_DRV_FEATURE_MIGRATION_PARAMS: + case VIR_DRV_FEATURE_MIGRATION_P2P: return 1; default: return 0; @@ -5336,9 +5337,15 @@ libxlDomainMigratePerform3Params(virDomainPtr dom, if (virDomainMigratePerform3ParamsEnsureACL(dom->conn, vm->def) < 0) goto cleanup; - if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, - uri, dname, flags) < 0) - goto cleanup; + if (flags & VIR_MIGRATE_PEER2PEER) { + if (libxlDomainMigrationPerformP2P(driver, vm, dom->conn, dom_xml, + dconnuri, uri, dname, flags) < 0) + goto cleanup; + } else { + if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, + uri, dname, flags) < 0) + goto cleanup; + } ret = 0; diff --git a/src/libxl/libxl_migration.c b/src/libxl/libxl_migration.c index 641bd4e..5993abc 100644 --- a/src/libxl/libxl_migration.c +++ b/src/libxl/libxl_migration.c @@ -42,6 +42,7 @@ #include "libxl_conf.h" #include "libxl_migration.h" #include "locking/domain_lock.h" +#include "virtypedparam.h" #define VIR_FROM_THIS VIR_FROM_LIBXL @@ -456,6 +457,202 @@ libxlDomainMigrationPrepare(virConnectPtr dconn, return ret; } +/* This function is a simplification of virDomainMigrateVersion3Full + * excluding tunnel support and restricting it to migration v3 + * with params since it was the first to be introduced in libxl. + */ +static int +libxlDoMigrateP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + virConnectPtr sconn, + const char *xmlin, + virConnectPtr dconn, + const char *dconnuri ATTRIBUTE_UNUSED, + const char *dname, + const char *uri, + unsigned int flags) +{ + virDomainPtr ddomain = NULL; + virTypedParameterPtr params = NULL; + int nparams = 0; + int maxparams = 0; + char *uri_out = NULL; + char *dom_xml = NULL; + unsigned long destflags; + bool cancelled = true; + virErrorPtr orig_err = NULL; + int ret = -1; + + dom_xml = libxlDomainMigrationBegin(sconn, vm, xmlin); + if (!dom_xml) + goto cleanup; + + if (virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_XML, dom_xml) < 0) + goto cleanup; + + if (dname && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_NAME, dname) < 0) + goto cleanup; + + if (uri && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_URI, uri) < 0) + goto cleanup; + + /* We don't require the destination to have P2P support + * as it looks to be normal migration from the receiver perpective. + */ + destflags = flags & ~(VIR_MIGRATE_PEER2PEER); + + VIR_DEBUG("Prepare3"); + virObjectUnlock(vm); + ret = dconn->driver->domainMigratePrepare3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, &uri_out, destflags); + virObjectLock(vm); + + if (ret == -1) + goto cleanup; + + if (uri_out) { + if (virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_URI, uri_out) < 0) { + orig_err = virSaveLastError(); + goto finish; + } + } else { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("domainMigratePrepare3 did not set uri")); + goto finish; + } + + VIR_DEBUG("Perform3 uri=%s", NULLSTR(uri_out)); + ret = libxlDomainMigrationPerform(driver, vm, NULL, NULL, + uri_out, NULL, flags); + + if (ret < 0) + orig_err = virSaveLastError(); + + cancelled = (ret < 0); + + finish: + VIR_DEBUG("Finish3 ret=%d", ret); + if (virTypedParamsGetString(params, nparams, + VIR_MIGRATE_PARAM_DEST_NAME, NULL) <= 0 && + virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_DEST_NAME, + vm->def->name) < 0) { + ddomain = NULL; + } else { + virObjectUnlock(vm); + ddomain = dconn->driver->domainMigrateFinish3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, + destflags, cancelled); + virObjectLock(vm); + } + + cancelled = (ddomain == NULL); + + /* If Finish3Params set an error, and we don't have an earlier + * one we need to preserve it in case confirm3 overwrites + */ + if (!orig_err) + orig_err = virSaveLastError(); + + VIR_DEBUG("Confirm3 cancelled=%d vm=%p", cancelled, vm); + ret = libxlDomainMigrationConfirm(driver, vm, flags, cancelled); + + if (ret < 0) + VIR_WARN("Guest %s probably left in 'paused' state on source", + vm->def->name); + + cleanup: + if (ddomain) { + virObjectUnref(ddomain); + ret = 0; + } else { + ret = -1; + } + + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + + VIR_FREE(dom_xml); + VIR_FREE(uri_out); + virTypedParamsFree(params, nparams); + return ret; +} + +static int virConnectCredType[] = { + VIR_CRED_AUTHNAME, + VIR_CRED_PASSPHRASE, +}; + +static virConnectAuth virConnectAuthConfig = { + .credtype = virConnectCredType, + .ncredtype = ARRAY_CARDINALITY(virConnectCredType), +}; + +/* On P2P mode there is only the Perform3 phase and we need to handle + * the connection with the destination libvirtd and perform the migration. + * Here we first tackle the first part of it, and libxlDoMigrationP2P handles + * the migration process with an established virConnectPtr to the destination. + */ +int +libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + virConnectPtr sconn, + const char *xmlin, + const char *dconnuri, + const char *uri_str ATTRIBUTE_UNUSED, + const char *dname, + unsigned int flags) +{ + int ret = -1; + bool useParams; + virConnectPtr dconn = NULL; + virErrorPtr orig_err = NULL; + + virObjectUnlock(vm); + dconn = virConnectOpenAuth(dconnuri, &virConnectAuthConfig, 0); + virObjectLock(vm); + + if (dconn == NULL) { + virReportError(VIR_ERR_OPERATION_FAILED, + _("Failed to connect to remote libvirt URI %s: %s"), + dconnuri, virGetLastErrorMessage()); + return ret; + } + + virObjectUnlock(vm); + useParams = VIR_DRV_SUPPORTS_FEATURE(dconn->driver, dconn, + VIR_DRV_FEATURE_MIGRATION_PARAMS); + virObjectLock(vm); + + if (!useParams) { + virReportError(VIR_ERR_OPERATION_FAILED, "%s", + _("Destination libvirt does not support migration with extensible parameters")); + goto cleanup; + } + + ret = libxlDoMigrateP2P(driver, vm, sconn, xmlin, dconn, dconnuri, + dname, uri_str, flags); + + cleanup: + orig_err = virSaveLastError(); + virObjectUnlock(vm); + virObjectUnref(dconn); + virObjectLock(vm); + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + return ret; +} + int libxlDomainMigrationPerform(libxlDriverPrivatePtr driver, virDomainObjPtr vm, diff --git a/src/libxl/libxl_migration.h b/src/libxl/libxl_migration.h index 20b45d8..0f83bb4 100644 --- a/src/libxl/libxl_migration.h +++ b/src/libxl/libxl_migration.h @@ -28,6 +28,7 @@ # define LIBXL_MIGRATION_FLAGS \ (VIR_MIGRATE_LIVE | \ + VIR_MIGRATE_PEER2PEER | \ VIR_MIGRATE_UNDEFINE_SOURCE | \ VIR_MIGRATE_PAUSED) @@ -56,6 +57,16 @@ libxlDomainMigrationPrepare(virConnectPtr dconn, unsigned int flags); int +libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + virConnectPtr sconn, + const char *dom_xml, + const char *dconnuri, + const char *uri_str, + const char *dname, + unsigned int flags); + +int libxlDomainMigrationPerform(libxlDriverPrivatePtr driver, virDomainObjPtr vm, const char *dom_xml, -- 2.1.4
This patch introduces keep alive messages support for P2P migration and it adds two new configuration entries namely 'keepalive_interval' 'keepalive_count' to control it. Behavior of these entries is the same as qemu driver thus the description is copied from there with just a few simplifications. Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- Note: v4 series requires the patch ("remote: set VIR_TYPED_PARAM_STRING_OKAY on migration") to make migration work again, but it's not reviewed yet. [https://www.redhat.com/archives/libvir-list/2016-February/msg00317.html] --- src/libxl/libvirtd_libxl.aug | 4 ++++ src/libxl/libxl.conf | 18 ++++++++++++++++++ src/libxl/libxl_conf.c | 26 ++++++++++++++++++++++++++ src/libxl/libxl_conf.h | 3 +++ src/libxl/libxl_migration.c | 6 ++++++ src/libxl/test_libvirtd_libxl.aug.in | 2 ++ 6 files changed, 59 insertions(+) diff --git a/src/libxl/libvirtd_libxl.aug b/src/libxl/libvirtd_libxl.aug index d5aa150..b31cc07 100644 --- a/src/libxl/libvirtd_libxl.aug +++ b/src/libxl/libvirtd_libxl.aug @@ -26,10 +26,14 @@ module Libvirtd_libxl = (* Config entry grouped by function - same order as example config *) let autoballoon_entry = bool_entry "autoballoon" let lock_entry = str_entry "lock_manager" + let keepalive_interval_entry = int_entry "keepalive_interval" + let keepalive_count_entry = int_entry "keepalive_count" (* Each entry in the config is one of the following ... *) let entry = autoballoon_entry | lock_entry + | keepalive_interval_entry + | keepalive_count_entry let comment = [ label "#comment" . del /#[ \t]*/ "# " . store /([^ \t\n][^\n]*)?/ . del /\n/ "\n" ] let empty = [ label "#empty" . eol ] diff --git a/src/libxl/libxl.conf b/src/libxl/libxl.conf index ba3de7a..82abdb6 100644 --- a/src/libxl/libxl.conf +++ b/src/libxl/libxl.conf @@ -20,3 +20,21 @@ # "lockd". Accepted values are "sanlock" and "lockd". # #lock_manager = "lockd" + + +# A keepalive message is sent to the daemon after keepalive_interval +# seconds of inactivity to check if the daemon is still responding; +# keepalive_count is a maximum number of keepalive messages that are +# allowed to be sent to the deamon without getting any response before +# the connection is considered broken. In other words, the connection +# is automatically closed approximately after keepalive_interval * +# (keepalive_count + 1) seconds since the last message received from +# the daemon. If keepalive_interval is set to -1, libxl driver will +# not send keepalive requests during peer-to-peer migration; however, +# the remote libvirtd can still send them and source libvirtd will +# send responses. When keepalive_count is set to 0, connections will +# be automatically closed after keepalive_interval seconds of +# inactivity without sending any keepalive messages. +# +#keepalive_interval = 5 +#keepalive_count = 5 diff --git a/src/libxl/libxl_conf.c b/src/libxl/libxl_conf.c index d7fb533..48b77d2 100644 --- a/src/libxl/libxl_conf.c +++ b/src/libxl/libxl_conf.c @@ -1659,6 +1659,10 @@ int libxlDriverConfigLoadFile(libxlDriverConfigPtr cfg, virConfValuePtr p; int ret = -1; + /* defaults for keepalive messages */ + cfg->keepAliveInterval = 5; + cfg->keepAliveCount = 5; + /* Check the file is readable before opening it, otherwise * libvirt emits an error. */ @@ -1686,6 +1690,28 @@ int libxlDriverConfigLoadFile(libxlDriverConfigPtr cfg, goto cleanup; } + if ((p = virConfGetValue(conf, "keepalive_interval"))) { + if (p->type != VIR_CONF_LONG && p->type != VIR_CONF_ULONG) { + virReportError(VIR_ERR_INTERNAL_ERROR, + "%s", + _("Unexpected type for 'keepalive_interval' setting")); + goto cleanup; + } + + cfg->keepAliveInterval = p->l; + } + + if ((p = virConfGetValue(conf, "keepalive_count"))) { + if (p->type != VIR_CONF_ULONG) { + virReportError(VIR_ERR_INTERNAL_ERROR, + "%s", + _("Unexpected type for 'keepalive_count' setting")); + goto cleanup; + } + + cfg->keepAliveCount = p->l; + } + ret = 0; cleanup: diff --git a/src/libxl/libxl_conf.h b/src/libxl/libxl_conf.h index 6ad9ad3..3c0eafb 100644 --- a/src/libxl/libxl_conf.h +++ b/src/libxl/libxl_conf.h @@ -105,6 +105,9 @@ struct _libxlDriverConfig { char *lockManagerName; + int keepAliveInterval; + unsigned int keepAliveCount; + /* Once created, caps are immutable */ virCapsPtr caps; diff --git a/src/libxl/libxl_migration.c b/src/libxl/libxl_migration.c index 5993abc..ab1f76e 100644 --- a/src/libxl/libxl_migration.c +++ b/src/libxl/libxl_migration.c @@ -615,6 +615,7 @@ libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, bool useParams; virConnectPtr dconn = NULL; virErrorPtr orig_err = NULL; + libxlDriverConfigPtr cfg = libxlDriverConfigGet(driver); virObjectUnlock(vm); dconn = virConnectOpenAuth(dconnuri, &virConnectAuthConfig, 0); @@ -627,6 +628,10 @@ libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, return ret; } + if (virConnectSetKeepAlive(dconn, cfg->keepAliveInterval, + cfg->keepAliveCount) < 0) + goto cleanup; + virObjectUnlock(vm); useParams = VIR_DRV_SUPPORTS_FEATURE(dconn->driver, dconn, VIR_DRV_FEATURE_MIGRATION_PARAMS); @@ -645,6 +650,7 @@ libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, orig_err = virSaveLastError(); virObjectUnlock(vm); virObjectUnref(dconn); + virObjectUnref(cfg); virObjectLock(vm); if (orig_err) { virSetError(orig_err); diff --git a/src/libxl/test_libvirtd_libxl.aug.in b/src/libxl/test_libvirtd_libxl.aug.in index baa8c79..63558e5 100644 --- a/src/libxl/test_libvirtd_libxl.aug.in +++ b/src/libxl/test_libvirtd_libxl.aug.in @@ -4,3 +4,5 @@ module Test_libvirtd_libxl = test Libvirtd_libxl.lns get conf = { "autoballoon" = "1" } { "lock_manager" = "lockd" } +{ "keepalive_interval" = "5" } +{ "keepalive_count" = "5" } -- 2.1.4
On 02/05/2016 01:45 PM, Joao Martins wrote:
This patch introduces keep alive messages support for P2P migration and it adds two new configuration entries namely 'keepalive_interval' 'keepalive_count' to control it. Behavior of these entries is the same as qemu driver thus the description is copied from there with just a few simplifications.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- Note: v4 series requires the patch ("remote: set VIR_TYPED_PARAM_STRING_OKAY on migration") to make migration work again, but it's not reviewed yet. [https://www.redhat.com/archives/libvir-list/2016-February/msg00317.html] --- src/libxl/libvirtd_libxl.aug | 4 ++++ src/libxl/libxl.conf | 18 ++++++++++++++++++ src/libxl/libxl_conf.c | 26 ++++++++++++++++++++++++++ src/libxl/libxl_conf.h | 3 +++ src/libxl/libxl_migration.c | 6 ++++++ src/libxl/test_libvirtd_libxl.aug.in | 2 ++ 6 files changed, 59 insertions(+)
diff --git a/src/libxl/libvirtd_libxl.aug b/src/libxl/libvirtd_libxl.aug index d5aa150..b31cc07 100644 --- a/src/libxl/libvirtd_libxl.aug +++ b/src/libxl/libvirtd_libxl.aug @@ -26,10 +26,14 @@ module Libvirtd_libxl = (* Config entry grouped by function - same order as example config *) let autoballoon_entry = bool_entry "autoballoon" let lock_entry = str_entry "lock_manager" + let keepalive_interval_entry = int_entry "keepalive_interval" + let keepalive_count_entry = int_entry "keepalive_count"
(* Each entry in the config is one of the following ... *) let entry = autoballoon_entry | lock_entry + | keepalive_interval_entry + | keepalive_count_entry
let comment = [ label "#comment" . del /#[ \t]*/ "# " . store /([^ \t\n][^\n]*)?/ . del /\n/ "\n" ] let empty = [ label "#empty" . eol ] diff --git a/src/libxl/libxl.conf b/src/libxl/libxl.conf index ba3de7a..82abdb6 100644 --- a/src/libxl/libxl.conf +++ b/src/libxl/libxl.conf @@ -20,3 +20,21 @@ # "lockd". Accepted values are "sanlock" and "lockd". # #lock_manager = "lockd" + + +# A keepalive message is sent to the daemon after keepalive_interval +# seconds of inactivity to check if the daemon is still responding; +# keepalive_count is a maximum number of keepalive messages that are +# allowed to be sent to the deamon without getting any response before +# the connection is considered broken. In other words, the connection +# is automatically closed approximately after keepalive_interval * +# (keepalive_count + 1) seconds since the last message received from +# the daemon. If keepalive_interval is set to -1, libxl driver will +# not send keepalive requests during peer-to-peer migration; however, +# the remote libvirtd can still send them and source libvirtd will +# send responses. When keepalive_count is set to 0, connections will +# be automatically closed after keepalive_interval seconds of +# inactivity without sending any keepalive messages. +# +#keepalive_interval = 5 +#keepalive_count = 5
Nit: I think the introductory sentence describing the purpose of the "keepalive protocol" in qemu.conf is worth retaining. I added it to your existing comment and pushed the series. Thanks for your patience! :-) Regards, Jim
On 02/05/2016 01:45 PM, Joao Martins wrote:
Introduce support for VIR_MIGRATE_PEER2PEER in libvirt migration. Most of the changes occur at the source and no modifications at the receiver.
In P2P mode there is only the Perform phase so we must handle the connection with the destination and actually perform the migration. libxlDomainPerformP2P implements the connection to the destination and libxlDoMigrateP2P implements the actual migration logic with virConnectPtr. In this function we take care of doing all phases of migration in the destination similar to virDomainMigrateVersion3Full. We appropriately save the last error reported in each of the phases to provide proper reporting. We don't yet support VIR_MIGRATE_TUNNELED and we always use V3 with extensible params, thus it also makes the implementation simpler.
It is worth noting that the receiver didn't have any changes, and since it's still the v3 sequence thus it is possible to migrate from a P2P to non-P2P host.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- Note: v4 series requires the patch ("remote: set VIR_TYPED_PARAM_STRING_OKAY on migration") [0] to make migration work again, but it's not reviewed yet. [https://www.redhat.com/archives/libvir-list/2016-February/msg00317.html]
Changes since v3: - Put keep alive support into a separate patch and do it with proper configuration. - Fixed spelling errors on commit message.
Changes since v2: - Remove Connect Close callback
Changes since v1: - Move Begin step to libxlDoMigrateP2P to have all 4 steps together. - Remove if before VIR_FREE(dom_xml) --- src/libxl/libxl_driver.c | 13 ++- src/libxl/libxl_migration.c | 197 ++++++++++++++++++++++++++++++++++++++++++++ src/libxl/libxl_migration.h | 11 +++ 3 files changed, 218 insertions(+), 3 deletions(-)
ACK. Regards, Jim
participants (2)
-
Jim Fehlig -
Joao Martins