[libvirt] [PATCH RFC 0/2] libxl: migration v3 and p2p support
Hey Jim, These series introduce support for peer2peer and plain v3 migration without extensible parameters. This patch series is organized as follows: The first patch introduces the P2P where the source host must take care of connection handling with destination libvirtd and is responsible for performing the migration. I introduced an additional method for the P2P Perform phase since the default Perform of libxl is then reused. The sequence is the same as v3 so it is possible to migrate to non-P2P host. libxlDoMigrateP2P actually does the migration using a virConnectPtr to the destination and thus The second patch is mostly mechanical, and functional wise is very much the same as the Params variant. The added value of introducing plain v3 is having access to older APIs such virDomainMigrateToURI2 and virDomainMigrate (in P2P mode) which are currently in use by Openstack/virt-manager. My testing was performed in xen stable-4.5 and 4.4.3 (with the added patches in the Xen CI loop) plus Openstack Kilo. In addition I backported the patches and tested also on libvirt 1.2.14 (whic is also ran by the Xen CI Openstack loop). Any comments or suggestions are welcome, Thanks! Joao Joao Martins (2): libxl: add p2p migration libxl: add v3 migration w/o params src/libxl/libxl_driver.c | 226 +++++++++++++++++++++++++++++++++++++++++++- src/libxl/libxl_migration.c | 220 ++++++++++++++++++++++++++++++++++++++++++ src/libxl/libxl_migration.h | 11 +++ 3 files changed, 454 insertions(+), 3 deletions(-) -- 2.1.4
Introduce support for VIR_MIGRATE_PEER2PEER in libvirt migration. Most of the changes occur at the source and no modifications at the receiver. In P2P mode there is only the Perform phase so we must handle the connection with the destination and actually perform the migration. libxlDomainPerformP2P implements the connection to the destination and let libxlDoMigrateP2P implements the actual migration logic with virConnectPtr. In this function we take of doing all phases of migration in the destination similar to virDomainMigrateVersion3Full. We appropriately save the last error reported in each of the phases to provide proper reporting. We don't yet support VIR_MIGRATE_TUNNELED yet and we always use V3 with extensible params, thus it also makes the implementation simpler. It is worth noting that the receiver didn't have any changes, and since it's still the v3 sequence thus it is possible to migrate from a P2P to non-P2P host. Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- src/libxl/libxl_driver.c | 13 ++- src/libxl/libxl_migration.c | 220 ++++++++++++++++++++++++++++++++++++++++++++ src/libxl/libxl_migration.h | 11 +++ 3 files changed, 241 insertions(+), 3 deletions(-) diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index 5f69b49..4d6821f 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -4710,6 +4710,7 @@ libxlConnectSupportsFeature(virConnectPtr conn, int feature) switch (feature) { case VIR_DRV_FEATURE_TYPED_PARAM_STRING: case VIR_DRV_FEATURE_MIGRATION_PARAMS: + case VIR_DRV_FEATURE_MIGRATION_P2P: return 1; default: return 0; @@ -5036,9 +5037,15 @@ libxlDomainMigratePerform3Params(virDomainPtr dom, if (virDomainMigratePerform3ParamsEnsureACL(dom->conn, vm->def) < 0) goto cleanup; - if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, - uri, dname, flags) < 0) - goto cleanup; + if (flags & VIR_MIGRATE_PEER2PEER) { + if (libxlDomainMigrationPerformP2P(driver, vm, dom->conn, dom_xml, + dconnuri, uri, dname, flags) < 0) + goto cleanup; + } else { + if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, + uri, dname, flags) < 0) + goto cleanup; + } ret = 0; diff --git a/src/libxl/libxl_migration.c b/src/libxl/libxl_migration.c index 0d23e5f..659f4d8 100644 --- a/src/libxl/libxl_migration.c +++ b/src/libxl/libxl_migration.c @@ -42,6 +42,7 @@ #include "libxl_conf.h" #include "libxl_migration.h" #include "locking/domain_lock.h" +#include "virtypedparam.h" #define VIR_FROM_THIS VIR_FROM_LIBXL @@ -456,6 +457,225 @@ libxlDomainMigrationPrepare(virConnectPtr dconn, return ret; } +/* This function is a simplification of virDomainMigrateVersion3Full + * excluding tunnel support and restricting it to migration v3 + * with params since it was the first to be introduced in libxl. + */ +static int +libxlDoMigrateP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + const char *dom_xml, + virConnectPtr dconn, + const char *dconnuri ATTRIBUTE_UNUSED, + const char *dname, + const char *uri, + unsigned int flags) +{ + virDomainPtr ddomain = NULL; + virTypedParameterPtr params = NULL; + int nparams = 0; + int maxparams = 0; + char *uri_out = NULL; + unsigned long destflags; + bool cancelled = true; + virErrorPtr orig_err = NULL; + int ret = -1; + + if (virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_XML, dom_xml) < 0) + goto cleanup; + + if (dname && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_NAME, dname) < 0) + goto cleanup; + + if (uri && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_URI, uri) < 0) + goto cleanup; + + /* We don't require the destination to have P2P support + * as it looks to be normal migration from the receiver perpective. + */ + destflags = flags & ~(VIR_MIGRATE_PEER2PEER); + + VIR_DEBUG("Prepare3"); + virObjectUnlock(vm); + ret = dconn->driver->domainMigratePrepare3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, &uri_out, destflags); + virObjectLock(vm); + + if (ret == -1) + goto cleanup; + + if (uri_out) { + if (virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_URI, uri_out) < 0) { + orig_err = virSaveLastError(); + goto finish; + } + } else { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("domainMigratePrepare3 did not set uri")); + goto finish; + } + + VIR_DEBUG("Perform3 uri=%s", NULLSTR(uri_out)); + ret = libxlDomainMigrationPerform(driver, vm, NULL, NULL, + uri_out, NULL, flags); + + if (ret < 0) + orig_err = virSaveLastError(); + + cancelled = (ret < 0); + + finish: + VIR_DEBUG("Finish3 ret=%d", ret); + if (virTypedParamsGetString(params, nparams, + VIR_MIGRATE_PARAM_DEST_NAME, NULL) <= 0 && + virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_DEST_NAME, + vm->def->name) < 0) { + ddomain = NULL; + } else { + virObjectUnlock(vm); + ddomain = dconn->driver->domainMigrateFinish3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, + destflags, cancelled); + virObjectLock(vm); + } + + cancelled = (ddomain == NULL); + + /* If Finish3Params set an error, and we don't have an earlier + * one we need to preserve it in case confirm3 overwrites + */ + if (!orig_err) + orig_err = virSaveLastError(); + + VIR_DEBUG("Confirm3 cancelled=%d vm=%p", cancelled, vm); + ret = libxlDomainMigrationConfirm(driver, vm, flags, cancelled); + + if (ret < 0) + VIR_WARN("Guest %s probably left in 'paused' state on source", + vm->def->name); + + cleanup: + if (ddomain) { + virObjectUnref(ddomain); + ret = 0; + } else { + ret = -1; + } + + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + + VIR_FREE(uri_out); + virTypedParamsFree(params, nparams); + return ret; +} + +static int virConnectCredType[] = { + VIR_CRED_AUTHNAME, + VIR_CRED_PASSPHRASE, +}; + + +static virConnectAuth virConnectAuthConfig = { + .credtype = virConnectCredType, + .ncredtype = ARRAY_CARDINALITY(virConnectCredType), +}; + +static void +libxlMigrationConnectionClosed(virConnectPtr conn, + int reason, + void *opaque) +{ + virDomainObjPtr vm = opaque; + + VIR_DEBUG("conn=%p, reason=%d, vm=%s", conn, reason, vm->def->name); + virDomainObjBroadcast(vm); +} + +/* On P2P mode there is only the Perform3 phase and we need to handle + * the connection with the destination libvirtd and perform the migration. + * Here we first tackle the first part of it, and libxlDoMigrationP2P handles + * the migration process with an established virConnectPtr to the destination. + */ +int +libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + virConnectPtr sconn, + const char *xmlin, + const char *dconnuri, + const char *uri_str ATTRIBUTE_UNUSED, + const char *dname, + unsigned int flags) +{ + char *dom_xml; + int ret = -1; + int keepAliveInterval = 5; + int keepAliveCount = 5; + bool useParams; + virConnectPtr dconn = NULL; + virErrorPtr orig_err = NULL; + + virObjectUnlock(vm); + dconn = virConnectOpenAuth(dconnuri, &virConnectAuthConfig, 0); + virObjectLock(vm); + + if (dconn == NULL) { + virReportError(VIR_ERR_OPERATION_FAILED, + _("Failed to connect to remote libvirt URI %s: %s"), + dconnuri, virGetLastErrorMessage()); + return ret; + } + + if (virConnectSetKeepAlive(dconn, keepAliveInterval, + keepAliveCount) < 0) + goto cleanup; + + if (virConnectRegisterCloseCallback(dconn, libxlMigrationConnectionClosed, + vm, NULL) < 0) { + goto cleanup; + } + + virObjectUnlock(vm); + useParams = VIR_DRV_SUPPORTS_FEATURE(dconn->driver, dconn, + VIR_DRV_FEATURE_MIGRATION_PARAMS); + virObjectLock(vm); + + if (!useParams) { + virReportError(VIR_ERR_OPERATION_FAILED, "%s", + _("Destination libvirt does not support migration with extensible parameters")); + goto cleanup; + } + + dom_xml = libxlDomainMigrationBegin(sconn, vm, xmlin); + if (!dom_xml) + goto cleanup; + + ret = libxlDoMigrateP2P(driver, vm, dom_xml, dconn, dconnuri, + dname, uri_str, flags); + + VIR_FREE(dom_xml); + cleanup: + orig_err = virSaveLastError(); + virObjectUnlock(vm); + virConnectUnregisterCloseCallback(dconn, libxlMigrationConnectionClosed); + virObjectUnref(dconn); + virObjectLock(vm); + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + return ret; +} + int libxlDomainMigrationPerform(libxlDriverPrivatePtr driver, virDomainObjPtr vm, diff --git a/src/libxl/libxl_migration.h b/src/libxl/libxl_migration.h index 20b45d8..0f83bb4 100644 --- a/src/libxl/libxl_migration.h +++ b/src/libxl/libxl_migration.h @@ -28,6 +28,7 @@ # define LIBXL_MIGRATION_FLAGS \ (VIR_MIGRATE_LIVE | \ + VIR_MIGRATE_PEER2PEER | \ VIR_MIGRATE_UNDEFINE_SOURCE | \ VIR_MIGRATE_PAUSED) @@ -56,6 +57,16 @@ libxlDomainMigrationPrepare(virConnectPtr dconn, unsigned int flags); int +libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + virConnectPtr sconn, + const char *dom_xml, + const char *dconnuri, + const char *uri_str, + const char *dname, + unsigned int flags); + +int libxlDomainMigrationPerform(libxlDriverPrivatePtr driver, virDomainObjPtr vm, const char *dom_xml, -- 2.1.4
On Tue, Sep 08, 2015 at 09:26:11AM +0100, Joao Martins wrote:
Introduce support for VIR_MIGRATE_PEER2PEER in libvirt migration. Most of the changes occur at the source and no modifications at the receiver.
In P2P mode there is only the Perform phase so we must handle the connection with the destination and actually perform the migration. libxlDomainPerformP2P implements the connection to the destination and let libxlDoMigrateP2P implements the actual migration logic with virConnectPtr. In this function we take of doing all phases of migration in the destination similar to virDomainMigrateVersion3Full. We appropriately save the last error reported in each of the phases to provide proper reporting. We don't yet support VIR_MIGRATE_TUNNELED yet and we always use V3 with extensible params, thus it also makes the implementation simpler.
It is worth noting that the receiver didn't have any changes, and since it's still the v3 sequence thus it is possible to migrate from a P2P to non-P2P host.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- src/libxl/libxl_driver.c | 13 ++- src/libxl/libxl_migration.c | 220 ++++++++++++++++++++++++++++++++++++++++++++ src/libxl/libxl_migration.h | 11 +++ 3 files changed, 241 insertions(+), 3 deletions(-)
diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index 5f69b49..4d6821f 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -4710,6 +4710,7 @@ libxlConnectSupportsFeature(virConnectPtr conn, int feature) switch (feature) { case VIR_DRV_FEATURE_TYPED_PARAM_STRING: case VIR_DRV_FEATURE_MIGRATION_PARAMS: + case VIR_DRV_FEATURE_MIGRATION_P2P: return 1; default: return 0; @@ -5036,9 +5037,15 @@ libxlDomainMigratePerform3Params(virDomainPtr dom, if (virDomainMigratePerform3ParamsEnsureACL(dom->conn, vm->def) < 0) goto cleanup;
- if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, - uri, dname, flags) < 0) - goto cleanup; + if (flags & VIR_MIGRATE_PEER2PEER) { + if (libxlDomainMigrationPerformP2P(driver, vm, dom->conn, dom_xml, + dconnuri, uri, dname, flags) < 0) + goto cleanup; + } else { + if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, + uri, dname, flags) < 0) + goto cleanup; + }
ret = 0;
diff --git a/src/libxl/libxl_migration.c b/src/libxl/libxl_migration.c index 0d23e5f..659f4d8 100644 --- a/src/libxl/libxl_migration.c +++ b/src/libxl/libxl_migration.c @@ -42,6 +42,7 @@ #include "libxl_conf.h" #include "libxl_migration.h" #include "locking/domain_lock.h" +#include "virtypedparam.h"
#define VIR_FROM_THIS VIR_FROM_LIBXL
@@ -456,6 +457,225 @@ libxlDomainMigrationPrepare(virConnectPtr dconn, return ret; }
+/* This function is a simplification of virDomainMigrateVersion3Full + * excluding tunnel support and restricting it to migration v3 + * with params since it was the first to be introduced in libxl. + */ +static int +libxlDoMigrateP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + const char *dom_xml, + virConnectPtr dconn, + const char *dconnuri ATTRIBUTE_UNUSED, + const char *dname, + const char *uri, + unsigned int flags) +{ + virDomainPtr ddomain = NULL; + virTypedParameterPtr params = NULL; + int nparams = 0; + int maxparams = 0; + char *uri_out = NULL; + unsigned long destflags; + bool cancelled = true; + virErrorPtr orig_err = NULL; + int ret = -1; + + if (virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_XML, dom_xml) < 0) + goto cleanup; + + if (dname && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_NAME, dname) < 0) + goto cleanup; + + if (uri && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_URI, uri) < 0) + goto cleanup; + + /* We don't require the destination to have P2P support + * as it looks to be normal migration from the receiver perpective. + */ + destflags = flags & ~(VIR_MIGRATE_PEER2PEER); + + VIR_DEBUG("Prepare3"); + virObjectUnlock(vm); + ret = dconn->driver->domainMigratePrepare3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, &uri_out, destflags); + virObjectLock(vm); + + if (ret == -1) + goto cleanup; + + if (uri_out) { + if (virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_URI, uri_out) < 0) { + orig_err = virSaveLastError(); + goto finish; + } + } else { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("domainMigratePrepare3 did not set uri")); + goto finish; + } + + VIR_DEBUG("Perform3 uri=%s", NULLSTR(uri_out)); + ret = libxlDomainMigrationPerform(driver, vm, NULL, NULL, + uri_out, NULL, flags); + + if (ret < 0) + orig_err = virSaveLastError(); + + cancelled = (ret < 0); + + finish: + VIR_DEBUG("Finish3 ret=%d", ret); + if (virTypedParamsGetString(params, nparams, + VIR_MIGRATE_PARAM_DEST_NAME, NULL) <= 0 && + virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_DEST_NAME, + vm->def->name) < 0) { + ddomain = NULL; + } else { + virObjectUnlock(vm); + ddomain = dconn->driver->domainMigrateFinish3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, + destflags, cancelled); + virObjectLock(vm); + } + + cancelled = (ddomain == NULL); + + /* If Finish3Params set an error, and we don't have an earlier + * one we need to preserve it in case confirm3 overwrites + */ + if (!orig_err) + orig_err = virSaveLastError(); + + VIR_DEBUG("Confirm3 cancelled=%d vm=%p", cancelled, vm); + ret = libxlDomainMigrationConfirm(driver, vm, flags, cancelled); + + if (ret < 0) + VIR_WARN("Guest %s probably left in 'paused' state on source", + vm->def->name); + + cleanup: + if (ddomain) { + virObjectUnref(ddomain); + ret = 0; + } else { + ret = -1; + } + + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + + VIR_FREE(uri_out); + virTypedParamsFree(params, nparams); + return ret; +} + +static int virConnectCredType[] = { + VIR_CRED_AUTHNAME, + VIR_CRED_PASSPHRASE, +}; + + +static virConnectAuth virConnectAuthConfig = { + .credtype = virConnectCredType, + .ncredtype = ARRAY_CARDINALITY(virConnectCredType), +}; + +static void +libxlMigrationConnectionClosed(virConnectPtr conn, + int reason, + void *opaque) +{ + virDomainObjPtr vm = opaque; + + VIR_DEBUG("conn=%p, reason=%d, vm=%s", conn, reason, vm->def->name); + virDomainObjBroadcast(vm); +} + +/* On P2P mode there is only the Perform3 phase and we need to handle + * the connection with the destination libvirtd and perform the migration. + * Here we first tackle the first part of it, and libxlDoMigrationP2P handles + * the migration process with an established virConnectPtr to the destination. + */ +int +libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + virConnectPtr sconn, + const char *xmlin, + const char *dconnuri, + const char *uri_str ATTRIBUTE_UNUSED, + const char *dname, + unsigned int flags) +{ + char *dom_xml; + int ret = -1; + int keepAliveInterval = 5; + int keepAliveCount = 5; + bool useParams; + virConnectPtr dconn = NULL; + virErrorPtr orig_err = NULL; + + virObjectUnlock(vm); + dconn = virConnectOpenAuth(dconnuri, &virConnectAuthConfig, 0); + virObjectLock(vm); + + if (dconn == NULL) { + virReportError(VIR_ERR_OPERATION_FAILED, + _("Failed to connect to remote libvirt URI %s: %s"), + dconnuri, virGetLastErrorMessage()); + return ret; + } + + if (virConnectSetKeepAlive(dconn, keepAliveInterval, + keepAliveCount) < 0) + goto cleanup; + + if (virConnectRegisterCloseCallback(dconn, libxlMigrationConnectionClosed, + vm, NULL) < 0) { + goto cleanup; + } + + virObjectUnlock(vm); + useParams = VIR_DRV_SUPPORTS_FEATURE(dconn->driver, dconn, + VIR_DRV_FEATURE_MIGRATION_PARAMS); + virObjectLock(vm); + + if (!useParams) { + virReportError(VIR_ERR_OPERATION_FAILED, "%s", + _("Destination libvirt does not support migration with extensible parameters")); + goto cleanup; + } + + dom_xml = libxlDomainMigrationBegin(sconn, vm, xmlin); + if (!dom_xml) + goto cleanup;
Since you have the other 4 steps of the V3 protocol invoked in libxlDoMigrateP2P, I'd rather expect the Begin step to be in that method too.
+ + ret = libxlDoMigrateP2P(driver, vm, dom_xml, dconn, dconnuri, + dname, uri_str, flags); + + VIR_FREE(dom_xml); + cleanup: + orig_err = virSaveLastError(); + virObjectUnlock(vm); + virConnectUnregisterCloseCallback(dconn, libxlMigrationConnectionClosed); + virObjectUnref(dconn); + virObjectLock(vm); + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + return ret; +}
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 09/09/2015 02:58 PM, Daniel P. Berrange wrote:
On Tue, Sep 08, 2015 at 09:26:11AM +0100, Joao Martins wrote:
Introduce support for VIR_MIGRATE_PEER2PEER in libvirt migration. Most of the changes occur at the source and no modifications at the receiver.
In P2P mode there is only the Perform phase so we must handle the connection with the destination and actually perform the migration. libxlDomainPerformP2P implements the connection to the destination and let libxlDoMigrateP2P implements the actual migration logic with virConnectPtr. In this function we take of doing all phases of migration in the destination similar to virDomainMigrateVersion3Full. We appropriately save the last error reported in each of the phases to provide proper reporting. We don't yet support VIR_MIGRATE_TUNNELED yet and we always use V3 with extensible params, thus it also makes the implementation simpler.
It is worth noting that the receiver didn't have any changes, and since it's still the v3 sequence thus it is possible to migrate from a P2P to non-P2P host.
Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- src/libxl/libxl_driver.c | 13 ++- src/libxl/libxl_migration.c | 220 ++++++++++++++++++++++++++++++++++++++++++++ src/libxl/libxl_migration.h | 11 +++ 3 files changed, 241 insertions(+), 3 deletions(-)
diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index 5f69b49..4d6821f 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -4710,6 +4710,7 @@ libxlConnectSupportsFeature(virConnectPtr conn, int feature) switch (feature) { case VIR_DRV_FEATURE_TYPED_PARAM_STRING: case VIR_DRV_FEATURE_MIGRATION_PARAMS: + case VIR_DRV_FEATURE_MIGRATION_P2P: return 1; default: return 0; @@ -5036,9 +5037,15 @@ libxlDomainMigratePerform3Params(virDomainPtr dom, if (virDomainMigratePerform3ParamsEnsureACL(dom->conn, vm->def) < 0) goto cleanup;
- if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, - uri, dname, flags) < 0) - goto cleanup; + if (flags & VIR_MIGRATE_PEER2PEER) { + if (libxlDomainMigrationPerformP2P(driver, vm, dom->conn, dom_xml, + dconnuri, uri, dname, flags) < 0) + goto cleanup; + } else { + if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, + uri, dname, flags) < 0) + goto cleanup; + }
ret = 0;
diff --git a/src/libxl/libxl_migration.c b/src/libxl/libxl_migration.c index 0d23e5f..659f4d8 100644 --- a/src/libxl/libxl_migration.c +++ b/src/libxl/libxl_migration.c @@ -42,6 +42,7 @@ #include "libxl_conf.h" #include "libxl_migration.h" #include "locking/domain_lock.h" +#include "virtypedparam.h"
#define VIR_FROM_THIS VIR_FROM_LIBXL
@@ -456,6 +457,225 @@ libxlDomainMigrationPrepare(virConnectPtr dconn, return ret; }
+/* This function is a simplification of virDomainMigrateVersion3Full + * excluding tunnel support and restricting it to migration v3 + * with params since it was the first to be introduced in libxl. + */ +static int +libxlDoMigrateP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + const char *dom_xml, + virConnectPtr dconn, + const char *dconnuri ATTRIBUTE_UNUSED, + const char *dname, + const char *uri, + unsigned int flags) +{ + virDomainPtr ddomain = NULL; + virTypedParameterPtr params = NULL; + int nparams = 0; + int maxparams = 0; + char *uri_out = NULL; + unsigned long destflags; + bool cancelled = true; + virErrorPtr orig_err = NULL; + int ret = -1; + + if (virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_XML, dom_xml) < 0) + goto cleanup; + + if (dname && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_DEST_NAME, dname) < 0) + goto cleanup; + + if (uri && + virTypedParamsAddString(¶ms, &nparams, &maxparams, + VIR_MIGRATE_PARAM_URI, uri) < 0) + goto cleanup; + + /* We don't require the destination to have P2P support + * as it looks to be normal migration from the receiver perpective. + */ + destflags = flags & ~(VIR_MIGRATE_PEER2PEER); + + VIR_DEBUG("Prepare3"); + virObjectUnlock(vm); + ret = dconn->driver->domainMigratePrepare3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, &uri_out, destflags); + virObjectLock(vm); + + if (ret == -1) + goto cleanup; + + if (uri_out) { + if (virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_URI, uri_out) < 0) { + orig_err = virSaveLastError(); + goto finish; + } + } else { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("domainMigratePrepare3 did not set uri")); + goto finish; + } + + VIR_DEBUG("Perform3 uri=%s", NULLSTR(uri_out)); + ret = libxlDomainMigrationPerform(driver, vm, NULL, NULL, + uri_out, NULL, flags); + + if (ret < 0) + orig_err = virSaveLastError(); + + cancelled = (ret < 0); + + finish: + VIR_DEBUG("Finish3 ret=%d", ret); + if (virTypedParamsGetString(params, nparams, + VIR_MIGRATE_PARAM_DEST_NAME, NULL) <= 0 && + virTypedParamsReplaceString(¶ms, &nparams, + VIR_MIGRATE_PARAM_DEST_NAME, + vm->def->name) < 0) { + ddomain = NULL; + } else { + virObjectUnlock(vm); + ddomain = dconn->driver->domainMigrateFinish3Params + (dconn, params, nparams, NULL, 0, NULL, NULL, + destflags, cancelled); + virObjectLock(vm); + } + + cancelled = (ddomain == NULL); + + /* If Finish3Params set an error, and we don't have an earlier + * one we need to preserve it in case confirm3 overwrites + */ + if (!orig_err) + orig_err = virSaveLastError(); + + VIR_DEBUG("Confirm3 cancelled=%d vm=%p", cancelled, vm); + ret = libxlDomainMigrationConfirm(driver, vm, flags, cancelled); + + if (ret < 0) + VIR_WARN("Guest %s probably left in 'paused' state on source", + vm->def->name); + + cleanup: + if (ddomain) { + virObjectUnref(ddomain); + ret = 0; + } else { + ret = -1; + } + + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + + VIR_FREE(uri_out); + virTypedParamsFree(params, nparams); + return ret; +} + +static int virConnectCredType[] = { + VIR_CRED_AUTHNAME, + VIR_CRED_PASSPHRASE, +}; + + +static virConnectAuth virConnectAuthConfig = { + .credtype = virConnectCredType, + .ncredtype = ARRAY_CARDINALITY(virConnectCredType), +}; + +static void +libxlMigrationConnectionClosed(virConnectPtr conn, + int reason, + void *opaque) +{ + virDomainObjPtr vm = opaque; + + VIR_DEBUG("conn=%p, reason=%d, vm=%s", conn, reason, vm->def->name); + virDomainObjBroadcast(vm); +} + +/* On P2P mode there is only the Perform3 phase and we need to handle + * the connection with the destination libvirtd and perform the migration. + * Here we first tackle the first part of it, and libxlDoMigrationP2P handles + * the migration process with an established virConnectPtr to the destination. + */ +int +libxlDomainMigrationPerformP2P(libxlDriverPrivatePtr driver, + virDomainObjPtr vm, + virConnectPtr sconn, + const char *xmlin, + const char *dconnuri, + const char *uri_str ATTRIBUTE_UNUSED, + const char *dname, + unsigned int flags) +{ + char *dom_xml; + int ret = -1; + int keepAliveInterval = 5; + int keepAliveCount = 5; + bool useParams; + virConnectPtr dconn = NULL; + virErrorPtr orig_err = NULL; + + virObjectUnlock(vm); + dconn = virConnectOpenAuth(dconnuri, &virConnectAuthConfig, 0); + virObjectLock(vm); + + if (dconn == NULL) { + virReportError(VIR_ERR_OPERATION_FAILED, + _("Failed to connect to remote libvirt URI %s: %s"), + dconnuri, virGetLastErrorMessage()); + return ret; + } + + if (virConnectSetKeepAlive(dconn, keepAliveInterval, + keepAliveCount) < 0) + goto cleanup; + + if (virConnectRegisterCloseCallback(dconn, libxlMigrationConnectionClosed, + vm, NULL) < 0) { + goto cleanup; + } + + virObjectUnlock(vm); + useParams = VIR_DRV_SUPPORTS_FEATURE(dconn->driver, dconn, + VIR_DRV_FEATURE_MIGRATION_PARAMS); + virObjectLock(vm); + + if (!useParams) { + virReportError(VIR_ERR_OPERATION_FAILED, "%s", + _("Destination libvirt does not support migration with extensible parameters")); + goto cleanup; + } + + dom_xml = libxlDomainMigrationBegin(sconn, vm, xmlin); + if (!dom_xml) + goto cleanup;
Since you have the other 4 steps of the V3 protocol invoked in libxlDoMigrateP2P, I'd rather expect the Begin step to be in that method too.
It makes things more clear indeed. I will change in v2 version.
+ + ret = libxlDoMigrateP2P(driver, vm, dom_xml, dconn, dconnuri, + dname, uri_str, flags); + + VIR_FREE(dom_xml); + cleanup: + orig_err = virSaveLastError(); + virObjectUnlock(vm); + virConnectUnregisterCloseCallback(dconn, libxlMigrationConnectionClosed); + virObjectUnref(dconn); + virObjectLock(vm); + if (orig_err) { + virSetError(orig_err); + virFreeError(orig_err); + } + return ret; +}
Regards, Daniel
This patch introduces migration v3 without ext. params. Most of the changes are mechanical and most of it is moving code and handling of the arguments in different way. Functional-wise it works same way as its "Params" variants. By having v3 migration we end up supporting older variants of virDomainMigrateToURI which don't use v3 with params. The latter is only supported by the latest virDomainMigrateToURI3, thus we broaden the API support of migration in libxl e.g. ability to use virDomainMigrateToURI2 and virDomainMigrate (in P2P mode). Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- src/libxl/libxl_driver.c | 213 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+) diff --git a/src/libxl/libxl_driver.c b/src/libxl/libxl_driver.c index 4d6821f..afb8a7d 100644 --- a/src/libxl/libxl_driver.c +++ b/src/libxl/libxl_driver.c @@ -4711,6 +4711,7 @@ libxlConnectSupportsFeature(virConnectPtr conn, int feature) case VIR_DRV_FEATURE_TYPED_PARAM_STRING: case VIR_DRV_FEATURE_MIGRATION_PARAMS: case VIR_DRV_FEATURE_MIGRATION_P2P: + case VIR_DRV_FEATURE_MIGRATION_V3: return 1; default: return 0; @@ -4890,6 +4891,42 @@ libxlNodeDeviceReset(virNodeDevicePtr dev) } static char * +libxlDomainMigrateBegin3(virDomainPtr domain, + const char *xmlin, + char **cookieout ATTRIBUTE_UNUSED, + int *cookieoutlen ATTRIBUTE_UNUSED, + unsigned long flags, + const char *dname ATTRIBUTE_UNUSED, + unsigned long resource ATTRIBUTE_UNUSED) +{ + virDomainObjPtr vm = NULL; + +#ifdef LIBXL_HAVE_NO_SUSPEND_RESUME + virReportUnsupportedError(); + return NULL; +#endif + + virCheckFlags(LIBXL_MIGRATION_FLAGS, NULL); + + if (!(vm = libxlDomObjFromDomain(domain))) + return NULL; + + if (virDomainMigrateBegin3EnsureACL(domain->conn, vm->def) < 0) { + virObjectUnlock(vm); + return NULL; + } + + if (!virDomainObjIsActive(vm)) { + virReportError(VIR_ERR_OPERATION_INVALID, + "%s", _("domain is not running")); + virObjectUnlock(vm); + return NULL; + } + + return libxlDomainMigrationBegin(domain->conn, vm, xmlin); +} + +static char * libxlDomainMigrateBegin3Params(virDomainPtr domain, virTypedParameterPtr params, int nparams, @@ -4939,6 +4976,46 @@ libxlDomainMigrateBegin3Params(virDomainPtr domain, } static int +libxlDomainMigratePrepare3(virConnectPtr dconn, + const char *cookiein ATTRIBUTE_UNUSED, + int cookieinlen ATTRIBUTE_UNUSED, + char **cookieout ATTRIBUTE_UNUSED, + int *cookieoutlen ATTRIBUTE_UNUSED, + const char *uri_in, + char **uri_out, + unsigned long flags, + const char *dname, + unsigned long resource ATTRIBUTE_UNUSED, + const char *dom_xml) +{ + libxlDriverPrivatePtr driver = dconn->privateData; + virDomainDefPtr def = NULL; + +#ifdef LIBXL_HAVE_NO_SUSPEND_RESUME + virReportUnsupportedError(); + return -1; +#endif + + virCheckFlags(LIBXL_MIGRATION_FLAGS, -1); + + if (!(def = libxlDomainMigrationPrepareDef(driver, dom_xml, dname))) + goto error; + + if (virDomainMigratePrepare3EnsureACL(dconn, def) < 0) + goto error; + + if (libxlDomainMigrationPrepare(dconn, &def, uri_in, uri_out, flags) < 0) + goto error; + + return 0; + + error: + virDomainDefFree(def); + return -1; + +} + +static int libxlDomainMigratePrepare3Params(virConnectPtr dconn, virTypedParameterPtr params, int nparams, @@ -4993,6 +5070,55 @@ libxlDomainMigratePrepare3Params(virConnectPtr dconn, } static int +libxlDomainMigratePerform3(virDomainPtr dom, + const char *dom_xml, + const char *cookiein ATTRIBUTE_UNUSED, + int cookieinlen ATTRIBUTE_UNUSED, + char **cookieout ATTRIBUTE_UNUSED, + int *cookieoutlen ATTRIBUTE_UNUSED, + const char *dconnuri, + const char *uri, + unsigned long flags, + const char *dname, + unsigned long resource ATTRIBUTE_UNUSED) +{ + libxlDriverPrivatePtr driver = dom->conn->privateData; + virDomainObjPtr vm = NULL; + int ret = -1; + +#ifdef LIBXL_HAVE_NO_SUSPEND_RESUME + virReportUnsupportedError(); + return -1; +#endif + + virCheckFlags(LIBXL_MIGRATION_FLAGS, -1); + + if (!(vm = libxlDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainMigratePerform3EnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (flags & VIR_MIGRATE_PEER2PEER) { + if (libxlDomainMigrationPerformP2P(driver, vm, dom->conn, dom_xml, + dconnuri, uri, dname, flags) < 0) + goto cleanup; + } else { + if (libxlDomainMigrationPerform(driver, vm, dom_xml, dconnuri, + uri, dname, flags) < 0) + goto cleanup; + } + + ret = 0; + + cleanup: + if (vm) + virObjectUnlock(vm); + return ret; + +} + +static int libxlDomainMigratePerform3Params(virDomainPtr dom, const char *dconnuri, virTypedParameterPtr params, @@ -5055,6 +5181,60 @@ libxlDomainMigratePerform3Params(virDomainPtr dom, return ret; } + +static virDomainPtr +libxlDomainMigrateFinish3(virConnectPtr dconn, + const char *dname, + const char *cookiein ATTRIBUTE_UNUSED, + int cookieinlen ATTRIBUTE_UNUSED, + char **cookieout ATTRIBUTE_UNUSED, + int *cookieoutlen ATTRIBUTE_UNUSED, + const char *dconnuri ATTRIBUTE_UNUSED, + const char *uri ATTRIBUTE_UNUSED, + unsigned long flags, + int cancelled) +{ + libxlDriverPrivatePtr driver = dconn->privateData; + virDomainObjPtr vm = NULL; + virDomainPtr ret = NULL; + +#ifdef LIBXL_HAVE_NO_SUSPEND_RESUME + virReportUnsupportedError(); + return NULL; +#endif + + virCheckFlags(LIBXL_MIGRATION_FLAGS, NULL); + + if (!dname || + !(vm = virDomainObjListFindByName(driver->domains, dname))) { + /* Migration obviously failed if the domain doesn't exist */ + virReportError(VIR_ERR_OPERATION_FAILED, + _("Migration failed. No domain on destination host " + "with matching name '%s'"), + NULLSTR(dname)); + return NULL; + } + + if (virDomainMigrateFinish3EnsureACL(dconn, vm->def) < 0) { + virDomainObjEndAPI(&vm); + return NULL; + } + + if (libxlDomainObjBeginJob(driver, vm, LIBXL_JOB_MODIFY) < 0) { + virDomainObjEndAPI(&vm); + return NULL; + } + + ret = libxlDomainMigrationFinish(dconn, vm, flags, cancelled); + + if (!libxlDomainObjEndJob(driver, vm)) + vm = NULL; + + virDomainObjEndAPI(&vm); + + return ret; +} + static virDomainPtr libxlDomainMigrateFinish3Params(virConnectPtr dconn, virTypedParameterPtr params, @@ -5116,6 +5296,34 @@ libxlDomainMigrateFinish3Params(virConnectPtr dconn, } static int +libxlDomainMigrateConfirm3(virDomainPtr domain, + const char *cookiein ATTRIBUTE_UNUSED, + int cookieinlen ATTRIBUTE_UNUSED, + unsigned long flags, + int cancelled) +{ + libxlDriverPrivatePtr driver = domain->conn->privateData; + virDomainObjPtr vm = NULL; + +#ifdef LIBXL_HAVE_NO_SUSPEND_RESUME + virReportUnsupportedError(); + return -1; +#endif + + virCheckFlags(LIBXL_MIGRATION_FLAGS, -1); + + if (!(vm = libxlDomObjFromDomain(domain))) + return -1; + + if (virDomainMigrateConfirm3EnsureACL(domain->conn, vm->def) < 0) { + virObjectUnlock(vm); + return -1; + } + + return libxlDomainMigrationConfirm(driver, vm, flags, cancelled); +} + +static int libxlDomainMigrateConfirm3Params(virDomainPtr domain, virTypedParameterPtr params, int nparams, @@ -5254,10 +5462,15 @@ static virHypervisorDriver libxlHypervisorDriver = { .nodeDeviceDetachFlags = libxlNodeDeviceDetachFlags, /* 1.2.3 */ .nodeDeviceReAttach = libxlNodeDeviceReAttach, /* 1.2.3 */ .nodeDeviceReset = libxlNodeDeviceReset, /* 1.2.3 */ + .domainMigrateBegin3 = libxlDomainMigrateBegin3, /* 1.2.20 */ .domainMigrateBegin3Params = libxlDomainMigrateBegin3Params, /* 1.2.6 */ + .domainMigratePrepare3 = libxlDomainMigratePrepare3, /* 1.2.20 */ .domainMigratePrepare3Params = libxlDomainMigratePrepare3Params, /* 1.2.6 */ + .domainMigratePerform3 = libxlDomainMigratePerform3, /* 1.2.20 */ .domainMigratePerform3Params = libxlDomainMigratePerform3Params, /* 1.2.6 */ + .domainMigrateFinish3 = libxlDomainMigrateFinish3, /* 1.2.20 */ .domainMigrateFinish3Params = libxlDomainMigrateFinish3Params, /* 1.2.6 */ + .domainMigrateConfirm3 = libxlDomainMigrateConfirm3, /* 1.2.20 */ .domainMigrateConfirm3Params = libxlDomainMigrateConfirm3Params, /* 1.2.6 */ .nodeGetSecurityModel = libxlNodeGetSecurityModel, /* 1.2.16 */ }; -- 2.1.4
On Tue, Sep 08, 2015 at 09:26:12AM +0100, Joao Martins wrote:
This patch introduces migration v3 without ext. params. Most of the changes are mechanical and most of it is moving code and handling of the arguments in different way. Functional-wise it works same way as its "Params" variants.
By having v3 migration we end up supporting older variants of virDomainMigrateToURI which don't use v3 with params. The latter is only supported by the latest virDomainMigrateToURI3, thus we broaden the API support of migration in libxl e.g. ability to use virDomainMigrateToURI2 and virDomainMigrate (in P2P mode).
Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- src/libxl/libxl_driver.c | 213 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+)
Hmm, this is the same issue that VZ driver has. It is madness for us to make every virt driver re-implement the V3 non-params variants. We could provide some standard stub impls in a common file that simply pack the parameters into a virTypedParam array and then invoke the V3 params variant APIs. That way we have on impl for all drivers. Would you like to create a file src/migration-helpers.c that contains impls of the domain(Prepare|Begin|Perform|Finish|Confirm)V3 methods, that simply create a virTypedParam array, and then re-invoke domain(Prepare|Begin|Perform|Finish|Confirm)V3Params methods. Then update QEMU driver to use those, and then also use them in libxl eg this:
@@ -5254,10 +5462,15 @@ static virHypervisorDriver libxlHypervisorDriver = { .nodeDeviceDetachFlags = libxlNodeDeviceDetachFlags, /* 1.2.3 */ .nodeDeviceReAttach = libxlNodeDeviceReAttach, /* 1.2.3 */ .nodeDeviceReset = libxlNodeDeviceReset, /* 1.2.3 */ + .domainMigrateBegin3 = libxlDomainMigrateBegin3, /* 1.2.20 */
would just do + .domainMigrateBegin3 = stubDomainMigrateBegin3, /* 1.2.20 */ where stubDomainMigrateBegin3 is the method from migration-helpers.c
.domainMigrateBegin3Params = libxlDomainMigrateBegin3Params, /* 1.2.6 */ + .domainMigratePrepare3 = libxlDomainMigratePrepare3, /* 1.2.20 */ .domainMigratePrepare3Params = libxlDomainMigratePrepare3Params, /* 1.2.6 */ + .domainMigratePerform3 = libxlDomainMigratePerform3, /* 1.2.20 */ .domainMigratePerform3Params = libxlDomainMigratePerform3Params, /* 1.2.6 */ + .domainMigrateFinish3 = libxlDomainMigrateFinish3, /* 1.2.20 */ .domainMigrateFinish3Params = libxlDomainMigrateFinish3Params, /* 1.2.6 */ + .domainMigrateConfirm3 = libxlDomainMigrateConfirm3, /* 1.2.20 */ .domainMigrateConfirm3Params = libxlDomainMigrateConfirm3Params, /* 1.2.6 */ .nodeGetSecurityModel = libxlNodeGetSecurityModel, /* 1.2.16 */ }; --
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 09/09/2015 03:03 PM, Daniel P. Berrange wrote:
On Tue, Sep 08, 2015 at 09:26:12AM +0100, Joao Martins wrote:
This patch introduces migration v3 without ext. params. Most of the changes are mechanical and most of it is moving code and handling of the arguments in different way. Functional-wise it works same way as its "Params" variants.
By having v3 migration we end up supporting older variants of virDomainMigrateToURI which don't use v3 with params. The latter is only supported by the latest virDomainMigrateToURI3, thus we broaden the API support of migration in libxl e.g. ability to use virDomainMigrateToURI2 and virDomainMigrate (in P2P mode).
Signed-off-by: Joao Martins <joao.m.martins@oracle.com> --- src/libxl/libxl_driver.c | 213 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 213 insertions(+)
Hmm, this is the same issue that VZ driver has. It is madness for us to make every virt driver re-implement the V3 non-params variants. We could provide some standard stub impls in a common file that simply pack the parameters into a virTypedParam array and then invoke the V3 params variant APIs. That way we have on impl for all drivers.
Initially I thought of implementing that way since it was "just" converting the virTypedParams, but I thought libxl was the only driver affected.
Would you like to create a file src/migration-helpers.c that contains impls of the domain(Prepare|Begin|Perform|Finish|Confirm)V3 methods, that simply create a virTypedParam array, and then re-invoke domain(Prepare|Begin|Perform|Finish|Confirm)V3Params methods. Then update QEMU driver to use those, and then also use them in libxl
Definitely! I will repost it on my next series.
eg this:
@@ -5254,10 +5462,15 @@ static virHypervisorDriver libxlHypervisorDriver = { .nodeDeviceDetachFlags = libxlNodeDeviceDetachFlags, /* 1.2.3 */ .nodeDeviceReAttach = libxlNodeDeviceReAttach, /* 1.2.3 */ .nodeDeviceReset = libxlNodeDeviceReset, /* 1.2.3 */ + .domainMigrateBegin3 = libxlDomainMigrateBegin3, /* 1.2.20 */
would just do
+ .domainMigrateBegin3 = stubDomainMigrateBegin3, /* 1.2.20 */
where stubDomainMigrateBegin3 is the method from migration-helpers.c
.domainMigrateBegin3Params = libxlDomainMigrateBegin3Params, /* 1.2.6 */ + .domainMigratePrepare3 = libxlDomainMigratePrepare3, /* 1.2.20 */ .domainMigratePrepare3Params = libxlDomainMigratePrepare3Params, /* 1.2.6 */ + .domainMigratePerform3 = libxlDomainMigratePerform3, /* 1.2.20 */ .domainMigratePerform3Params = libxlDomainMigratePerform3Params, /* 1.2.6 */ + .domainMigrateFinish3 = libxlDomainMigrateFinish3, /* 1.2.20 */ .domainMigrateFinish3Params = libxlDomainMigrateFinish3Params, /* 1.2.6 */ + .domainMigrateConfirm3 = libxlDomainMigrateConfirm3, /* 1.2.20 */ .domainMigrateConfirm3Params = libxlDomainMigrateConfirm3Params, /* 1.2.6 */ .nodeGetSecurityModel = libxlNodeGetSecurityModel, /* 1.2.16 */ }; --
Regards, Daniel
participants (2)
-
Daniel P. Berrange -
Joao Martins