[libvirt] [PATCH 00/16] Implement post-copy migration
(See "Add public APIs for post-copy migration" patch for more details about post-copy migration.) Post-copy support was originally written by Cristian Klein in 2014, but no one touched the series since then. Some patches in this series are modified versions of the old patches from Cristian, some patches had to be rewritten from scratch since libvirt code changed a lot (we started using migration events), and some patches are completely new. While post-copy migration is included in QEMU 2.5.0, it didn't support everything libvirt needs. Thus you need QEMU from git to use post-copy. Luckily, the QEMU migration capability we need to enable to use post-copy is still prefixed with "x-", which means it's still considered experimental. Once we are sure QEMU gives us all we need, the "x-" prefix will be removed. Seamless SPICE migration doesn't work with post-copy now, but I'll look at that. This series (VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY support) depends on "Introduce migration iteration event" series I sent yesterday. Cristian Klein (6): Add public APIs for post-copy migration qemu: Add QMP functions for post-copy migration qemu: Add support for VIR_MIGRATE_POSTCOPY flag qemu: Implement virDomainMigrateStartPostCopy virsh: Add support for post-copy migration virsh: Add --postcopy-after-precopy option to migrate Jiri Denemark (10): Add event and state details for post-copy qemu: Don't explicitly stop CPUs after migration qemu: Handle postcopy-active migration state qemu: Add flags to qemuMigrationWaitForCompletion qemu: Add support for VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag virsh: Configurable migrate --timeout action qemu: Don't kill running migrated domain on daemon restart qemu: Refactor qemuProcessRecoverMigration qemu: Handle post-copy migration failures qemu: Refuse to abort migration in post-copy mode examples/object-events/event-test.c | 9 ++ include/libvirt/libvirt-domain.h | 12 ++ src/conf/domain_conf.c | 7 +- src/driver-hypervisor.h | 5 + src/libvirt-domain.c | 140 +++++++++++++++++ src/libvirt_public.syms | 4 + src/qemu/qemu_domain.c | 1 + src/qemu/qemu_domain.h | 3 + src/qemu/qemu_driver.c | 72 ++++++++- src/qemu/qemu_migration.c | 299 +++++++++++++++++++++++++++++++----- src/qemu/qemu_migration.h | 7 +- src/qemu/qemu_monitor.c | 16 +- src/qemu/qemu_monitor.h | 4 + src/qemu/qemu_monitor_json.c | 23 +++ src/qemu/qemu_monitor_json.h | 3 + src/qemu/qemu_process.c | 252 +++++++++++++++++++----------- src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 13 +- src/remote_protocol-structs | 5 + tools/virsh-domain-monitor.c | 7 +- tools/virsh-domain.c | 131 ++++++++++++++-- tools/virsh.pod | 31 +++- 22 files changed, 884 insertions(+), 161 deletions(-) -- 2.7.0
VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY and VIR_DOMAIN_PAUSED_POSTCOPY are used on the source host once migration enters post-copy mode (which means the domain gets paused on the source. After the destination host takes over the execution of the domain, its virtual CPUs are resumed and the domain enters VIR_DOMAIN_RUNNING_POSTCOPY state and VIR_DOMAIN_EVENT_RESUMED_POSTCOPY event is emitted. In case migration fails during post-copy mode and none of the hosts have complete state of the domain, both domains will remain paused with VIR_DOMAIN_PAUSED_POSTCOPY_FAILED reason and an upper layer may decide what to do. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- examples/object-events/event-test.c | 9 +++++++++ include/libvirt/libvirt-domain.h | 7 +++++++ src/conf/domain_conf.c | 7 +++++-- tools/virsh-domain-monitor.c | 7 +++++-- tools/virsh-domain.c | 7 +++++-- 5 files changed, 31 insertions(+), 6 deletions(-) diff --git a/examples/object-events/event-test.c b/examples/object-events/event-test.c index dcae981..70f1186 100644 --- a/examples/object-events/event-test.c +++ b/examples/object-events/event-test.c @@ -159,6 +159,12 @@ static const char *eventDetailToString(int event, int detail) { case VIR_DOMAIN_EVENT_SUSPENDED_API_ERROR: ret = "API error"; break; + case VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY: + ret = "Post-copy"; + break; + case VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED: + ret = "Post-copy Error"; + break; } break; case VIR_DOMAIN_EVENT_RESUMED: @@ -172,6 +178,9 @@ static const char *eventDetailToString(int event, int detail) { case VIR_DOMAIN_EVENT_RESUMED_FROM_SNAPSHOT: ret = "Snapshot"; break; + case VIR_DOMAIN_EVENT_RESUMED_POSTCOPY: + ret = "Post-copy"; + break; } break; case VIR_DOMAIN_EVENT_STOPPED: diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 65f1618..50342d2 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -90,6 +90,7 @@ typedef enum { VIR_DOMAIN_RUNNING_WAKEUP = 8, /* returned from pmsuspended due to wakeup event */ VIR_DOMAIN_RUNNING_CRASHED = 9, /* resumed from crashed */ + VIR_DOMAIN_RUNNING_POSTCOPY = 10, /* running in post-copy migration mode */ # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_RUNNING_LAST @@ -117,6 +118,8 @@ typedef enum { VIR_DOMAIN_PAUSED_SNAPSHOT = 9, /* paused while creating a snapshot */ VIR_DOMAIN_PAUSED_CRASHED = 10, /* paused due to a guest crash */ VIR_DOMAIN_PAUSED_STARTING_UP = 11, /* the domain is being started */ + VIR_DOMAIN_PAUSED_POSTCOPY = 12, /* paused for post-copy migration */ + VIR_DOMAIN_PAUSED_POSTCOPY_FAILED = 13, /* paused after failed post-copy */ # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_PAUSED_LAST @@ -2380,6 +2383,8 @@ typedef enum { VIR_DOMAIN_EVENT_SUSPENDED_RESTORED = 4, /* Restored from paused state file */ VIR_DOMAIN_EVENT_SUSPENDED_FROM_SNAPSHOT = 5, /* Restored from paused snapshot */ VIR_DOMAIN_EVENT_SUSPENDED_API_ERROR = 6, /* suspended after failure during libvirt API call */ + VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY = 7, /* suspended for post-copy migration */ + VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED = 8, /* suspended after failed post-copy */ # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_EVENT_SUSPENDED_LAST @@ -2395,6 +2400,8 @@ typedef enum { VIR_DOMAIN_EVENT_RESUMED_UNPAUSED = 0, /* Normal resume due to admin unpause */ VIR_DOMAIN_EVENT_RESUMED_MIGRATED = 1, /* Resumed for completion of migration */ VIR_DOMAIN_EVENT_RESUMED_FROM_SNAPSHOT = 2, /* Resumed from snapshot */ + VIR_DOMAIN_EVENT_RESUMED_POSTCOPY = 3, /* Resumed, but migration is still + running in post-copy mode */ # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_EVENT_RESUMED_LAST diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index a9706b0..3814d40 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -675,7 +675,8 @@ VIR_ENUM_IMPL(virDomainRunningReason, VIR_DOMAIN_RUNNING_LAST, "migration canceled", "save canceled", "wakeup", - "crashed") + "crashed", + "post-copy") VIR_ENUM_IMPL(virDomainBlockedReason, VIR_DOMAIN_BLOCKED_LAST, "unknown") @@ -692,7 +693,9 @@ VIR_ENUM_IMPL(virDomainPausedReason, VIR_DOMAIN_PAUSED_LAST, "shutdown", "snapshot", "panicked", - "starting up") + "starting up", + "post-copy", + "post-copy failed") VIR_ENUM_IMPL(virDomainShutdownReason, VIR_DOMAIN_SHUTDOWN_LAST, "unknown", diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index 7eb387d..22fe97b 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -182,7 +182,8 @@ VIR_ENUM_IMPL(virshDomainRunningReason, N_("migration canceled"), N_("save canceled"), N_("event wakeup"), - N_("crashed")) + N_("crashed"), + N_("post-copy")) VIR_ENUM_DECL(virshDomainBlockedReason) VIR_ENUM_IMPL(virshDomainBlockedReason, @@ -203,7 +204,9 @@ VIR_ENUM_IMPL(virshDomainPausedReason, N_("shutting down"), N_("creating snapshot"), N_("crashed"), - N_("starting up")) + N_("starting up"), + N_("post-copy"), + N_("post-copy failed")) VIR_ENUM_DECL(virshDomainShutdownReason) VIR_ENUM_IMPL(virshDomainShutdownReason, diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index 6d04b71..d1699cb 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -11400,14 +11400,17 @@ VIR_ENUM_IMPL(virshDomainEventSuspended, N_("Watchdog"), N_("Restored"), N_("Snapshot"), - N_("API error")) + N_("API error"), + N_("Post-copy"), + N_("Post-copy Error")) VIR_ENUM_DECL(virshDomainEventResumed) VIR_ENUM_IMPL(virshDomainEventResumed, VIR_DOMAIN_EVENT_RESUMED_LAST, N_("Unpaused"), N_("Migrated"), - N_("Snapshot")) + N_("Snapshot"), + N_("Post-copy")) VIR_ENUM_DECL(virshDomainEventStopped) VIR_ENUM_IMPL(virshDomainEventStopped, -- 2.7.0
From: Cristian Klein <cristiklein@gmail.com> To use post-copy one has to start the migration with VIR_MIGRATE_POSTCOPY flag and VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY or, while migration is in progress, call virDomainMigrateStartPostCopy() to switch from pre-copy to post-copy. Signed-off-by: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 5 ++ src/driver-hypervisor.h | 5 ++ src/libvirt-domain.c | 140 +++++++++++++++++++++++++++++++++++++++ src/libvirt_public.syms | 4 ++ src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 13 +++- src/remote_protocol-structs | 5 ++ 7 files changed, 172 insertions(+), 1 deletion(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 50342d2..221d1e3 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -662,6 +662,8 @@ typedef enum { VIR_MIGRATE_ABORT_ON_ERROR = (1 << 12), /* abort migration on I/O errors happened during migration */ VIR_MIGRATE_AUTO_CONVERGE = (1 << 13), /* force convergence */ VIR_MIGRATE_RDMA_PIN_ALL = (1 << 14), /* RDMA memory pinning */ + VIR_MIGRATE_POSTCOPY = (1 << 15), /* enable (but do not start) post-copy migration */ + VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY = (1 << 16), /* switch to post-copy after the first pre-copy iteration */ } virDomainMigrateFlags; @@ -810,6 +812,9 @@ int virDomainMigrateGetMaxSpeed(virDomainPtr domain, unsigned long *bandwidth, unsigned int flags); +int virDomainMigrateStartPostCopy(virDomainPtr domain, + unsigned int flags); + char * virConnectGetDomainCapabilities(virConnectPtr conn, const char *emulatorbin, const char *arch, diff --git a/src/driver-hypervisor.h b/src/driver-hypervisor.h index ae2ec4d..68a7730 100644 --- a/src/driver-hypervisor.h +++ b/src/driver-hypervisor.h @@ -638,6 +638,10 @@ typedef int const char *dom_xml); typedef int +(*virDrvDomainMigrateStartPostCopy)(virDomainPtr domain, + unsigned int flags); + +typedef int (*virDrvConnectIsEncrypted)(virConnectPtr conn); typedef int @@ -1443,6 +1447,7 @@ struct _virHypervisorDriver { virDrvDomainGetFSInfo domainGetFSInfo; virDrvDomainInterfaceAddresses domainInterfaceAddresses; virDrvDomainSetUserPassword domainSetUserPassword; + virDrvDomainMigrateStartPostCopy domainMigrateStartPostCopy; }; diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 9491845..a822113 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -3537,6 +3537,9 @@ virDomainMigrateUnmanaged(virDomainPtr domain, * automatically when supported). * VIR_MIGRATE_UNSAFE Force migration even if it is considered unsafe. * VIR_MIGRATE_OFFLINE Migrate offline + * VIR_MIGRATE_POSTCOPY Enable (but do not start) post-copy + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY Switch to post-copy after the first + * pre-copy iteration. * * VIR_MIGRATE_TUNNELLED requires that VIR_MIGRATE_PEER2PEER be set. * Applications using the VIR_MIGRATE_PEER2PEER flag will probably @@ -3573,6 +3576,14 @@ virDomainMigrateUnmanaged(virDomainPtr domain, * not support this feature and will return an error if bandwidth * is not 0. * + * Enabling the VIR_MIGRATE_POSTCOPY flag tells libvirt to enable post-copy + * migration. Without VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag the migration + * will be running normally until virDomainMigrateStartPostCopy is called to + * switch migration into the post-copy mode. When + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is set the switch to post-copy + * mode will happen automatically after the first iteration of pre-copy. + * See virDomainMigrateStartPostCopy for more details about post-copy. + * * To see which features are supported by the current hypervisor, * see virConnectGetCapabilities, /capabilities/host/migration_features. * @@ -3748,6 +3759,9 @@ virDomainMigrate(virDomainPtr domain, * automatically when supported). * VIR_MIGRATE_UNSAFE Force migration even if it is considered unsafe. * VIR_MIGRATE_OFFLINE Migrate offline + * VIR_MIGRATE_POSTCOPY Enable (but do not start) post-copy + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY Switch to post-copy after the first + * pre-copy iteration. * * VIR_MIGRATE_TUNNELLED requires that VIR_MIGRATE_PEER2PEER be set. * Applications using the VIR_MIGRATE_PEER2PEER flag will probably @@ -3784,6 +3798,14 @@ virDomainMigrate(virDomainPtr domain, * not support this feature and will return an error if bandwidth * is not 0. * + * Enabling the VIR_MIGRATE_POSTCOPY flag tells libvirt to enable post-copy + * migration. Without VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag the migration + * will be running normally until virDomainMigrateStartPostCopy is called to + * switch migration into the post-copy mode. When + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is set the switch to post-copy + * mode will happen automatically after the first iteration of pre-copy. + * See virDomainMigrateStartPostCopy for more details about post-copy. + * * To see which features are supported by the current hypervisor, * see virConnectGetCapabilities, /capabilities/host/migration_features. * @@ -3968,6 +3990,14 @@ virDomainMigrate2(virDomainPtr domain, * can use either VIR_MIGRATE_NON_SHARED_DISK or * VIR_MIGRATE_NON_SHARED_INC as they are mutually exclusive. * + * Enabling the VIR_MIGRATE_POSTCOPY flag tells libvirt to enable post-copy + * migration. Without VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag the migration + * will be running normally until virDomainMigrateStartPostCopy is called to + * switch migration into the post-copy mode. When + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is set the switch to post-copy + * mode will happen automatically after the first iteration of pre-copy. + * See virDomainMigrateStartPostCopy for more details about post-copy. + * * There are many limitations on migration imposed by the underlying * technology - for example it may not be possible to migrate between * different processors even with the same architecture, or between @@ -4208,6 +4238,9 @@ int virDomainMigrateUnmanagedCheckCompat(virDomainPtr domain, * automatically when supported). * VIR_MIGRATE_UNSAFE Force migration even if it is considered unsafe. * VIR_MIGRATE_OFFLINE Migrate offline + * VIR_MIGRATE_POSTCOPY Enable (but do not start) post-copy + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY Switch to post-copy after the first + * pre-copy iteration. * * The operation of this API hinges on the VIR_MIGRATE_PEER2PEER flag. * If the VIR_MIGRATE_PEER2PEER flag is NOT set, the duri parameter @@ -4240,6 +4273,14 @@ int virDomainMigrateUnmanagedCheckCompat(virDomainPtr domain, * not support this feature and will return an error if bandwidth * is not 0. * + * Enabling the VIR_MIGRATE_POSTCOPY flag tells libvirt to enable post-copy + * migration. Without VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag the migration + * will be running normally until virDomainMigrateStartPostCopy is called to + * switch migration into the post-copy mode. When + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is set the switch to post-copy + * mode will happen automatically after the first iteration of pre-copy. + * See virDomainMigrateStartPostCopy for more details about post-copy. + * * To see which features are supported by the current hypervisor, * see virConnectGetCapabilities, /capabilities/host/migration_features. * @@ -4321,6 +4362,9 @@ virDomainMigrateToURI(virDomainPtr domain, * automatically when supported). * VIR_MIGRATE_UNSAFE Force migration even if it is considered unsafe. * VIR_MIGRATE_OFFLINE Migrate offline + * VIR_MIGRATE_POSTCOPY Enable (but do not start) post-copy + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY Switch to post-copy after the first + * pre-copy iteration. * * The operation of this API hinges on the VIR_MIGRATE_PEER2PEER flag. * @@ -4366,6 +4410,14 @@ virDomainMigrateToURI(virDomainPtr domain, * not support this feature and will return an error if bandwidth * is not 0. * + * Enabling the VIR_MIGRATE_POSTCOPY flag tells libvirt to enable post-copy + * migration. Without VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag the migration + * will be running normally until virDomainMigrateStartPostCopy is called to + * switch migration into the post-copy mode. When + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is set the switch to post-copy + * mode will happen automatically after the first iteration of pre-copy. + * See virDomainMigrateStartPostCopy for more details about post-copy. + * * To see which features are supported by the current hypervisor, * see virConnectGetCapabilities, /capabilities/host/migration_features. * @@ -4446,6 +4498,14 @@ virDomainMigrateToURI2(virDomainPtr domain, * can use either VIR_MIGRATE_NON_SHARED_DISK or * VIR_MIGRATE_NON_SHARED_INC as they are mutually exclusive. * + * Enabling the VIR_MIGRATE_POSTCOPY flag tells libvirt to enable post-copy + * migration. Without VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag the migration + * will be running normally until virDomainMigrateStartPostCopy is called to + * switch migration into the post-copy mode. When + * VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is set the switch to post-copy + * mode will happen automatically after the first iteration of pre-copy. + * See virDomainMigrateStartPostCopy for more details about post-copy. + * * There are many limitations on migration imposed by the underlying * technology - for example it may not be possible to migrate between * different processors even with the same architecture, or between @@ -9163,6 +9223,86 @@ virDomainMigrateGetMaxSpeed(virDomainPtr domain, /** + * virDomainMigrateStartPostCopy: + * @domain: a domain object + * @flags: extra flags; not used yet, so callers should always pass 0 + * + * Starts post-copy migration. This function has to be called while + * migration (initiated with VIR_MIGRATE_POSTCOPY flag) is in progress. + * + * Traditional post-copy migration iteratively walks through guest memory + * pages and migrates those that changed since the previous iteration. The + * iterative phase stops when the number of dirty pages is low enough so that + * the virtual CPUs can be paused, all dirty pages transferred to the + * destination, where the virtual CPUs are unpaused, and all this can happen + * within a predefined downtime period. It's clear that this process may never + * converge if downtime is too short and/or the guest keeps changing a lot of + * memory pages. + * + * When migration is switched to post-copy mode, the virtual CPUs are paused + * immediately, only a minimum set of pages is transferred, and the CPUs are + * unpaused on destination. The source keeps sending all remaining memory pages + * to the destination while the guest is already running there. Whenever the + * guest tries to read a memory page which has not been migrated yet, the + * hypervisor has to tell the source to transfer that page in a priority + * channel. To minimize such page faults, it is a good idea to run at least one + * iteration of pre-copy migration before switching to post-copy. + * + * Post-copy migration is guaranteed to converge since each page is transferred + * at most once no matter how fast it changes. On the other hand once the + * guest is running on the destination host, the migration can no longer be + * rolled back because none of the hosts has complete state. If this happens, + * libvirt will leave the domain paused on both hosts with + * VIR_DOMAIN_PAUSED_POSTCOPY_FAILED reason. It's up to the upper layer to + * decide what to do in such case. + * + * The following domain life cycle events are emitted during post-copy + * migration: + * VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY (on the source) -- migration entered + * post-copy mode. + * VIR_DOMAIN_EVENT_RESUMED_POSTCOPY (on the destination) -- the guest is + * running on the destination host while some of its memory pages still + * remain on the source host; neither the source nor the destination host + * contain a complete guest state from this point until migration + * finishes. + * VIR_DOMAIN_EVENT_RESUMED_MIGRATED (on the destination), + * VIR_DOMAIN_EVENT_STOPPED_MIGRATED (on the source) -- migration finished + * successfully and the destination host holds a complete guest state. + * VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED (on either side) -- emitted + * when migration fails in post-copy mode and it's unclear whether any + * of the hosts has a complete guest state. + * + * Returns 0 in case of success, -1 otherwise. + */ +int +virDomainMigrateStartPostCopy(virDomainPtr domain, + unsigned int flags) +{ + virConnectPtr conn; + + VIR_DOMAIN_DEBUG(domain); + + virResetLastError(); + + virCheckDomainReturn(domain, -1); + conn = domain->conn; + + virCheckReadOnlyGoto(conn->flags, error); + + if (conn->driver->domainMigrateStartPostCopy) { + if (conn->driver->domainMigrateStartPostCopy(domain, flags) < 0) + goto error; + return 0; + } + + virReportUnsupportedError(); + error: + virDispatchError(conn); + return -1; +} + + +/** * virConnectDomainEventRegisterAny: * @conn: pointer to the connection * @dom: pointer to the domain diff --git a/src/libvirt_public.syms b/src/libvirt_public.syms index dd94191..d98414d 100644 --- a/src/libvirt_public.syms +++ b/src/libvirt_public.syms @@ -725,4 +725,8 @@ LIBVIRT_1.2.19 { virDomainRename; } LIBVIRT_1.2.17; +LIBVIRT_1.3.2 { + virDomainMigrateStartPostCopy; +} LIBVIRT_1.2.19; + # .... define new API here using predicted next version number .... diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c index d9d7ec8..f0a81a6 100644 --- a/src/remote/remote_driver.c +++ b/src/remote/remote_driver.c @@ -8462,6 +8462,7 @@ static virHypervisorDriver hypervisor_driver = { .domainInterfaceAddresses = remoteDomainInterfaceAddresses, /* 1.2.14 */ .domainSetUserPassword = remoteDomainSetUserPassword, /* 1.2.16 */ .domainRename = remoteDomainRename, /* 1.2.19 */ + .domainMigrateStartPostCopy = remoteDomainMigrateStartPostCopy, /* 1.3.2 */ }; static virNetworkDriver network_driver = { diff --git a/src/remote/remote_protocol.x b/src/remote/remote_protocol.x index bfdbce7..1e1b301 100644 --- a/src/remote/remote_protocol.x +++ b/src/remote/remote_protocol.x @@ -3228,6 +3228,11 @@ struct remote_domain_event_callback_migration_iteration_msg { int iteration; }; +struct remote_domain_migrate_start_post_copy_args { + remote_nonnull_domain dom; + unsigned int flags; +}; + /*----- Protocol. -----*/ /* Define the program number, protocol version and procedure numbers here. */ @@ -5706,5 +5711,11 @@ enum remote_procedure { * @generate: both * @acl: none */ - REMOTE_PROC_DOMAIN_EVENT_CALLBACK_MIGRATION_ITERATION = 359 + REMOTE_PROC_DOMAIN_EVENT_CALLBACK_MIGRATION_ITERATION = 359, + + /** + * @generate: both + * @acl: domain:migrate + */ + REMOTE_PROC_DOMAIN_MIGRATE_START_POST_COPY = 360 }; diff --git a/src/remote_protocol-structs b/src/remote_protocol-structs index dff54e8..b372f49 100644 --- a/src/remote_protocol-structs +++ b/src/remote_protocol-structs @@ -2697,6 +2697,10 @@ struct remote_domain_event_callback_migration_iteration_msg { remote_nonnull_domain dom; int iteration; }; +struct remote_domain_migrate_start_post_copy_args { + remote_nonnull_domain dom; + u_int flags; +}; enum remote_procedure { REMOTE_PROC_CONNECT_OPEN = 1, REMOTE_PROC_CONNECT_CLOSE = 2, @@ -3057,4 +3061,5 @@ enum remote_procedure { REMOTE_PROC_DOMAIN_SET_USER_PASSWORD = 357, REMOTE_PROC_DOMAIN_RENAME = 358, REMOTE_PROC_DOMAIN_EVENT_CALLBACK_MIGRATION_ITERATION = 359, + REMOTE_PROC_DOMAIN_MIGRATE_START_POST_COPY = 360, }; -- 2.7.0
On Tue, Jan 19, 2016 at 10:50:07AM +0100, Jiri Denemark wrote:
From: Cristian Klein <cristiklein@gmail.com>
To use post-copy one has to start the migration with VIR_MIGRATE_POSTCOPY flag and VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY or, while migration is in progress, call virDomainMigrateStartPostCopy() to switch from pre-copy to post-copy.
The VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is something that feels a little wrong to me. It is not something QEMU directly supports, but rather it is implementing a very specific usage policy in libvirt. I can see it could be considered convenient, but we have always tried to stay away from implementing specific policies in libvirt since inevitably they will not serve everyone else. Since we have ability to report to apps how many iterations we've done, and apps generally monitor migration progress anyway (even virsh does), it doesn't seem like it is hard for apps to make the decision to switch to postcopy and avoid this policy in libvirt Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Jan 19, 2016 at 10:23:40 +0000, Daniel P. Berrange wrote:
On Tue, Jan 19, 2016 at 10:50:07AM +0100, Jiri Denemark wrote:
From: Cristian Klein <cristiklein@gmail.com>
To use post-copy one has to start the migration with VIR_MIGRATE_POSTCOPY flag and VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY or, while migration is in progress, call virDomainMigrateStartPostCopy() to switch from pre-copy to post-copy.
The VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY flag is something that feels a little wrong to me. It is not something QEMU directly supports, but rather it is implementing a very specific usage policy in libvirt. I can see it could be considered convenient, but we have always tried to stay away from implementing specific policies in libvirt since inevitably they will not serve everyone else. Since we have ability to report to apps how many iterations we've done, and apps generally monitor migration progress anyway (even virsh does), it doesn't seem like it is hard for apps to make the decision to switch to postcopy and avoid this policy in libvirt
Yeah, I wasn't completely convinced this is a good idea either :-) It was just easier to implement in libvirt rather than doing the same in virsh or any other client. But it shouldn't be too hard to do it there either. The patches affected by this change will be 2/16 (only slightly), 9/16 (will be removed), 12/16 (will implement --postcopy-after-precopy in a client). Jirka
With a very old QEMU which doesn't support events we need to explicitly call qemuMigrationSetOffline at the end of migration to update our internal state. On the other hand, if we talk to QEMU using QMP, we should just wait for the STOP event and let the event handler update the state and trigger a libvirt event. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domain.h | 2 ++ src/qemu/qemu_migration.c | 27 ++++++++++++++++----------- src/qemu/qemu_process.c | 20 +++++++++++++++----- 3 files changed, 33 insertions(+), 16 deletions(-) diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 7fc4fff..9fd9076 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -203,6 +203,8 @@ struct _qemuDomainObjPrivate { bool signalIOError; /* true if the domain condition should be signalled on I/O error */ + bool signalStop; /* true if the domain condition should be signalled on + QMP STOP event */ }; # define QEMU_DOMAIN_DISK_PRIVATE(disk) \ diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 51e7125..c927888 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4563,19 +4563,24 @@ qemuMigrationRun(virQEMUDriverPtr driver, else if (rc == -1) goto cleanup; - /* When migration completed, QEMU will have paused the - * CPUs for us, but unless we're using the JSON monitor - * we won't have been notified of this, so might still - * think we're running. For v2 protocol this doesn't - * matter because we'll kill the VM soon, but for v3 - * this is important because we stay paused until the - * confirm3 step, but need to release the lock state + /* When migration completed, QEMU will have paused the CPUs for us. + * Wait for the STOP event to be processed or explicitly stop CPUs + * (for old QEMU which does not send events) to release the lock state. */ - if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) { - if (qemuMigrationSetOffline(driver, vm) < 0) { - priv->job.current->type = VIR_DOMAIN_JOB_FAILED; - goto cleanup; + if (priv->monJSON) { + while (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) { + priv->signalStop = true; + rc = virDomainObjWait(vm); + priv->signalStop = false; + if (rc < 0) { + priv->job.current->type = VIR_DOMAIN_JOB_FAILED; + goto cleanup; + } } + } else if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING && + qemuMigrationSetOffline(driver, vm) < 0) { + priv->job.current->type = VIR_DOMAIN_JOB_FAILED; + goto cleanup; } ret = 0; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index d465b4f..cab1aee 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -701,6 +701,8 @@ qemuProcessHandleStop(qemuMonitorPtr mon ATTRIBUTE_UNUSED, { virQEMUDriverPtr driver = opaque; virObjectEventPtr event = NULL; + virDomainPausedReason reason = VIR_DOMAIN_PAUSED_UNKNOWN; + virDomainEventSuspendedDetailType detail = VIR_DOMAIN_EVENT_SUSPENDED_PAUSED; virQEMUDriverConfigPtr cfg = virQEMUDriverGetConfig(driver); virObjectLock(vm); @@ -712,16 +714,24 @@ qemuProcessHandleStop(qemuMonitorPtr mon ATTRIBUTE_UNUSED, goto unlock; } - VIR_DEBUG("Transitioned guest %s to paused state", - vm->def->name); + if (priv->job.asyncJob == QEMU_ASYNC_JOB_MIGRATION_OUT) { + reason = VIR_DOMAIN_PAUSED_MIGRATION; + detail = VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED; + } + + VIR_DEBUG("Transitioned guest %s to paused state, reason %s", + vm->def->name, virDomainPausedReasonTypeToString(reason)); if (priv->job.current) ignore_value(virTimeMillisNow(&priv->job.current->stopped)); - virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, VIR_DOMAIN_PAUSED_UNKNOWN); + if (priv->signalStop) + virDomainObjBroadcast(vm); + + virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, reason); event = virDomainEventLifecycleNewFromObj(vm, - VIR_DOMAIN_EVENT_SUSPENDED, - VIR_DOMAIN_EVENT_SUSPENDED_PAUSED); + VIR_DOMAIN_EVENT_SUSPENDED, + detail); VIR_FREE(priv->lockState); if (virDomainLockProcessPause(driver->lockManager, vm, &priv->lockState) < 0) -- 2.7.0
Migration enters "postcopy-active" state after QEMU switches to post-copy and pauses guest CPUs. From libvirt's point of view this state is similar to "completed" because we need to transfer guest execution to the destination host. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 84 +++++++++++++++++++++++++++++++++++++------- src/qemu/qemu_monitor.c | 2 +- src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 1 + src/qemu/qemu_process.c | 10 ++++-- 5 files changed, 82 insertions(+), 16 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c927888..6c98c5c 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2549,6 +2549,7 @@ qemuMigrationUpdateJobType(qemuDomainJobInfoPtr jobInfo) case QEMU_MONITOR_MIGRATION_STATUS_SETUP: case QEMU_MONITOR_MIGRATION_STATUS_ACTIVE: + case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY: case QEMU_MONITOR_MIGRATION_STATUS_CANCELLING: case QEMU_MONITOR_MIGRATION_STATUS_LAST: break; @@ -2666,6 +2667,7 @@ enum qemuMigrationCompletedFlags { QEMU_MIGRATION_COMPLETED_ABORT_ON_ERROR = (1 << 0), QEMU_MIGRATION_COMPLETED_CHECK_STORAGE = (1 << 1), QEMU_MIGRATION_COMPLETED_UPDATE_STATS = (1 << 2), + QEMU_MIGRATION_COMPLETED_POSTCOPY = (1 << 3), }; /** @@ -2707,6 +2709,20 @@ qemuMigrationCompleted(virQEMUDriverPtr driver, goto error; } + /* In case of postcopy the source considers migration completed at the + * moment it switched from active to postcopy-active state. The destination + * will continue waiting until the migrate state changes to completed. + */ + if (flags & QEMU_MIGRATION_COMPLETED_POSTCOPY && + jobInfo->type == VIR_DOMAIN_JOB_UNBOUNDED && + jobInfo->stats.status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY) { + VIR_DEBUG("Migration switched to post-copy"); + if (updateStats && + qemuMigrationUpdateJobStatus(driver, vm, asyncJob) < 0) + goto error; + return 1; + } + if (jobInfo->type == VIR_DOMAIN_JOB_COMPLETED) return 1; else @@ -2740,9 +2756,11 @@ qemuMigrationWaitForCompletion(virQEMUDriverPtr driver, qemuDomainObjPrivatePtr priv = vm->privateData; qemuDomainJobInfoPtr jobInfo = priv->job.current; bool events = virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); - unsigned int flags = QEMU_MIGRATION_COMPLETED_UPDATE_STATS; + unsigned int flags; int rv; + flags = QEMU_MIGRATION_COMPLETED_UPDATE_STATS | + QEMU_MIGRATION_COMPLETED_POSTCOPY; if (abort_on_error) flags |= QEMU_MIGRATION_COMPLETED_ABORT_ON_ERROR; if (storage) @@ -2781,9 +2799,11 @@ qemuMigrationWaitForCompletion(virQEMUDriverPtr driver, static int qemuMigrationWaitForDestCompletion(virQEMUDriverPtr driver, virDomainObjPtr vm, - qemuDomainAsyncJob asyncJob) + qemuDomainAsyncJob asyncJob, + bool postcopy) { qemuDomainObjPrivatePtr priv = vm->privateData; + unsigned int flags = 0; int rv; if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) @@ -2791,7 +2811,11 @@ qemuMigrationWaitForDestCompletion(virQEMUDriverPtr driver, VIR_DEBUG("Waiting for incoming migration to complete"); - while ((rv = qemuMigrationCompleted(driver, vm, asyncJob, NULL, 0)) != 1) { + if (postcopy) + flags = QEMU_MIGRATION_COMPLETED_POSTCOPY; + + while ((rv = qemuMigrationCompleted(driver, vm, asyncJob, + NULL, flags)) != 1) { if (rv < 0 || virDomainObjWait(vm) < 0) return -1; } @@ -2992,7 +3016,7 @@ qemuMigrationRunIncoming(virQEMUDriverPtr driver, goto cleanup; } - if (qemuMigrationWaitForDestCompletion(driver, vm, asyncJob) < 0) + if (qemuMigrationWaitForDestCompletion(driver, vm, asyncJob, false) < 0) goto cleanup; ret = 0; @@ -4378,6 +4402,7 @@ qemuMigrationRun(virQEMUDriverPtr driver, unsigned int cookieFlags = 0; bool abort_on_error = !!(flags & VIR_MIGRATE_ABORT_ON_ERROR); bool events = virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); + bool inPostCopy = false; int rc; VIR_DEBUG("driver=%p, vm=%p, cookiein=%s, cookieinlen=%d, " @@ -4563,6 +4588,9 @@ qemuMigrationRun(virQEMUDriverPtr driver, else if (rc == -1) goto cleanup; + if (priv->job.current->stats.status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY) + inPostCopy = true; + /* When migration completed, QEMU will have paused the CPUs for us. * Wait for the STOP event to be processed or explicitly stop CPUs * (for old QEMU which does not send events) to release the lock state. @@ -4572,15 +4600,12 @@ qemuMigrationRun(virQEMUDriverPtr driver, priv->signalStop = true; rc = virDomainObjWait(vm); priv->signalStop = false; - if (rc < 0) { - priv->job.current->type = VIR_DOMAIN_JOB_FAILED; - goto cleanup; - } + if (rc < 0) + goto cancelPostCopy; } } else if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING && qemuMigrationSetOffline(driver, vm) < 0) { - priv->job.current->type = VIR_DOMAIN_JOB_FAILED; - goto cleanup; + goto cancelPostCopy; } ret = 0; @@ -4609,7 +4634,7 @@ qemuMigrationRun(virQEMUDriverPtr driver, ignore_value(virTimeMillisNow(&priv->job.completed->sent)); } - if (priv->job.current->type == VIR_DOMAIN_JOB_UNBOUNDED) + if (priv->job.current->type == VIR_DOMAIN_JOB_UNBOUNDED && !inPostCopy) priv->job.current->type = VIR_DOMAIN_JOB_FAILED; cookieFlags |= QEMU_MIGRATION_COOKIE_NETWORK | @@ -4649,6 +4674,13 @@ qemuMigrationRun(virQEMUDriverPtr driver, } } goto cleanup; + + cancelPostCopy: + priv->job.current->type = VIR_DOMAIN_JOB_FAILED; + if (inPostCopy) + goto cancel; + else + goto cleanup; } /* Perform migration using QEMU's native migrate support, @@ -5800,6 +5832,7 @@ qemuMigrationFinish(virQEMUDriverPtr driver, unsigned short port; unsigned long long timeReceived = 0; virObjectEventPtr event; + bool inPostCopy = false; VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " "cookieout=%p, cookieoutlen=%p, flags=%lx, retcode=%d", @@ -5891,7 +5924,8 @@ qemuMigrationFinish(virQEMUDriverPtr driver, * before starting guest CPUs. */ if (qemuMigrationWaitForDestCompletion(driver, vm, - QEMU_ASYNC_JOB_MIGRATION_IN) < 0) { + QEMU_ASYNC_JOB_MIGRATION_IN, + !!(flags & VIR_MIGRATE_POSTCOPY)) < 0) { /* There's not much we can do for v2 protocol since the * original domain on the source host is already gone. */ @@ -5899,13 +5933,17 @@ qemuMigrationFinish(virQEMUDriverPtr driver, goto endjob; } + if (priv->job.current->stats.status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY) + inPostCopy = true; + if (!(flags & VIR_MIGRATE_PAUSED)) { /* run 'cont' on the destination, which allows migration on qemu * >= 0.10.6 to work properly. This isn't strictly necessary on * older qemu's, but it also doesn't hurt anything there */ if (qemuProcessStartCPUs(driver, vm, dconn, - VIR_DOMAIN_RUNNING_MIGRATED, + inPostCopy ? VIR_DOMAIN_RUNNING_POSTCOPY + : VIR_DOMAIN_RUNNING_MIGRATED, QEMU_ASYNC_JOB_MIGRATION_IN) < 0) { if (virGetLastError() == NULL) virReportError(VIR_ERR_INTERNAL_ERROR, @@ -5926,6 +5964,13 @@ qemuMigrationFinish(virQEMUDriverPtr driver, if (v3proto) goto endjob; } + + if (inPostCopy) { + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_RESUMED, + VIR_DOMAIN_EVENT_RESUMED_POSTCOPY); + qemuDomainEventQueue(driver, event); + } } if (mig->jobInfo) { @@ -5942,6 +5987,19 @@ qemuMigrationFinish(virQEMUDriverPtr driver, qemuDomainJobInfoUpdateDowntime(priv->job.completed); } + if (inPostCopy) { + if (qemuMigrationWaitForDestCompletion(driver, vm, + QEMU_ASYNC_JOB_MIGRATION_IN, + false) < 0) { + goto endjob; + } + if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) { + virDomainObjSetState(vm, + VIR_DOMAIN_RUNNING, + VIR_DOMAIN_RUNNING_MIGRATED); + } + } + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid); event = virDomainEventLifecycleNewFromObj(vm, diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 6b23e88..edb3310 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -161,7 +161,7 @@ VIR_ONCE_GLOBAL_INIT(qemuMonitor) VIR_ENUM_IMPL(qemuMonitorMigrationStatus, QEMU_MONITOR_MIGRATION_STATUS_LAST, "inactive", "setup", - "active", + "active", "postcopy-active", "completed", "failed", "cancelling", "cancelled") diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 9d7d5f3..420d82c 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -467,6 +467,7 @@ typedef enum { QEMU_MONITOR_MIGRATION_STATUS_INACTIVE, QEMU_MONITOR_MIGRATION_STATUS_SETUP, QEMU_MONITOR_MIGRATION_STATUS_ACTIVE, + QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY, QEMU_MONITOR_MIGRATION_STATUS_COMPLETED, QEMU_MONITOR_MIGRATION_STATUS_ERROR, QEMU_MONITOR_MIGRATION_STATUS_CANCELLING, diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 24a8865..8d71dd4 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -2493,6 +2493,7 @@ qemuMonitorJSONGetMigrationStatsReply(virJSONValuePtr reply, break; case QEMU_MONITOR_MIGRATION_STATUS_ACTIVE: + case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY: case QEMU_MONITOR_MIGRATION_STATUS_COMPLETED: case QEMU_MONITOR_MIGRATION_STATUS_CANCELLING: ram = virJSONValueObjectGetObject(ret, "ram"); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index cab1aee..5ced279 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -715,8 +715,14 @@ qemuProcessHandleStop(qemuMonitorPtr mon ATTRIBUTE_UNUSED, } if (priv->job.asyncJob == QEMU_ASYNC_JOB_MIGRATION_OUT) { - reason = VIR_DOMAIN_PAUSED_MIGRATION; - detail = VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED; + if (priv->job.current->stats.status == + QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY) { + reason = VIR_DOMAIN_PAUSED_POSTCOPY; + detail = VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY; + } else { + reason = VIR_DOMAIN_PAUSED_MIGRATION; + detail = VIR_DOMAIN_EVENT_SUSPENDED_MIGRATED; + } } VIR_DEBUG("Transitioned guest %s to paused state, reason %s", -- 2.7.0
From: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- Notes: The migration capability is called x-postcopy-ram in QEMU, which means it's still considered experimental. The main reason for it is to make sure the feature is complete from libvirt's point of view. The "x-" prefix will be removed once we have a working implementation. src/qemu/qemu_monitor.c | 14 +++++++++++++- src/qemu/qemu_monitor.h | 3 +++ src/qemu/qemu_monitor_json.c | 22 ++++++++++++++++++++++ src/qemu/qemu_monitor_json.h | 3 +++ 4 files changed, 41 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index edb3310..d34878b 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -167,7 +167,8 @@ VIR_ENUM_IMPL(qemuMonitorMigrationStatus, VIR_ENUM_IMPL(qemuMonitorMigrationCaps, QEMU_MONITOR_MIGRATION_CAPS_LAST, - "xbzrle", "auto-converge", "rdma-pin-all", "events") + "xbzrle", "auto-converge", "rdma-pin-all", "events", + "x-postcopy-ram") VIR_ENUM_IMPL(qemuMonitorVMStatus, QEMU_MONITOR_VM_STATUS_LAST, @@ -3817,3 +3818,14 @@ qemuMonitorMigrateIncoming(qemuMonitorPtr mon, return qemuMonitorJSONMigrateIncoming(mon, uri); } + + +int +qemuMonitorMigrateStartPostCopy(qemuMonitorPtr mon) +{ + VIR_DEBUG("mon=%p", mon); + + QEMU_CHECK_MONITOR_JSON(mon); + + return qemuMonitorJSONMigrateStartPostCopy(mon); +} diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 420d82c..a655437 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -526,6 +526,7 @@ typedef enum { QEMU_MONITOR_MIGRATION_CAPS_AUTO_CONVERGE, QEMU_MONITOR_MIGRATION_CAPS_RDMA_PIN_ALL, QEMU_MONITOR_MIGRATION_CAPS_EVENTS, + QEMU_MONITOR_MIGRATION_CAPS_POSTCOPY, QEMU_MONITOR_MIGRATION_CAPS_LAST } qemuMonitorMigrationCaps; @@ -944,6 +945,8 @@ int qemuMonitorGetMemoryDeviceInfo(qemuMonitorPtr mon, int qemuMonitorMigrateIncoming(qemuMonitorPtr mon, const char *uri); +int qemuMonitorMigrateStartPostCopy(qemuMonitorPtr mon); + /** * When running two dd process and using <> redirection, we need a * shell that will not truncate files. These two strings serve that diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 8d71dd4..463ec4e 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -6682,3 +6682,25 @@ qemuMonitorJSONMigrateIncoming(qemuMonitorPtr mon, virJSONValueFree(reply); return ret; } + + +int +qemuMonitorJSONMigrateStartPostCopy(qemuMonitorPtr mon) +{ + int ret = -1; + virJSONValuePtr cmd; + virJSONValuePtr reply = NULL; + + if (!(cmd = qemuMonitorJSONMakeCommand("migrate-start-postcopy", NULL))) + return -1; + + if (qemuMonitorJSONCommand(mon, cmd, &reply) < 0) + goto cleanup; + + ret = qemuMonitorJSONCheckError(cmd, reply); + + cleanup: + virJSONValueFree(cmd); + virJSONValueFree(reply); + return ret; +} diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h index 2c27c6f..99effa4 100644 --- a/src/qemu/qemu_monitor_json.h +++ b/src/qemu/qemu_monitor_json.h @@ -485,4 +485,7 @@ int qemuMonitorJSONMigrateIncoming(qemuMonitorPtr mon, const char *uri) ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2); +int qemuMonitorJSONMigrateStartPostCopy(qemuMonitorPtr mon) + ATTRIBUTE_NONNULL(1); + #endif /* QEMU_MONITOR_JSON_H */ -- 2.7.0
From: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 76 +++++++++++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_migration.h | 3 +- 2 files changed, 78 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 6c98c5c..64c1eb8 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2498,6 +2498,54 @@ qemuMigrationSetPinAll(virQEMUDriverPtr driver, return ret; } + +static int +qemuMigrationSetPostCopy(virQEMUDriverPtr driver, + virDomainObjPtr vm, + bool state, + qemuDomainAsyncJob job) +{ + qemuDomainObjPrivatePtr priv = vm->privateData; + int ret; + + if (qemuDomainObjEnterMonitorAsync(driver, vm, job) < 0) + return -1; + + ret = qemuMonitorGetMigrationCapability( + priv->mon, + QEMU_MONITOR_MIGRATION_CAPS_POSTCOPY); + + if (ret < 0) { + goto cleanup; + } else if (ret == 0 && !state) { + /* Unsupported but we want it off anyway */ + goto cleanup; + } else if (ret == 0) { + if (job == QEMU_ASYNC_JOB_MIGRATION_IN) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("Post-copy migration is not supported by " + "target QEMU binary")); + } else { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("Post-copy migration is not supported by " + "source QEMU binary")); + } + ret = -1; + goto cleanup; + } + + ret = qemuMonitorSetMigrationCapability( + priv->mon, + QEMU_MONITOR_MIGRATION_CAPS_POSTCOPY, + state); + + cleanup: + if (qemuDomainObjExitMonitor(driver, vm) < 0) + ret = -1; + return ret; +} + + static int qemuMigrationWaitForSpice(virDomainObjPtr vm) { @@ -3132,6 +3180,15 @@ qemuMigrationBeginPhase(virQEMUDriverPtr driver, !qemuMigrationIsSafe(vm->def, nmigrate_disks, migrate_disks)) goto cleanup; + if (flags & VIR_MIGRATE_POSTCOPY && + (!(flags & VIR_MIGRATE_LIVE) || + flags & VIR_MIGRATE_PAUSED)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("post-copy migration is not supported with non-live " + "or paused migration")); + goto cleanup; + } + if (flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC)) { bool has_drive_mirror = virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_DRIVE_MIRROR); @@ -3471,6 +3528,15 @@ qemuMigrationPrepareAny(virQEMUDriverPtr driver, cookieFlags = QEMU_MIGRATION_COOKIE_GRAPHICS; } + if (flags & VIR_MIGRATE_POSTCOPY && + (!(flags & VIR_MIGRATE_LIVE) || + flags & VIR_MIGRATE_PAUSED)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("post-copy migration is not supported with non-live " + "or paused migration")); + goto cleanup; + } + if (!(caps = virQEMUDriverGetCapabilities(driver, false))) goto cleanup; @@ -3621,6 +3687,11 @@ qemuMigrationPrepareAny(virQEMUDriverPtr driver, QEMU_ASYNC_JOB_MIGRATION_IN) < 0) goto stopjob; + if (qemuMigrationSetPostCopy(driver, vm, + flags & VIR_MIGRATE_POSTCOPY, + QEMU_ASYNC_JOB_MIGRATION_IN) < 0) + goto stopjob; + if (mig->nbd && flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC) && virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_SERVER)) { @@ -4487,6 +4558,11 @@ qemuMigrationRun(virQEMUDriverPtr driver, QEMU_ASYNC_JOB_MIGRATION_OUT) < 0) goto cleanup; + if (qemuMigrationSetPostCopy(driver, vm, + flags & VIR_MIGRATE_POSTCOPY, + QEMU_ASYNC_JOB_MIGRATION_OUT) < 0) + goto cleanup; + if (qemuDomainObjEnterMonitorAsync(driver, vm, QEMU_ASYNC_JOB_MIGRATION_OUT) < 0) goto cleanup; diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index 2445e13..e30b2ec 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -41,7 +41,8 @@ VIR_MIGRATE_COMPRESSED | \ VIR_MIGRATE_ABORT_ON_ERROR | \ VIR_MIGRATE_AUTO_CONVERGE | \ - VIR_MIGRATE_RDMA_PIN_ALL) + VIR_MIGRATE_RDMA_PIN_ALL | \ + VIR_MIGRATE_POSTCOPY) /* All supported migration parameters and their types. */ # define QEMU_MIGRATION_PARAMETERS \ -- 2.7.0
From: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domain.c | 1 + src/qemu/qemu_domain.h | 1 + src/qemu/qemu_driver.c | 58 +++++++++++++++++++++++++++++++++++++++++++++++ src/qemu/qemu_migration.c | 2 ++ 4 files changed, 62 insertions(+) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 0fa2dbe..15a9aa8 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -188,6 +188,7 @@ qemuDomainObjResetAsyncJob(qemuDomainObjPrivatePtr priv) job->dump_memory_only = false; job->abortJob = false; job->spiceMigrated = false; + job->postcopyEnabled = false; VIR_FREE(job->current); } diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 9fd9076..2bea482 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -138,6 +138,7 @@ struct qemuDomainJobObj { qemuDomainJobInfoPtr completed; /* statistics data of a recently completed job */ bool abortJob; /* abort of the job requested */ bool spiceMigrated; /* spice migration completed */ + bool postcopyEnabled; /* post-copy migration was enabled */ }; typedef void (*qemuDomainCleanupCallback)(virQEMUDriverPtr driver, diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 8ccf68b..c62561b 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -13421,6 +13421,63 @@ qemuDomainMigrateGetMaxSpeed(virDomainPtr dom, } +static int +qemuDomainMigrateStartPostCopy(virDomainPtr dom, + unsigned int flags) +{ + virQEMUDriverPtr driver = dom->conn->privateData; + virDomainObjPtr vm; + qemuDomainObjPrivatePtr priv; + int ret = -1; + + virCheckFlags(0, -1); + + if (!(vm = qemuDomObjFromDomain(dom))) + goto cleanup; + + if (virDomainMigrateStartPostCopyEnsureACL(dom->conn, vm->def) < 0) + goto cleanup; + + if (qemuDomainObjBeginJob(driver, vm, QEMU_JOB_MIGRATION_OP) < 0) + goto cleanup; + + if (!virDomainObjIsActive(vm)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("domain is not running")); + goto endjob; + } + + priv = vm->privateData; + + if (priv->job.asyncJob != QEMU_ASYNC_JOB_MIGRATION_OUT) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("post-copy can only be started while " + "outgoing migration is in progress")); + goto endjob; + } + + if (!priv->job.postcopyEnabled) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("switching to post-copy requires migration to be " + "started with VIR_MIGRATE_POSTCOPY flag")); + goto endjob; + } + + VIR_DEBUG("Starting post-copy"); + qemuDomainObjEnterMonitor(driver, vm); + ret = qemuMonitorMigrateStartPostCopy(priv->mon); + if (qemuDomainObjExitMonitor(driver, vm) < 0) + ret = -1; + + endjob: + qemuDomainObjEndJob(driver, vm); + + cleanup: + virDomainObjEndAPI(&vm); + return ret; +} + + typedef enum { VIR_DISK_CHAIN_NO_ACCESS, VIR_DISK_CHAIN_READ_ONLY, @@ -20222,6 +20279,7 @@ static virHypervisorDriver qemuHypervisorDriver = { .domainInterfaceAddresses = qemuDomainInterfaceAddresses, /* 1.2.14 */ .domainSetUserPassword = qemuDomainSetUserPassword, /* 1.2.16 */ .domainRename = qemuDomainRename, /* 1.2.19 */ + .domainMigrateStartPostCopy = qemuDomainMigrateStartPostCopy, /* 1.3.2 */ }; diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 64c1eb8..c5bd2fc 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2542,6 +2542,8 @@ qemuMigrationSetPostCopy(virQEMUDriverPtr driver, cleanup: if (qemuDomainObjExitMonitor(driver, vm) < 0) ret = -1; + if (ret == 0) + priv->job.postcopyEnabled = state; return ret; } -- 2.7.0
The function already takes two bool arguments and we'd add a third one. Switching to flags makes it a lot easier to read. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 25 +++++++++++++------------ 1 file changed, 13 insertions(+), 12 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c5bd2fc..7ccb0ba 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2800,21 +2800,14 @@ qemuMigrationWaitForCompletion(virQEMUDriverPtr driver, virDomainObjPtr vm, qemuDomainAsyncJob asyncJob, virConnectPtr dconn, - bool abort_on_error, - bool storage) + unsigned int flags) { qemuDomainObjPrivatePtr priv = vm->privateData; qemuDomainJobInfoPtr jobInfo = priv->job.current; bool events = virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); - unsigned int flags; int rv; - flags = QEMU_MIGRATION_COMPLETED_UPDATE_STATS | - QEMU_MIGRATION_COMPLETED_POSTCOPY; - if (abort_on_error) - flags |= QEMU_MIGRATION_COMPLETED_ABORT_ON_ERROR; - if (storage) - flags |= QEMU_MIGRATION_COMPLETED_CHECK_STORAGE; + flags |= QEMU_MIGRATION_COMPLETED_UPDATE_STATS; jobInfo->type = VIR_DOMAIN_JOB_UNBOUNDED; while ((rv = qemuMigrationCompleted(driver, vm, asyncJob, @@ -4476,6 +4469,7 @@ qemuMigrationRun(virQEMUDriverPtr driver, bool abort_on_error = !!(flags & VIR_MIGRATE_ABORT_ON_ERROR); bool events = virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); bool inPostCopy = false; + unsigned int waitFlags; int rc; VIR_DEBUG("driver=%p, vm=%p, cookiein=%s, cookieinlen=%d, " @@ -4658,9 +4652,17 @@ qemuMigrationRun(virQEMUDriverPtr driver, fd = -1; } + waitFlags = 0; + if (abort_on_error) + waitFlags |= QEMU_MIGRATION_COMPLETED_ABORT_ON_ERROR; + if (mig->nbd) + waitFlags |= QEMU_MIGRATION_COMPLETED_CHECK_STORAGE; + if (flags & VIR_MIGRATE_POSTCOPY) + waitFlags |= QEMU_MIGRATION_COMPLETED_POSTCOPY; + rc = qemuMigrationWaitForCompletion(driver, vm, QEMU_ASYNC_JOB_MIGRATION_OUT, - dconn, abort_on_error, !!mig->nbd); + dconn, waitFlags); if (rc == -2) goto cancel; else if (rc == -1) @@ -6270,8 +6272,7 @@ qemuMigrationToFile(virQEMUDriverPtr driver, virDomainObjPtr vm, if (rc < 0) goto cleanup; - rc = qemuMigrationWaitForCompletion(driver, vm, asyncJob, - NULL, false, false); + rc = qemuMigrationWaitForCompletion(driver, vm, asyncJob, NULL, 0); if (rc < 0) { if (rc == -2) { -- 2.7.0
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 16 ++++++++++++++++ src/qemu/qemu_migration.h | 3 ++- src/qemu/qemu_process.c | 3 +++ 3 files changed, 21 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 7ccb0ba..296e54e 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2718,6 +2718,7 @@ enum qemuMigrationCompletedFlags { QEMU_MIGRATION_COMPLETED_CHECK_STORAGE = (1 << 1), QEMU_MIGRATION_COMPLETED_UPDATE_STATS = (1 << 2), QEMU_MIGRATION_COMPLETED_POSTCOPY = (1 << 3), + QEMU_MIGRATION_COMPLETED_POSTCOPY_AUTO = (1 << 4), }; /** @@ -2815,6 +2816,19 @@ qemuMigrationWaitForCompletion(virQEMUDriverPtr driver, if (rv < 0) return rv; + if (flags & QEMU_MIGRATION_COMPLETED_POSTCOPY_AUTO && + priv->job.postcopyEnabled && + jobInfo->stats.ram_iteration > 1) { + flags ^= QEMU_MIGRATION_COMPLETED_POSTCOPY_AUTO; + + VIR_DEBUG("One pre-copy iteration finished; switching to post-copy"); + if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) < 0) + return -2; + rv = qemuMonitorMigrateStartPostCopy(priv->mon); + if (qemuDomainObjExitMonitor(driver, vm) < 0 || rv < 0) + return -2; + } + if (events) { if (virDomainObjWait(vm) < 0) { jobInfo->type = VIR_DOMAIN_JOB_FAILED; @@ -4659,6 +4673,8 @@ qemuMigrationRun(virQEMUDriverPtr driver, waitFlags |= QEMU_MIGRATION_COMPLETED_CHECK_STORAGE; if (flags & VIR_MIGRATE_POSTCOPY) waitFlags |= QEMU_MIGRATION_COMPLETED_POSTCOPY; + if (flags & VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY) + waitFlags |= QEMU_MIGRATION_COMPLETED_POSTCOPY_AUTO; rc = qemuMigrationWaitForCompletion(driver, vm, QEMU_ASYNC_JOB_MIGRATION_OUT, diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index e30b2ec..ccef71b 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -42,7 +42,8 @@ VIR_MIGRATE_ABORT_ON_ERROR | \ VIR_MIGRATE_AUTO_CONVERGE | \ VIR_MIGRATE_RDMA_PIN_ALL | \ - VIR_MIGRATE_POSTCOPY) + VIR_MIGRATE_POSTCOPY | \ + VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY) /* All supported migration parameters and their types. */ # define QEMU_MIGRATION_PARAMETERS \ diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 5ced279..e067c3f 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -1552,6 +1552,9 @@ qemuProcessHandleMigrationPass(qemuMonitorPtr mon ATTRIBUTE_UNUSED, goto cleanup; } + priv->job.current->stats.ram_iteration = pass; + virDomainObjBroadcast(vm); + qemuDomainEventQueue(driver, virDomainEventMigrationIterationNewFromObj(vm, pass)); -- 2.7.0
From: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- tools/virsh-domain.c | 55 ++++++++++++++++++++++++++++++++++++++++++++++++++++ tools/virsh.pod | 11 ++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-) diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index d1699cb..d1d6650 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -9623,6 +9623,10 @@ static const vshCmdOptDef opts_migrate[] = { .type = VSH_OT_BOOL, .help = N_("abort on soft errors during migration") }, + {.name = "postcopy", + .type = VSH_OT_BOOL, + .help = N_("enable post-copy migration; switch to it using migrate-postcopy command") + }, {.name = "migrateuri", .type = VSH_OT_STRING, .help = N_("migration URI, usually can be omitted") @@ -9789,6 +9793,9 @@ doMigrate(void *opaque) if (vshCommandOptBool(cmd, "abort-on-error")) flags |= VIR_MIGRATE_ABORT_ON_ERROR; + if (vshCommandOptBool(cmd, "postcopy")) + flags |= VIR_MIGRATE_POSTCOPY; + if (flags & VIR_MIGRATE_PEER2PEER || vshCommandOptBool(cmd, "direct")) { if (virDomainMigrateToURI3(dom, desturi, params, nparams, flags) == 0) ret = '0'; @@ -10092,6 +10099,48 @@ cmdMigrateGetMaxSpeed(vshControl *ctl, const vshCmd *cmd) } /* + * "migrate-postcopy" command + */ +static const vshCmdInfo info_migrate_postcopy[] = { + {.name = "help", + .data = N_("Switch running migration from pre-copy to post-copy") + }, + {.name = "desc", + .data = N_("Switch running migration from pre-copy to post-copy. " + "The migration must have been started with --postcopy option.") + }, + {.name = NULL} +}; + +static const vshCmdOptDef opts_migrate_postcopy[] = { + {.name = "domain", + .type = VSH_OT_DATA, + .flags = VSH_OFLAG_REQ, + .help = N_("domain name, id or uuid") + }, + {.name = NULL} +}; + +static bool +cmdMigratePostCopy(vshControl *ctl, const vshCmd *cmd) +{ + virDomainPtr dom; + bool ret = false; + + if (!(dom = virshCommandOptDomain(ctl, cmd, NULL))) + return false; + + if (virDomainMigrateStartPostCopy(dom, 0) < 0) + goto cleanup; + + ret = true; + + cleanup: + virDomainFree(dom); + return ret; +} + +/* * "domdisplay" command */ static const vshCmdInfo info_domdisplay[] = { @@ -12902,6 +12951,12 @@ const vshCmdDef domManagementCmds[] = { .info = info_migrate_getspeed, .flags = 0 }, + {.name = "migrate-postcopy", + .handler = cmdMigratePostCopy, + .opts = opts_migrate_postcopy, + .info = info_migrate_postcopy, + .flags = 0 + }, {.name = "numatune", .handler = cmdNumatune, .opts = opts_numatune, diff --git a/tools/virsh.pod b/tools/virsh.pod index e830c59..109c608 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -1529,7 +1529,7 @@ to the I<uri> namespace is displayed instead of being modified. =item B<migrate> [I<--live>] [I<--offline>] [I<--direct>] [I<--p2p> [I<--tunnelled>]] [I<--persistent>] [I<--undefinesource>] [I<--suspend>] [I<--copy-storage-all>] [I<--copy-storage-inc>] [I<--change-protection>] [I<--unsafe>] [I<--verbose>] -[I<--compressed>] [I<--abort-on-error>] [I<--auto-converge>] +[I<--compressed>] [I<--abort-on-error>] [I<--auto-converge>] [I<--postcopy>] I<domain> I<desturi> [I<migrateuri>] [I<graphicsuri>] [I<listen-address>] [I<dname>] [I<--timeout> B<seconds>] [I<--xml> B<file>] [I<--migrate-disks> B<disk-list>] @@ -1559,6 +1559,10 @@ of migration. I<--compressed> activates compression of memory pages that have to be transferred repeatedly during live migration. I<--abort-on-error> cancels the migration if a soft error (for example I/O error) happens during the migration. I<--auto-converge> forces convergence during live migration. +I<--postcopy> enables post-copy logic in migration, but does not +actually start post-copy, i.e., migration is started in pre-copy mode. +Once migration is running, the user may switch to post-copy using the +B<migrate-postcopy> command sent from another virsh instance. B<Note>: Individual hypervisors usually do not support all possible types of migration. For example, QEMU does not support direct migration. @@ -1689,6 +1693,11 @@ reject the value or convert it to the maximum value allowed. Get the maximum migration bandwidth (in MiB/s) for a domain. +=item B<migrate-postcopy> I<domain> + +Switch the current migration from pre-copy to post-copy. This is only +supported for a migration started with I<--postcopy> option. + =item B<numatune> I<domain> [I<--mode> B<mode>] [I<--nodeset> B<nodeset>] [[I<--config>] [I<--live>] | [I<--current>]] -- 2.7.0
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- tools/virsh-domain.c | 63 +++++++++++++++++++++++++++++++++++++++++++++------- tools/virsh.pod | 15 ++++++++----- 2 files changed, 65 insertions(+), 13 deletions(-) diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index d1d6650..55ad342 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -9645,7 +9645,16 @@ static const vshCmdOptDef opts_migrate[] = { }, {.name = "timeout", .type = VSH_OT_INT, - .help = N_("force guest to suspend if live migration exceeds timeout (in seconds)") + .help = N_("run action specified by --timeout-* option (suspend by " + "default) if live migration exceeds timeout (in seconds)") + }, + {.name = "timeout-suspend", + .type = VSH_OT_BOOL, + .help = N_("suspend the guest after timeout") + }, + {.name = "timeout-postcopy", + .type = VSH_OT_BOOL, + .help = N_("switch to post-copy after timeout") }, {.name = "xml", .type = VSH_OT_STRING, @@ -9823,14 +9832,35 @@ doMigrate(void *opaque) goto out; } +typedef enum { + VIRSH_MIGRATE_TIMEOUT_DEFAULT, + VIRSH_MIGRATE_TIMEOUT_SUSPEND, + VIRSH_MIGRATE_TIMEOUT_POSTCOPY, +} virshMigrateTimeoutAction; + static void -virshMigrationTimeout(vshControl *ctl, - virDomainPtr dom, - void *opaque ATTRIBUTE_UNUSED) +virshMigrateTimeout(vshControl *ctl, + virDomainPtr dom, + void *opaque) { - vshDebug(ctl, VSH_ERR_DEBUG, "suspending the domain, " - "since migration timed out\n"); - virDomainSuspend(dom); + virshMigrateTimeoutAction action = *(virshMigrateTimeoutAction *) opaque; + + switch (action) { + case VIRSH_MIGRATE_TIMEOUT_DEFAULT: /* unreachable */ + case VIRSH_MIGRATE_TIMEOUT_SUSPEND: + vshDebug(ctl, VSH_ERR_DEBUG, + "migration timed out; suspending domain\n"); + if (virDomainSuspend(dom) < 0) + vshDebug(ctl, VSH_ERR_INFO, "suspending domain failed\n"); + break; + + case VIRSH_MIGRATE_TIMEOUT_POSTCOPY: + vshDebug(ctl, VSH_ERR_DEBUG, + "migration timed out; switching to post-copy\n"); + if (virDomainMigrateStartPostCopy(dom, 0) < 0) + vshDebug(ctl, VSH_ERR_INFO, "switching to post-copy failed\n"); + break; + } } static bool @@ -9842,9 +9872,12 @@ cmdMigrate(vshControl *ctl, const vshCmd *cmd) bool verbose = false; bool functionReturn = false; int timeout = 0; + virshMigrateTimeoutAction timeoutAction = VIRSH_MIGRATE_TIMEOUT_DEFAULT; bool live_flag = false; virshCtrlData data = { .dconn = NULL }; + VSH_EXCLUSIVE_OPTIONS("timeout-suspend", "timeout-postcopy"); + if (!(dom = virshCommandOptDomain(ctl, cmd, NULL))) return false; @@ -9861,6 +9894,19 @@ cmdMigrate(vshControl *ctl, const vshCmd *cmd) goto cleanup; } + if (vshCommandOptBool(cmd, "timeout-suspend")) + timeoutAction = VIRSH_MIGRATE_TIMEOUT_SUSPEND; + if (vshCommandOptBool(cmd, "timeout-postcopy")) + timeoutAction = VIRSH_MIGRATE_TIMEOUT_POSTCOPY; + if (timeout > 0) { + if (timeoutAction == VIRSH_MIGRATE_TIMEOUT_DEFAULT) + timeoutAction = VIRSH_MIGRATE_TIMEOUT_SUSPEND; + } else if (timeoutAction) { + vshError(ctl, "%s", + _("migrate: Unexpected --timeout-* option without --timeout")); + goto cleanup; + } + if (pipe(p) < 0) goto cleanup; @@ -9891,7 +9937,8 @@ cmdMigrate(vshControl *ctl, const vshCmd *cmd) &data) < 0) goto cleanup; functionReturn = virshWatchJob(ctl, dom, verbose, p[0], timeout, - virshMigrationTimeout, NULL, _("Migration")); + virshMigrateTimeout, + &timeoutAction, _("Migration")); virThreadJoin(&workerThread); diff --git a/tools/virsh.pod b/tools/virsh.pod index 109c608..64b4287 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -1531,8 +1531,8 @@ to the I<uri> namespace is displayed instead of being modified. [I<--copy-storage-inc>] [I<--change-protection>] [I<--unsafe>] [I<--verbose>] [I<--compressed>] [I<--abort-on-error>] [I<--auto-converge>] [I<--postcopy>] I<domain> I<desturi> [I<migrateuri>] [I<graphicsuri>] [I<listen-address>] -[I<dname>] [I<--timeout> B<seconds>] [I<--xml> B<file>] -[I<--migrate-disks> B<disk-list>] +[I<dname>] [I<--timeout> B<seconds> [I<--timeout-suspend> | I<--timeout-postcopy>]] +[I<--xml> B<file>] [I<--migrate-disks> B<disk-list>] Migrate domain to another host. Add I<--live> for live migration; <--p2p> for peer-2-peer migration; I<--direct> for direct migration; or I<--tunnelled> @@ -1582,9 +1582,14 @@ the destination to supply a larger set of changes to any host-specific portions of the domain XML, such as accounting for naming differences between source and destination in accessing underlying storage. -I<--timeout> B<seconds> forces guest to suspend when live migration exceeds -that many seconds, and -then the migration will complete offline. It can only be used with I<--live>. +I<--timeout> B<seconds> tells virsh to run a specified action when live +migration exceeds that many seconds. It can only be used with I<--live>. +If I<--timeout-suspend> is specified, the domain will be suspended after +the timeout and the migration will complete offline; this is the default +if no I<--timeout-*> option is specified on the command line. When +I<--timeout-postcopy> is used, virsh will switch migration from pre-copy +to post-copy upon timeout; migration has to be started with I<--postcopy> +option for this to work. Running migration can be canceled by interrupting virsh (usually using C<Ctrl-C>) or by B<domjobabort> command sent from another virsh instance. -- 2.7.0
From: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Cristian Klein <cristiklein@gmail.com> Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- tools/virsh-domain.c | 6 ++++++ tools/virsh.pod | 9 ++++++--- 2 files changed, 12 insertions(+), 3 deletions(-) diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index 55ad342..d9db9fa 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -9627,6 +9627,10 @@ static const vshCmdOptDef opts_migrate[] = { .type = VSH_OT_BOOL, .help = N_("enable post-copy migration; switch to it using migrate-postcopy command") }, + {.name = "postcopy-after-precopy", + .type = VSH_OT_BOOL, + .help = N_("automatically switch to post-copy migration after one pass of pre-copy") + }, {.name = "migrateuri", .type = VSH_OT_STRING, .help = N_("migration URI, usually can be omitted") @@ -9804,6 +9808,8 @@ doMigrate(void *opaque) if (vshCommandOptBool(cmd, "postcopy")) flags |= VIR_MIGRATE_POSTCOPY; + if (vshCommandOptBool(cmd, "postcopy-after-precopy")) + flags |= VIR_MIGRATE_POSTCOPY_AFTER_PRECOPY; if (flags & VIR_MIGRATE_PEER2PEER || vshCommandOptBool(cmd, "direct")) { if (virDomainMigrateToURI3(dom, desturi, params, nparams, flags) == 0) diff --git a/tools/virsh.pod b/tools/virsh.pod index 64b4287..2ace976 100644 --- a/tools/virsh.pod +++ b/tools/virsh.pod @@ -1530,8 +1530,9 @@ to the I<uri> namespace is displayed instead of being modified. [I<--persistent>] [I<--undefinesource>] [I<--suspend>] [I<--copy-storage-all>] [I<--copy-storage-inc>] [I<--change-protection>] [I<--unsafe>] [I<--verbose>] [I<--compressed>] [I<--abort-on-error>] [I<--auto-converge>] [I<--postcopy>] -I<domain> I<desturi> [I<migrateuri>] [I<graphicsuri>] [I<listen-address>] -[I<dname>] [I<--timeout> B<seconds> [I<--timeout-suspend> | I<--timeout-postcopy>]] +[I<--postcopy-after-precopy>] I<domain> I<desturi> [I<migrateuri>] +[I<graphicsuri>] [I<listen-address>] [I<dname>] +[I<--timeout> B<seconds> [I<--timeout-suspend> | I<--timeout-postcopy>]] [I<--xml> B<file>] [I<--migrate-disks> B<disk-list>] Migrate domain to another host. Add I<--live> for live migration; <--p2p> @@ -1562,7 +1563,9 @@ migration. I<--auto-converge> forces convergence during live migration. I<--postcopy> enables post-copy logic in migration, but does not actually start post-copy, i.e., migration is started in pre-copy mode. Once migration is running, the user may switch to post-copy using the -B<migrate-postcopy> command sent from another virsh instance. +B<migrate-postcopy> command sent from another virsh instance or use +I<--postcopy-after-precopy> to let libvirt automatically switch to +post-copy after the first pass of pre-copy is finished. B<Note>: Individual hypervisors usually do not support all possible types of migration. For example, QEMU does not support direct migration. -- 2.7.0
When destination libvirtd is restarted during migration in Finish phase just after the point we started guest CPUs, we should not kill the domain. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index e067c3f..240aa04 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3307,9 +3307,13 @@ qemuProcessRecoverMigration(virQEMUDriverPtr driver, case QEMU_MIGRATION_PHASE_FINISH3: /* migration finished, we started resuming the domain but didn't - * confirm success or failure yet; killing it seems safest */ - VIR_DEBUG("Killing migrated domain %s", vm->def->name); - return -1; + * confirm success or failure yet; killing it seems safest unless + * we already started guest CPUs */ + if (state != VIR_DOMAIN_RUNNING) { + VIR_DEBUG("Killing migrated domain %s", vm->def->name); + return -1; + } + break; } } else if (job == QEMU_ASYNC_JOB_MIGRATION_OUT) { switch (phase) { -- 2.7.0
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 206 ++++++++++++++++++++++++++---------------------- 1 file changed, 110 insertions(+), 96 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 240aa04..d6a67b0 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3268,113 +3268,122 @@ qemuProcessUpdateState(virQEMUDriverPtr driver, virDomainObjPtr vm) } static int -qemuProcessRecoverMigration(virQEMUDriverPtr driver, - virDomainObjPtr vm, - virConnectPtr conn, - qemuDomainAsyncJob job, - qemuMigrationJobPhase phase, - virDomainState state, - int reason) +qemuProcessRecoverMigrationIn(virQEMUDriverPtr driver, + virDomainObjPtr vm, + virConnectPtr conn, + qemuMigrationJobPhase phase, + virDomainState state, + int reason ATTRIBUTE_UNUSED) { - if (job == QEMU_ASYNC_JOB_MIGRATION_IN) { - switch (phase) { - case QEMU_MIGRATION_PHASE_NONE: - case QEMU_MIGRATION_PHASE_PERFORM2: - case QEMU_MIGRATION_PHASE_BEGIN3: - case QEMU_MIGRATION_PHASE_PERFORM3: - case QEMU_MIGRATION_PHASE_PERFORM3_DONE: - case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: - case QEMU_MIGRATION_PHASE_CONFIRM3: - case QEMU_MIGRATION_PHASE_LAST: - break; + switch (phase) { + case QEMU_MIGRATION_PHASE_NONE: + case QEMU_MIGRATION_PHASE_PERFORM2: + case QEMU_MIGRATION_PHASE_BEGIN3: + case QEMU_MIGRATION_PHASE_PERFORM3: + case QEMU_MIGRATION_PHASE_PERFORM3_DONE: + case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: + case QEMU_MIGRATION_PHASE_CONFIRM3: + case QEMU_MIGRATION_PHASE_LAST: + /* N/A for incoming migration */ + break; - case QEMU_MIGRATION_PHASE_PREPARE: - VIR_DEBUG("Killing unfinished incoming migration for domain %s", - vm->def->name); + case QEMU_MIGRATION_PHASE_PREPARE: + VIR_DEBUG("Killing unfinished incoming migration for domain %s", + vm->def->name); + return -1; + + case QEMU_MIGRATION_PHASE_FINISH2: + /* source domain is already killed so let's just resume the domain + * and hope we are all set */ + VIR_DEBUG("Incoming migration finished, resuming domain %s", + vm->def->name); + if (qemuProcessStartCPUs(driver, vm, conn, + VIR_DOMAIN_RUNNING_UNPAUSED, + QEMU_ASYNC_JOB_NONE) < 0) { + VIR_WARN("Could not resume domain %s", vm->def->name); + } + break; + + case QEMU_MIGRATION_PHASE_FINISH3: + /* migration finished, we started resuming the domain but didn't + * confirm success or failure yet; killing it seems safest unless + * we already started guest CPUs */ + if (state != VIR_DOMAIN_RUNNING) { + VIR_DEBUG("Killing migrated domain %s", vm->def->name); return -1; + } + break; + } - case QEMU_MIGRATION_PHASE_FINISH2: - /* source domain is already killed so let's just resume the domain - * and hope we are all set */ - VIR_DEBUG("Incoming migration finished, resuming domain %s", - vm->def->name); - if (qemuProcessStartCPUs(driver, vm, conn, - VIR_DOMAIN_RUNNING_UNPAUSED, - QEMU_ASYNC_JOB_NONE) < 0) { - VIR_WARN("Could not resume domain %s", vm->def->name); - } - break; + return 0; +} - case QEMU_MIGRATION_PHASE_FINISH3: - /* migration finished, we started resuming the domain but didn't - * confirm success or failure yet; killing it seems safest unless - * we already started guest CPUs */ - if (state != VIR_DOMAIN_RUNNING) { - VIR_DEBUG("Killing migrated domain %s", vm->def->name); - return -1; - } - break; +static int +qemuProcessRecoverMigrationOut(virQEMUDriverPtr driver, + virDomainObjPtr vm, + virConnectPtr conn, + qemuMigrationJobPhase phase, + virDomainState state, + int reason) +{ + switch (phase) { + case QEMU_MIGRATION_PHASE_NONE: + case QEMU_MIGRATION_PHASE_PREPARE: + case QEMU_MIGRATION_PHASE_FINISH2: + case QEMU_MIGRATION_PHASE_FINISH3: + case QEMU_MIGRATION_PHASE_LAST: + /* N/A for outgoing migration */ + break; + + case QEMU_MIGRATION_PHASE_BEGIN3: + /* nothing happened so far, just forget we were about to migrate the + * domain */ + break; + + case QEMU_MIGRATION_PHASE_PERFORM2: + case QEMU_MIGRATION_PHASE_PERFORM3: + /* migration is still in progress, let's cancel it and resume the + * domain */ + VIR_DEBUG("Cancelling unfinished migration of domain %s", + vm->def->name); + if (qemuMigrationCancel(driver, vm) < 0) { + VIR_WARN("Could not cancel ongoing migration of domain %s", + vm->def->name); } - } else if (job == QEMU_ASYNC_JOB_MIGRATION_OUT) { - switch (phase) { - case QEMU_MIGRATION_PHASE_NONE: - case QEMU_MIGRATION_PHASE_PREPARE: - case QEMU_MIGRATION_PHASE_FINISH2: - case QEMU_MIGRATION_PHASE_FINISH3: - case QEMU_MIGRATION_PHASE_LAST: - break; + goto resume; - case QEMU_MIGRATION_PHASE_BEGIN3: - /* nothing happen so far, just forget we were about to migrate the - * domain */ - break; + case QEMU_MIGRATION_PHASE_PERFORM3_DONE: + /* migration finished but we didn't have a chance to get the result + * of Finish3 step; third party needs to check what to do next + */ + break; - case QEMU_MIGRATION_PHASE_PERFORM2: - case QEMU_MIGRATION_PHASE_PERFORM3: - /* migration is still in progress, let's cancel it and resume the - * domain */ - if (qemuMigrationCancel(driver, vm) < 0) - return -1; - /* resume the domain but only if it was paused as a result of - * migration */ - if (state == VIR_DOMAIN_PAUSED && - (reason == VIR_DOMAIN_PAUSED_MIGRATION || - reason == VIR_DOMAIN_PAUSED_UNKNOWN)) { - if (qemuProcessStartCPUs(driver, vm, conn, - VIR_DOMAIN_RUNNING_UNPAUSED, - QEMU_ASYNC_JOB_NONE) < 0) { - VIR_WARN("Could not resume domain %s", vm->def->name); - } - } - break; + case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: + /* Finish3 failed, we need to resume the domain */ + VIR_DEBUG("Resuming domain %s after failed migration", + vm->def->name); + goto resume; - case QEMU_MIGRATION_PHASE_PERFORM3_DONE: - /* migration finished but we didn't have a chance to get the result - * of Finish3 step; third party needs to check what to do next - */ - break; + case QEMU_MIGRATION_PHASE_CONFIRM3: + /* migration completed, we need to kill the domain here */ + return -1; + } - case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: - /* Finish3 failed, we need to resume the domain */ - VIR_DEBUG("Resuming domain %s after failed migration", - vm->def->name); - if (state == VIR_DOMAIN_PAUSED && - (reason == VIR_DOMAIN_PAUSED_MIGRATION || - reason == VIR_DOMAIN_PAUSED_UNKNOWN)) { - if (qemuProcessStartCPUs(driver, vm, conn, - VIR_DOMAIN_RUNNING_UNPAUSED, - QEMU_ASYNC_JOB_NONE) < 0) { - VIR_WARN("Could not resume domain %s", vm->def->name); - } - } - break; + return 0; - case QEMU_MIGRATION_PHASE_CONFIRM3: - /* migration completed, we need to kill the domain here */ - return -1; + resume: + /* resume the domain but only if it was paused as a result of + * migration + */ + if (state == VIR_DOMAIN_PAUSED && + (reason == VIR_DOMAIN_PAUSED_MIGRATION || + reason == VIR_DOMAIN_PAUSED_UNKNOWN)) { + if (qemuProcessStartCPUs(driver, vm, conn, + VIR_DOMAIN_RUNNING_UNPAUSED, + QEMU_ASYNC_JOB_NONE) < 0) { + VIR_WARN("Could not resume domain %s", vm->def->name); } } - return 0; } @@ -3392,9 +3401,14 @@ qemuProcessRecoverJob(virQEMUDriverPtr driver, switch (job->asyncJob) { case QEMU_ASYNC_JOB_MIGRATION_OUT: + if (qemuProcessRecoverMigrationOut(driver, vm, conn, job->phase, + state, reason) < 0) + return -1; + break; + case QEMU_ASYNC_JOB_MIGRATION_IN: - if (qemuProcessRecoverMigration(driver, vm, conn, job->asyncJob, - job->phase, state, reason) < 0) + if (qemuProcessRecoverMigrationIn(driver, vm, conn, job->phase, + state, reason) < 0) return -1; break; -- 2.7.0
When migration fails in the post-copy mode, it's impossible to just kill the destination domain and resume the source since the source no longer contains current guest state. Let's mark domains on both sides as VIR_DOMAIN_PAUSED_POSTCOPY_FAILED to let the upper layer decide what to do with them. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 87 +++++++++++++++++++++++++++++++++++++++-------- src/qemu/qemu_migration.h | 3 ++ src/qemu/qemu_process.c | 59 ++++++++++++++++++++++++-------- 3 files changed, 119 insertions(+), 30 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 296e54e..f2f2927 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1475,14 +1475,21 @@ qemuMigrationRestoreDomainState(virConnectPtr conn, virDomainObjPtr vm) { virQEMUDriverPtr driver = conn->privateData; qemuDomainObjPrivatePtr priv = vm->privateData; - int state = virDomainObjGetState(vm, NULL); + int reason; + virDomainState state = virDomainObjGetState(vm, &reason); bool ret = false; - VIR_DEBUG("driver=%p, vm=%p, pre-mig-state=%d, state=%d", - driver, vm, priv->preMigrationState, state); + VIR_DEBUG("driver=%p, vm=%p, pre-mig-state=%s, state=%s, reason=%s", + driver, vm, + virDomainStateTypeToString(priv->preMigrationState), + virDomainStateTypeToString(state), + virDomainStateReasonToString(state, reason)); - if (state == VIR_DOMAIN_PAUSED && - priv->preMigrationState == VIR_DOMAIN_RUNNING) { + if (state != VIR_DOMAIN_PAUSED || + reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED) + goto cleanup; + + if (priv->preMigrationState == VIR_DOMAIN_RUNNING) { /* This is basically the only restore possibility that's safe * and we should attempt to do */ @@ -2365,6 +2372,46 @@ qemuMigrationSetOffline(virQEMUDriverPtr driver, } +void +qemuMigrationPostcopyFailed(virQEMUDriverPtr driver, + virDomainObjPtr vm) +{ + virObjectEventPtr event; + virDomainState state; + int reason; + + state = virDomainObjGetState(vm, &reason); + + if (state != VIR_DOMAIN_PAUSED && + state != VIR_DOMAIN_RUNNING) + return; + + if (state == VIR_DOMAIN_PAUSED && + reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED) + return; + + VIR_WARN("Migration of domain %s failed during post-copy; " + "leaving the domain paused", vm->def->name); + + if (state == VIR_DOMAIN_RUNNING) { + if (qemuProcessStopCPUs(driver, vm, + VIR_DOMAIN_PAUSED_POSTCOPY_FAILED, + QEMU_ASYNC_JOB_MIGRATION_IN) < 0) { + VIR_WARN("Unable to pause guest CPUs for %s", vm->def->name); + return; + } + } else { + virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, + VIR_DOMAIN_PAUSED_POSTCOPY_FAILED); + } + + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_SUSPENDED, + VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED); + qemuDomainEventQueue(driver, event); +} + + static int qemuMigrationSetCompression(virQEMUDriverPtr driver, virDomainObjPtr vm, @@ -4085,8 +4132,8 @@ qemuMigrationConfirmPhase(virQEMUDriverPtr driver, if (flags & VIR_MIGRATE_OFFLINE) goto done; - /* Did the migration go as planned? If yes, kill off the - * domain object, but if no, resume CPUs + /* Did the migration go as planned? If yes, kill off the domain object. + * If something failed, resume CPUs, but only if we didn't use post-copy. */ if (retcode == 0) { /* If guest uses SPICE and supports seamless migration we have to hold @@ -4102,6 +4149,7 @@ qemuMigrationConfirmPhase(virQEMUDriverPtr driver, VIR_DOMAIN_EVENT_STOPPED_MIGRATED); } else { virErrorPtr orig_err = virSaveLastError(); + int reason; /* cancel any outstanding NBD jobs */ qemuMigrationCancelDriveMirror(driver, vm, false, @@ -4110,7 +4158,10 @@ qemuMigrationConfirmPhase(virQEMUDriverPtr driver, virSetError(orig_err); virFreeError(orig_err); - if (qemuMigrationRestoreDomainState(conn, vm)) { + if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && + reason == VIR_DOMAIN_PAUSED_POSTCOPY) { + qemuMigrationPostcopyFailed(driver, vm); + } else if (qemuMigrationRestoreDomainState(conn, vm)) { event = virDomainEventLifecycleNewFromObj(vm, VIR_DOMAIN_EVENT_RESUMED, VIR_DOMAIN_EVENT_RESUMED_MIGRATED); @@ -5929,6 +5980,7 @@ qemuMigrationFinish(virQEMUDriverPtr driver, unsigned long long timeReceived = 0; virObjectEventPtr event; bool inPostCopy = false; + bool kill = true; VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " "cookieout=%p, cookieoutlen=%p, flags=%lx, retcode=%d", @@ -6062,6 +6114,7 @@ qemuMigrationFinish(virQEMUDriverPtr driver, } if (inPostCopy) { + kill = false; event = virDomainEventLifecycleNewFromObj(vm, VIR_DOMAIN_EVENT_RESUMED, VIR_DOMAIN_EVENT_RESUMED_POSTCOPY); @@ -6122,13 +6175,17 @@ qemuMigrationFinish(virQEMUDriverPtr driver, if (!dom && !(flags & VIR_MIGRATE_OFFLINE) && virDomainObjIsActive(vm)) { - qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, - VIR_QEMU_PROCESS_STOP_MIGRATED); - virDomainAuditStop(vm, "failed"); - event = virDomainEventLifecycleNewFromObj(vm, - VIR_DOMAIN_EVENT_STOPPED, - VIR_DOMAIN_EVENT_STOPPED_FAILED); - qemuDomainEventQueue(driver, event); + if (kill) { + qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, + VIR_QEMU_PROCESS_STOP_MIGRATED); + virDomainAuditStop(vm, "failed"); + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_STOPPED, + VIR_DOMAIN_EVENT_STOPPED_FAILED); + qemuDomainEventQueue(driver, event); + } else { + qemuMigrationPostcopyFailed(driver, vm); + } } if (dom && diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index ccef71b..6398117 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -212,4 +212,7 @@ int qemuMigrationRunIncoming(virQEMUDriverPtr driver, const char *uri, qemuDomainAsyncJob asyncJob); +void qemuMigrationPostcopyFailed(virQEMUDriverPtr driver, + virDomainObjPtr vm); + #endif /* __QEMU_MIGRATION_H__ */ diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index d6a67b0..a3b2953 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3273,8 +3273,13 @@ qemuProcessRecoverMigrationIn(virQEMUDriverPtr driver, virConnectPtr conn, qemuMigrationJobPhase phase, virDomainState state, - int reason ATTRIBUTE_UNUSED) + int reason) { + bool postcopy = (state == VIR_DOMAIN_PAUSED && + reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED) || + (state == VIR_DOMAIN_RUNNING && + reason == VIR_DOMAIN_RUNNING_POSTCOPY); + switch (phase) { case QEMU_MIGRATION_PHASE_NONE: case QEMU_MIGRATION_PHASE_PERFORM2: @@ -3307,8 +3312,10 @@ qemuProcessRecoverMigrationIn(virQEMUDriverPtr driver, case QEMU_MIGRATION_PHASE_FINISH3: /* migration finished, we started resuming the domain but didn't * confirm success or failure yet; killing it seems safest unless - * we already started guest CPUs */ - if (state != VIR_DOMAIN_RUNNING) { + * we already started guest CPUs or we were in post-copy mode */ + if (postcopy) { + qemuMigrationPostcopyFailed(driver, vm); + } else if (state != VIR_DOMAIN_RUNNING) { VIR_DEBUG("Killing migrated domain %s", vm->def->name); return -1; } @@ -3326,6 +3333,10 @@ qemuProcessRecoverMigrationOut(virQEMUDriverPtr driver, virDomainState state, int reason) { + bool postcopy = state == VIR_DOMAIN_PAUSED && + (reason == VIR_DOMAIN_PAUSED_POSTCOPY || + reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED); + switch (phase) { case QEMU_MIGRATION_PHASE_NONE: case QEMU_MIGRATION_PHASE_PREPARE: @@ -3343,26 +3354,44 @@ qemuProcessRecoverMigrationOut(virQEMUDriverPtr driver, case QEMU_MIGRATION_PHASE_PERFORM2: case QEMU_MIGRATION_PHASE_PERFORM3: /* migration is still in progress, let's cancel it and resume the - * domain */ - VIR_DEBUG("Cancelling unfinished migration of domain %s", - vm->def->name); - if (qemuMigrationCancel(driver, vm) < 0) { - VIR_WARN("Could not cancel ongoing migration of domain %s", - vm->def->name); + * domain; however we can only do that before migration enters + * post-copy mode + */ + if (postcopy) { + qemuMigrationPostcopyFailed(driver, vm); + } else { + VIR_DEBUG("Cancelling unfinished migration of domain %s", + vm->def->name); + if (qemuMigrationCancel(driver, vm) < 0) { + VIR_WARN("Could not cancel ongoing migration of domain %s", + vm->def->name); + } + goto resume; } - goto resume; + break; case QEMU_MIGRATION_PHASE_PERFORM3_DONE: /* migration finished but we didn't have a chance to get the result - * of Finish3 step; third party needs to check what to do next + * of Finish3 step; third party needs to check what to do next; in + * post-copy mode we can use PAUSED_POSTCOPY_FAILED state for this */ + if (postcopy) + qemuMigrationPostcopyFailed(driver, vm); break; case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: - /* Finish3 failed, we need to resume the domain */ - VIR_DEBUG("Resuming domain %s after failed migration", - vm->def->name); - goto resume; + /* Finish3 failed, we need to resume the domain, but once we enter + * post-copy mode there's no way back, so let's just mark the domain + * as broken in that case + */ + if (postcopy) { + qemuMigrationPostcopyFailed(driver, vm); + } else { + VIR_DEBUG("Resuming domain %s after failed migration", + vm->def->name); + goto resume; + } + break; case QEMU_MIGRATION_PHASE_CONFIRM3: /* migration completed, we need to kill the domain here */ -- 2.7.0
In post-copy mode none of the hosts has a complete guest state and rolling back migration is impossible. Thus aborting it would be equivalent to destroying the domain. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index c62561b..341c731 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -13138,6 +13138,7 @@ static int qemuDomainAbortJob(virDomainPtr dom) virDomainObjPtr vm; int ret = -1; qemuDomainObjPrivatePtr priv; + int reason; if (!(vm = qemuDomObjFromDomain(dom))) goto cleanup; @@ -13160,13 +13161,24 @@ static int qemuDomainAbortJob(virDomainPtr dom) virReportError(VIR_ERR_OPERATION_INVALID, "%s", _("no job is active on the domain")); goto endjob; - } else if (priv->job.asyncJob == QEMU_ASYNC_JOB_MIGRATION_IN) { + } + + if (priv->job.asyncJob == QEMU_ASYNC_JOB_MIGRATION_IN) { virReportError(VIR_ERR_OPERATION_INVALID, "%s", _("cannot abort incoming migration;" " use virDomainDestroy instead")); goto endjob; } + if (priv->job.asyncJob == QEMU_ASYNC_JOB_MIGRATION_OUT && + (priv->job.current->stats.status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY || + (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && + reason == VIR_DOMAIN_PAUSED_POSTCOPY))) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("cannot abort migration in post-copy mode")); + goto endjob; + } + VIR_DEBUG("Cancelling job at client request"); qemuDomainObjAbortAsyncJob(vm); qemuDomainObjEnterMonitor(driver, vm); -- 2.7.0
participants (2)
-
Daniel P. Berrange -
Jiri Denemark