[libvirt PATCH 00/80] Add support for post-copy recovery
This series implements a new VIR_MIGRATE_POSTCOPY_RESUME flag (virsh migrate --resume) for recovering from a failed post-copy migration. You can also fetch the series from my gitlab fork: git fetch https://gitlab.com/jirkade/libvirt.git post-copy-recovery Jiri Denemark (80): qemu: Add debug messages to job recovery code qemumonitorjsontest: Test more migration capabilities qemu: Return state from qemuMonitorGetMigrationCapabilities qemu: Enable migration events only when disabled Introduce VIR_DOMAIN_RUNNING_POSTCOPY_FAILED qemu: Keep domain running on dst on failed post-copy migration qemu: Explicitly emit events on post-copy failure qemu: Make qemuDomainCleanupAdd return void conf: Introduce virDomainObjIsFailedPostcopy helper conf: Introduce virDomainObjIsPostcopy helper qemu: Introduce qemuProcessCleanupMigrationJob qemu: Rename qemuDomainObjRestoreJob as qemuDomainObjPreserveJob qemu: Add qemuDomainObjRestoreAsyncJob qemu: Keep migration job active after failed post-copy qemu: Abort failed post-copy when we haven't called Finish yet qemu: Restore failed migration job on reconnect qemu: Restore async job start timestamp on reconnect qemu: Drop forward declarations in migration code qemu: Don't wait for migration job when migration is running qemu: Use switch in qemuDomainGetJobInfoMigrationStats qemu: Fetch paused migration stats qemu: Handle 'postcopy-paused' migration state qemu: Add support for postcopy-recover QEMU migration state qemu: Create domain object at the end of qemuMigrationDstFinish qemu: Move success-only code out of endjob in qemuMigrationDstFinish qemu: Separate success and failure path in qemuMigrationDstFinish qemu: Rename "endjob" label in qemuMigrationDstFinish qemu: Generate migration cookie in Finish phase earlier qemu: Make final part of migration Finish phase reusable qemu: Drop obsolete comment in qemuMigrationDstFinish qemu: Preserve error in qemuMigrationDstFinish qemu: Introduce qemuMigrationDstFinishFresh qemu: Introduce qemuMigrationDstFinishOffline qemu: Separate cookie parsing for qemuMigrationDstFinishOffline qemu: Introduce qemuMigrationDstFinishActive qemu: Handle migration job in qemuMigrationDstFinish qemu: Make final part of migration Confirm phase reusable qemu: Make sure migrationPort is released even in callbacks qemu: Pass qemuDomainJobObj to qemuMigrationDstComplete qemu: Finish completed unattended migration qemu: Ignore missing memory statistics in query-migrate qemu: Improve post-copy migration handling on reconnect qemu: Check flags incompatible with offline migration earlier qemu: Introduce qemuMigrationSrcBeginXML helper qemu: Add new migration phases for post-copy recovery qemu: Separate protocol checks from qemuMigrationJobSetPhase qemu: Make qemuMigrationCheckPhase failure fatal qemu: Refactor qemuDomainObjSetJobPhase qemu: Do not set job owner in qemuMigrationJobSetPhase qemu: Use QEMU_MIGRATION_PHASE_POSTCOPY_FAILED Introduce VIR_MIGRATE_POSTCOPY_RESUME flag virsh: Add --postcopy-resume option for migrate command qemu: Don't set VIR_MIGRATE_PAUSED for post-copy resume qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Begin phase qmeu: Refactor qemuMigrationSrcPerformPhase qemu: Separate starting migration from qemuMigrationSrcRun qemu: Add support for 'resume' parameter of migrate QMP command qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Perform phase qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Confirm phase qemu: Introduce qemuMigrationDstPrepareFresh qemu: Refactor qemuMigrationDstPrepareFresh qemu: Simplify cleanup in qemuMigrationDstPrepareFresh qemu: Add support for migrate-recover QMP command qemu: Rename qemuMigrationSrcCleanup qemu: Refactor qemuMigrationAnyConnectionClosed qemu: Handle incoming migration in qemuMigrationAnyConnectionClosed qemu: Start a migration phase in qemuMigrationAnyConnectionClosed qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Finish phase qemu: Create completed jobData in qemuMigrationSrcComplete qemu: Register qemuProcessCleanupMigrationJob after Begin phase qemu: Call qemuDomainCleanupAdd from qemuMigrationJobContinue qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for peer-to-peer migration qemu: Enable support for VIR_MIGRATE_POSTCOPY_RESUME Add virDomainAbortJobFlags public API qemu: Implement virDomainAbortJobFlags Add VIR_DOMAIN_ABORT_JOB_POSTCOPY flag for virDomainAbortJobFlags qemu: Implement VIR_DOMAIN_ABORT_JOB_POSTCOPY flag virsh: Add --postcopy option for domjobabort command NEWS: Add support for post-copy recovery NEWS.rst | 5 + docs/manpages/virsh.rst | 17 +- examples/c/misc/event-test.c | 3 + include/libvirt/libvirt-domain.h | 26 + src/conf/domain_conf.c | 33 + src/conf/domain_conf.h | 8 + src/driver-hypervisor.h | 5 + src/hypervisor/domain_job.c | 1 + src/hypervisor/domain_job.h | 1 + src/libvirt-domain.c | 72 +- src/libvirt_private.syms | 2 + src/libvirt_public.syms | 1 + src/qemu/qemu_capabilities.c | 2 +- src/qemu/qemu_domain.c | 9 +- src/qemu/qemu_domain.h | 5 +- src/qemu/qemu_domainjob.c | 103 +- src/qemu/qemu_domainjob.h | 16 +- src/qemu/qemu_driver.c | 83 +- src/qemu/qemu_migration.c | 2383 +++++++++++------ src/qemu/qemu_migration.h | 37 +- src/qemu/qemu_migration_params.c | 19 +- src/qemu/qemu_monitor.c | 27 +- src/qemu/qemu_monitor.h | 13 +- src/qemu/qemu_monitor_json.c | 151 +- src/qemu/qemu_monitor_json.h | 10 +- src/qemu/qemu_process.c | 355 ++- src/qemu/qemu_process.h | 3 + src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 14 +- src/remote_protocol-structs | 5 + tests/qemumonitorjsontest.c | 42 +- .../migration-in-params-in.xml | 2 +- .../migration-out-nbd-bitmaps-in.xml | 2 +- .../migration-out-nbd-out.xml | 2 +- .../migration-out-nbd-tls-out.xml | 2 +- .../migration-out-params-in.xml | 2 +- tools/virsh-domain-event.c | 3 +- tools/virsh-domain-monitor.c | 1 + tools/virsh-domain.c | 24 +- 39 files changed, 2556 insertions(+), 934 deletions(-) -- 2.35.1
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index b0b00eb0a2..1925559fad 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3445,6 +3445,9 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, (state == VIR_DOMAIN_RUNNING && reason == VIR_DOMAIN_RUNNING_POSTCOPY); + VIR_DEBUG("Active incoming migration in phase %s", + qemuMigrationJobPhaseTypeToString(job->phase)); + switch ((qemuMigrationJobPhase) job->phase) { case QEMU_MIGRATION_PHASE_NONE: case QEMU_MIGRATION_PHASE_PERFORM2: @@ -3506,6 +3509,9 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED); bool resume = false; + VIR_DEBUG("Active outgoing migration in phase %s", + qemuMigrationJobPhaseTypeToString(job->phase)); + switch ((qemuMigrationJobPhase) job->phase) { case QEMU_MIGRATION_PHASE_NONE: case QEMU_MIGRATION_PHASE_PREPARE: @@ -3601,6 +3607,13 @@ qemuProcessRecoverJob(virQEMUDriver *driver, state = virDomainObjGetState(vm, &reason); + VIR_DEBUG("Recovering job for domain %s, state=%s(%s), async=%s, job=%s", + vm->def->name, + virDomainStateTypeToString(state), + virDomainStateReasonToString(state, reason), + virDomainAsyncJobTypeToString(job->asyncJob), + virDomainJobTypeToString(job->active)); + switch (job->asyncJob) { case VIR_ASYNC_JOB_MIGRATION_OUT: if (qemuProcessRecoverMigrationOut(driver, vm, job, -- 2.35.1
On Tue, May 10, 2022 at 17:20:22 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Hi I'm testing this patch series, but it has conflict with latest code. Could you please rebase to latest code and provide a V2? Thanks. BR, Fangge Jin On Tue, May 10, 2022 at 11:22 PM Jiri Denemark <jdenemar@redhat.com> wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index b0b00eb0a2..1925559fad 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3445,6 +3445,9 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, (state == VIR_DOMAIN_RUNNING && reason == VIR_DOMAIN_RUNNING_POSTCOPY);
+ VIR_DEBUG("Active incoming migration in phase %s", + qemuMigrationJobPhaseTypeToString(job->phase)); + switch ((qemuMigrationJobPhase) job->phase) { case QEMU_MIGRATION_PHASE_NONE: case QEMU_MIGRATION_PHASE_PERFORM2: @@ -3506,6 +3509,9 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED); bool resume = false;
+ VIR_DEBUG("Active outgoing migration in phase %s", + qemuMigrationJobPhaseTypeToString(job->phase)); + switch ((qemuMigrationJobPhase) job->phase) { case QEMU_MIGRATION_PHASE_NONE: case QEMU_MIGRATION_PHASE_PREPARE: @@ -3601,6 +3607,13 @@ qemuProcessRecoverJob(virQEMUDriver *driver,
state = virDomainObjGetState(vm, &reason);
+ VIR_DEBUG("Recovering job for domain %s, state=%s(%s), async=%s, job=%s", + vm->def->name, + virDomainStateTypeToString(state), + virDomainStateReasonToString(state, reason), + virDomainAsyncJobTypeToString(job->asyncJob), + virDomainJobTypeToString(job->active)); + switch (job->asyncJob) { case VIR_ASYNC_JOB_MIGRATION_OUT: if (qemuProcessRecoverMigrationOut(driver, vm, job, -- 2.35.1
On Tue, May 17, 2022 at 11:59:42 +0800, Fangge Jin wrote:
Hi
I'm testing this patch series, but it has conflict with latest code. Could you please rebase to latest code and provide a V2? Thanks.
You can fetch the original version including the proper base from Jirka's gitlab. In the cover letter he states: You can also fetch the series from my gitlab fork: git fetch https://gitlab.com/jirkade/libvirt.git post-copy-recovery But AFAIK he intends to send another one, so doing a pure rebase is pointless.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- tests/qemumonitorjsontest.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-) diff --git a/tests/qemumonitorjsontest.c b/tests/qemumonitorjsontest.c index 278d7ba765..99198de0ed 100644 --- a/tests/qemumonitorjsontest.c +++ b/tests/qemumonitorjsontest.c @@ -2047,7 +2047,7 @@ testQemuMonitorJSONqemuMonitorJSONGetMigrationCapabilities(const void *opaque) { const testGenericData *data = opaque; virDomainXMLOption *xmlopt = data->xmlopt; - const char *cap; + size_t cap; g_auto(GStrv) caps = NULL; g_autoptr(virBitmap) bitmap = NULL; g_autoptr(virJSONValue) json = NULL; @@ -2057,6 +2057,10 @@ testQemuMonitorJSONqemuMonitorJSONGetMigrationCapabilities(const void *opaque) " {" " \"state\": false," " \"capability\": \"xbzrle\"" + " }," + " {" + " \"state\": true," + " \"capability\": \"events\"" " }" " ]," " \"id\": \"libvirt-22\"" @@ -2075,11 +2079,25 @@ testQemuMonitorJSONqemuMonitorJSONGetMigrationCapabilities(const void *opaque) &caps) < 0) return -1; - cap = qemuMigrationCapabilityTypeToString(QEMU_MIGRATION_CAP_XBZRLE); - if (!g_strv_contains((const char **) caps, cap)) { - virReportError(VIR_ERR_INTERNAL_ERROR, - "Expected capability %s is missing", cap); - return -1; + for (cap = 0; cap < QEMU_MIGRATION_CAP_LAST; cap++) { + const char *capStr = qemuMigrationCapabilityTypeToString(cap); + bool present = g_strv_contains((const char **) caps, capStr); + + switch (cap) { + case QEMU_MIGRATION_CAP_XBZRLE: + case QEMU_MIGRATION_CAP_EVENTS: + if (!present) { + VIR_TEST_VERBOSE("Expected capability %s is missing", capStr); + return -1; + } + break; + + default: + if (present) { + VIR_TEST_VERBOSE("Unexpected capability %s found", capStr); + return -1; + } + } } bitmap = virBitmapNew(QEMU_MIGRATION_CAP_LAST); -- 2.35.1
On Tue, May 10, 2022 at 17:20:23 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- tests/qemumonitorjsontest.c | 30 ++++++++++++++++++++++++------ 1 file changed, 24 insertions(+), 6 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The function can now optionally return a bitmap describing the current state of each migration capability. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_capabilities.c | 2 +- src/qemu/qemu_migration_params.c | 2 +- src/qemu/qemu_monitor.c | 5 +++-- src/qemu/qemu_monitor.h | 3 ++- src/qemu/qemu_monitor_json.c | 18 +++++++++++++++++- src/qemu/qemu_monitor_json.h | 3 ++- tests/qemumonitorjsontest.c | 12 +++++++++++- 7 files changed, 37 insertions(+), 8 deletions(-) diff --git a/src/qemu/qemu_capabilities.c b/src/qemu/qemu_capabilities.c index 1ed4cda7f0..cd13847918 100644 --- a/src/qemu/qemu_capabilities.c +++ b/src/qemu/qemu_capabilities.c @@ -3347,7 +3347,7 @@ virQEMUCapsProbeQMPMigrationCapabilities(virQEMUCaps *qemuCaps, { g_auto(GStrv) caps = NULL; - if (qemuMonitorGetMigrationCapabilities(mon, &caps) < 0) + if (qemuMonitorGetMigrationCapabilities(mon, &caps, NULL) < 0) return -1; virQEMUCapsProcessStringFlags(qemuCaps, diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c index df2384b213..34416f89be 100644 --- a/src/qemu/qemu_migration_params.c +++ b/src/qemu/qemu_migration_params.c @@ -1397,7 +1397,7 @@ qemuMigrationCapsCheck(virQEMUDriver *driver, if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) < 0) return -1; - rc = qemuMonitorGetMigrationCapabilities(priv->mon, &caps); + rc = qemuMonitorGetMigrationCapabilities(priv->mon, &caps, NULL); qemuDomainObjExitMonitor(vm); if (rc < 0) diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 316cff5b9b..98cf1c949e 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -3636,11 +3636,12 @@ qemuMonitorGetTargetArch(qemuMonitor *mon) int qemuMonitorGetMigrationCapabilities(qemuMonitor *mon, - char ***capabilities) + char ***capabilities, + virBitmap **state) { QEMU_CHECK_MONITOR(mon); - return qemuMonitorJSONGetMigrationCapabilities(mon, capabilities); + return qemuMonitorJSONGetMigrationCapabilities(mon, capabilities, state); } diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 5c2a749282..abc29eaf4c 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -890,7 +890,8 @@ int qemuMonitorGetMigrationStats(qemuMonitor *mon, char **error); int qemuMonitorGetMigrationCapabilities(qemuMonitor *mon, - char ***capabilities); + char ***capabilities, + virBitmap **state); int qemuMonitorSetMigrationCapabilities(qemuMonitor *mon, virJSONValue **caps); diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 9e611e93e8..532aad348e 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -6210,12 +6210,14 @@ qemuMonitorJSONGetTargetArch(qemuMonitor *mon) int qemuMonitorJSONGetMigrationCapabilities(qemuMonitor *mon, - char ***capabilities) + char ***capabilities, + virBitmap **state) { g_autoptr(virJSONValue) cmd = NULL; g_autoptr(virJSONValue) reply = NULL; virJSONValue *caps; g_auto(GStrv) list = NULL; + g_autoptr(virBitmap) bitmap = NULL; size_t i; size_t n; @@ -6235,10 +6237,12 @@ qemuMonitorJSONGetMigrationCapabilities(qemuMonitor *mon, n = virJSONValueArraySize(caps); list = g_new0(char *, n + 1); + bitmap = virBitmapNew(n); for (i = 0; i < n; i++) { virJSONValue *cap = virJSONValueArrayGet(caps, i); const char *name; + bool enabled = false; if (!cap || virJSONValueGetType(cap) != VIR_JSON_TYPE_OBJECT) { virReportError(VIR_ERR_INTERNAL_ERROR, "%s", @@ -6252,10 +6256,22 @@ qemuMonitorJSONGetMigrationCapabilities(qemuMonitor *mon, return -1; } + if (virJSONValueObjectGetBoolean(cap, "state", &enabled) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing migration capability state")); + return -1; + } + list[i] = g_strdup(name); + + if (enabled) + ignore_value(virBitmapSetBit(bitmap, i)); } *capabilities = g_steal_pointer(&list); + if (state) + *state = g_steal_pointer(&bitmap); + return n; } diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h index 982fbad44e..afd100f653 100644 --- a/src/qemu/qemu_monitor_json.h +++ b/src/qemu/qemu_monitor_json.h @@ -182,7 +182,8 @@ qemuMonitorJSONGetMigrationStats(qemuMonitor *mon, int qemuMonitorJSONGetMigrationCapabilities(qemuMonitor *mon, - char ***capabilities); + char ***capabilities, + virBitmap **state); int qemuMonitorJSONSetMigrationCapabilities(qemuMonitor *mon, virJSONValue **caps); diff --git a/tests/qemumonitorjsontest.c b/tests/qemumonitorjsontest.c index 99198de0ed..8ce96885e3 100644 --- a/tests/qemumonitorjsontest.c +++ b/tests/qemumonitorjsontest.c @@ -2051,6 +2051,9 @@ testQemuMonitorJSONqemuMonitorJSONGetMigrationCapabilities(const void *opaque) g_auto(GStrv) caps = NULL; g_autoptr(virBitmap) bitmap = NULL; g_autoptr(virJSONValue) json = NULL; + g_autoptr(virBitmap) state = NULL; + g_autofree char *stateActual = NULL; + const char *stateExpected = "1"; const char *reply = "{" " \"return\": [" @@ -2076,7 +2079,7 @@ testQemuMonitorJSONqemuMonitorJSONGetMigrationCapabilities(const void *opaque) return -1; if (qemuMonitorGetMigrationCapabilities(qemuMonitorTestGetMonitor(test), - &caps) < 0) + &caps, &state) < 0) return -1; for (cap = 0; cap < QEMU_MIGRATION_CAP_LAST; cap++) { @@ -2100,6 +2103,13 @@ testQemuMonitorJSONqemuMonitorJSONGetMigrationCapabilities(const void *opaque) } } + stateActual = virBitmapFormat(state); + if (STRNEQ_NULLABLE(stateExpected, stateActual)) { + VIR_TEST_VERBOSE("Excepted capabilities state: '%s', got: '%s'", + NULLSTR(stateExpected), NULLSTR(stateActual)); + return -1; + } + bitmap = virBitmapNew(QEMU_MIGRATION_CAP_LAST); ignore_value(virBitmapSetBit(bitmap, QEMU_MIGRATION_CAP_XBZRLE)); if (!(json = qemuMigrationCapsToJSON(bitmap, bitmap))) -- 2.35.1
On Tue, May 10, 2022 at 17:20:24 +0200, Jiri Denemark wrote:
The function can now optionally return a bitmap describing the current state of each migration capability.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_capabilities.c | 2 +- src/qemu/qemu_migration_params.c | 2 +- src/qemu/qemu_monitor.c | 5 +++-- src/qemu/qemu_monitor.h | 3 ++- src/qemu/qemu_monitor_json.c | 18 +++++++++++++++++- src/qemu/qemu_monitor_json.h | 3 ++- tests/qemumonitorjsontest.c | 12 +++++++++++- 7 files changed, 37 insertions(+), 8 deletions(-)
[...]
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 9e611e93e8..532aad348e 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -6210,12 +6210,14 @@ qemuMonitorJSONGetTargetArch(qemuMonitor *mon)
I'm not exactly a fan of such loose correlation of data. Asking you to return a list of stucts would probably cause too much conflicts though, so as an alternative please document the relation between the 'capabilities' array and the positions in the 'state' bitmap in the comment for this function (that you'll need to add).
int qemuMonitorJSONGetMigrationCapabilities(qemuMonitor *mon, - char ***capabilities) + char ***capabilities, + virBitmap **state) { g_autoptr(virJSONValue) cmd = NULL; g_autoptr(virJSONValue) reply = NULL; virJSONValue *caps; g_auto(GStrv) list = NULL; + g_autoptr(virBitmap) bitmap = NULL; size_t i; size_t n;
@@ -6235,10 +6237,12 @@ qemuMonitorJSONGetMigrationCapabilities(qemuMonitor *mon, n = virJSONValueArraySize(caps);
list = g_new0(char *, n + 1); + bitmap = virBitmapNew(n);
for (i = 0; i < n; i++) { virJSONValue *cap = virJSONValueArrayGet(caps, i); const char *name; + bool enabled = false;
if (!cap || virJSONValueGetType(cap) != VIR_JSON_TYPE_OBJECT) { virReportError(VIR_ERR_INTERNAL_ERROR, "%s", @@ -6252,10 +6256,22 @@ qemuMonitorJSONGetMigrationCapabilities(qemuMonitor *mon, return -1; }
+ if (virJSONValueObjectGetBoolean(cap, "state", &enabled) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("missing migration capability state")); + return -1; + } + list[i] = g_strdup(name); + + if (enabled) + ignore_value(virBitmapSetBit(bitmap, i)); }
*capabilities = g_steal_pointer(&list); + if (state) + *state = g_steal_pointer(&bitmap); + return n; }
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 10:30:24 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:24 +0200, Jiri Denemark wrote:
The function can now optionally return a bitmap describing the current state of each migration capability.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_capabilities.c | 2 +- src/qemu/qemu_migration_params.c | 2 +- src/qemu/qemu_monitor.c | 5 +++-- src/qemu/qemu_monitor.h | 3 ++- src/qemu/qemu_monitor_json.c | 18 +++++++++++++++++- src/qemu/qemu_monitor_json.h | 3 ++- tests/qemumonitorjsontest.c | 12 +++++++++++- 7 files changed, 37 insertions(+), 8 deletions(-)
[...]
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 9e611e93e8..532aad348e 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -6210,12 +6210,14 @@ qemuMonitorJSONGetTargetArch(qemuMonitor *mon)
I'm not exactly a fan of such loose correlation of data. Asking you to return a list of stucts would probably cause too much conflicts though, so as an alternative please document the relation between the 'capabilities' array and the positions in the 'state' bitmap in the comment for this function (that you'll need to add).
This patch was replaced with the "qemu: Drop QEMU_CAPS_MIGRATION_EVENT" series. Jirka
When connecting to a QEMU monitor, we always try to enable migration events, but this is an invalid operation during migration. Thus reconnecting to a domain with active migration would fail. Let's check the state of migration events capability and only try to enable it when it is disabled. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration_params.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/src/qemu/qemu_migration_params.c b/src/qemu/qemu_migration_params.c index 34416f89be..ed2d9ac103 100644 --- a/src/qemu/qemu_migration_params.c +++ b/src/qemu/qemu_migration_params.c @@ -1391,13 +1391,16 @@ qemuMigrationCapsCheck(virQEMUDriver *driver, g_autoptr(virBitmap) migEvent = NULL; g_autoptr(virJSONValue) json = NULL; g_auto(GStrv) caps = NULL; + g_autoptr(virBitmap) capState = NULL; + g_autoptr(virBitmap) enabled = virBitmapNew(QEMU_MIGRATION_CAP_LAST); char **capStr; + size_t i; int rc; if (qemuDomainObjEnterMonitorAsync(driver, vm, asyncJob) < 0) return -1; - rc = qemuMonitorGetMigrationCapabilities(priv->mon, &caps, NULL); + rc = qemuMonitorGetMigrationCapabilities(priv->mon, &caps, &capState); qemuDomainObjExitMonitor(vm); if (rc < 0) @@ -1408,7 +1411,7 @@ qemuMigrationCapsCheck(virQEMUDriver *driver, priv->migrationCaps = virBitmapNew(QEMU_MIGRATION_CAP_LAST); - for (capStr = caps; *capStr; capStr++) { + for (i = 0, capStr = caps; *capStr; capStr++, i++) { int cap = qemuMigrationCapabilityTypeFromString(*capStr); if (cap < 0) { @@ -1416,10 +1419,20 @@ qemuMigrationCapsCheck(virQEMUDriver *driver, } else { ignore_value(virBitmapSetBit(priv->migrationCaps, cap)); VIR_DEBUG("Found migration capability: '%s'", *capStr); + + if (virBitmapIsBitSet(capState, i)) + ignore_value(virBitmapSetBit(enabled, cap)); } } - if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + if (virBitmapIsBitSet(enabled, QEMU_MIGRATION_CAP_EVENTS)) { + if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + VIR_DEBUG("Migration events already enabled"); + } else { + VIR_DEBUG("Migration events enabled; setting capability"); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); + } + } else if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { migEvent = virBitmapNew(QEMU_MIGRATION_CAP_LAST); ignore_value(virBitmapSetBit(migEvent, QEMU_MIGRATION_CAP_EVENTS)); -- 2.35.1
On Tue, May 10, 2022 at 17:20:25 +0200, Jiri Denemark wrote:
When connecting to a QEMU monitor, we always try to enable migration events, but this is an invalid operation during migration. Thus reconnecting to a domain with active migration would fail. Let's check the state of migration events capability and only try to enable it when it is disabled.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration_params.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
[...]
@@ -1416,10 +1419,20 @@ qemuMigrationCapsCheck(virQEMUDriver *driver, } else { ignore_value(virBitmapSetBit(priv->migrationCaps, cap)); VIR_DEBUG("Found migration capability: '%s'", *capStr); + + if (virBitmapIsBitSet(capState, i)) + ignore_value(virBitmapSetBit(enabled, cap)); } }
- if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + if (virBitmapIsBitSet(enabled, QEMU_MIGRATION_CAP_EVENTS)) { + if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + VIR_DEBUG("Migration events already enabled"); + } else { + VIR_DEBUG("Migration events enabled; setting capability"); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); + }
Is all of the "DEBUG" and checking 'qemuCaps' dance really needed? I think you can simply unconditionally enable the capability, but is there any reason it would not be enabled?
+ } else if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { migEvent = virBitmapNew(QEMU_MIGRATION_CAP_LAST);
ignore_value(virBitmapSetBit(migEvent, QEMU_MIGRATION_CAP_EVENTS));
If you provide a good enough explanation: Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 10:37:25 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:25 +0200, Jiri Denemark wrote:
When connecting to a QEMU monitor, we always try to enable migration events, but this is an invalid operation during migration. Thus reconnecting to a domain with active migration would fail. Let's check the state of migration events capability and only try to enable it when it is disabled.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration_params.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
[...]
@@ -1416,10 +1419,20 @@ qemuMigrationCapsCheck(virQEMUDriver *driver, } else { ignore_value(virBitmapSetBit(priv->migrationCaps, cap)); VIR_DEBUG("Found migration capability: '%s'", *capStr); + + if (virBitmapIsBitSet(capState, i)) + ignore_value(virBitmapSetBit(enabled, cap)); } }
- if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + if (virBitmapIsBitSet(enabled, QEMU_MIGRATION_CAP_EVENTS)) { + if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + VIR_DEBUG("Migration events already enabled"); + } else { + VIR_DEBUG("Migration events enabled; setting capability"); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); + }
Is all of the "DEBUG" and checking 'qemuCaps' dance really needed? I think you can simply unconditionally enable the capability, but is there any reason it would not be enabled?
Hmm, it looks like migration events are indeed supported by all QEMU releases we support so we could drop some parts of this and most likely even the previous patch :-) Jirka
On Wed, May 11, 2022 at 10:37:25 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:25 +0200, Jiri Denemark wrote:
When connecting to a QEMU monitor, we always try to enable migration events, but this is an invalid operation during migration. Thus reconnecting to a domain with active migration would fail. Let's check the state of migration events capability and only try to enable it when it is disabled.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration_params.c | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-)
[...]
@@ -1416,10 +1419,20 @@ qemuMigrationCapsCheck(virQEMUDriver *driver, } else { ignore_value(virBitmapSetBit(priv->migrationCaps, cap)); VIR_DEBUG("Found migration capability: '%s'", *capStr); + + if (virBitmapIsBitSet(capState, i)) + ignore_value(virBitmapSetBit(enabled, cap)); } }
- if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + if (virBitmapIsBitSet(enabled, QEMU_MIGRATION_CAP_EVENTS)) { + if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { + VIR_DEBUG("Migration events already enabled"); + } else { + VIR_DEBUG("Migration events enabled; setting capability"); + virQEMUCapsSet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); + }
Is all of the "DEBUG" and checking 'qemuCaps' dance really needed? I think you can simply unconditionally enable the capability, but is there any reason it would not be enabled?
+ } else if (virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT)) { migEvent = virBitmapNew(QEMU_MIGRATION_CAP_LAST);
ignore_value(virBitmapSetBit(migEvent, QEMU_MIGRATION_CAP_EVENTS));
If you provide a good enough explanation:
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
This patch was replaced with the "qemu: Drop QEMU_CAPS_MIGRATION_EVENT" series. Jirka
This new "post-copy failed" reason for the running state will be used on the destination host when post-copy migration fails while the domain is already running there. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- examples/c/misc/event-test.c | 3 +++ include/libvirt/libvirt-domain.h | 2 ++ src/conf/domain_conf.c | 1 + src/libvirt-domain.c | 15 ++++++++++----- src/qemu/qemu_domain.c | 3 +++ tools/virsh-domain-event.c | 3 ++- tools/virsh-domain-monitor.c | 1 + 7 files changed, 22 insertions(+), 6 deletions(-) diff --git a/examples/c/misc/event-test.c b/examples/c/misc/event-test.c index 1165469a65..64652b0153 100644 --- a/examples/c/misc/event-test.c +++ b/examples/c/misc/event-test.c @@ -196,6 +196,9 @@ eventDetailToString(int event, case VIR_DOMAIN_EVENT_RESUMED_POSTCOPY: return "Post-copy"; + case VIR_DOMAIN_EVENT_RESUMED_POSTCOPY_FAILED: + return "Post-copy Error"; + case VIR_DOMAIN_EVENT_RESUMED_LAST: break; } diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index cf9d9efd51..34b8adc2bf 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -108,6 +108,7 @@ typedef enum { wakeup event (Since: 0.9.11) */ VIR_DOMAIN_RUNNING_CRASHED = 9, /* resumed from crashed (Since: 1.1.1) */ VIR_DOMAIN_RUNNING_POSTCOPY = 10, /* running in post-copy migration mode (Since: 1.3.3) */ + VIR_DOMAIN_RUNNING_POSTCOPY_FAILED = 11, /* running in failed post-copy migration (Since: 8.4.0) */ # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_RUNNING_LAST /* (Since: 0.9.10) */ @@ -3801,6 +3802,7 @@ typedef enum { VIR_DOMAIN_EVENT_RESUMED_FROM_SNAPSHOT = 2, /* Resumed from snapshot (Since: 0.9.5) */ VIR_DOMAIN_EVENT_RESUMED_POSTCOPY = 3, /* Resumed, but migration is still running in post-copy mode (Since: 1.3.3) */ + VIR_DOMAIN_EVENT_RESUMED_POSTCOPY_FAILED = 4, /* Running, but migration failed in post-copy (Since: 8.4.0) */ # ifdef VIR_ENUM_SENTINELS VIR_DOMAIN_EVENT_RESUMED_LAST /* (Since: 0.9.10) */ diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index bd2884088c..f8c6b78c92 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -1102,6 +1102,7 @@ VIR_ENUM_IMPL(virDomainRunningReason, "wakeup", "crashed", "post-copy", + "post-copy failed", ); VIR_ENUM_IMPL(virDomainBlockedReason, diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index d1d62daa71..2e39687e27 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -9753,10 +9753,14 @@ virDomainMigrateGetMaxSpeed(virDomainPtr domain, * at most once no matter how fast it changes. On the other hand once the * guest is running on the destination host, the migration can no longer be * rolled back because none of the hosts has complete state. If this happens, - * libvirt will leave the domain paused on both hosts with - * VIR_DOMAIN_PAUSED_POSTCOPY_FAILED reason. It's up to the upper layer to - * decide what to do in such case. Because of this, libvirt will refuse to - * cancel post-copy migration via virDomainAbortJob. + * libvirt will leave the domain paused on the source host with + * VIR_DOMAIN_PAUSED_POSTCOPY_FAILED reason. The domain on the destination host + * will remain running, but the reason will change to + * VIR_DOMAIN_RUNNING_POSTCOPY_FAILED. Even though the virtual CPUs remain + * running, individual tasks in the guest may be blocked waiting for a page that + * has not been migrated yet. It's up to the upper layer to decide what to do + * in such case. Because of this, libvirt will refuse to cancel post-copy + * migration via virDomainAbortJob. * * The following domain life cycle events are emitted during post-copy * migration: @@ -9770,7 +9774,8 @@ virDomainMigrateGetMaxSpeed(virDomainPtr domain, * VIR_DOMAIN_EVENT_RESUMED_MIGRATED (on the destination), * VIR_DOMAIN_EVENT_STOPPED_MIGRATED (on the source) -- migration finished * successfully and the destination host holds a complete guest state. - * VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED (on the destination) -- emitted + * VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED (on the source), + * VIR_DOMAIN_EVENT_RESUMED_POSTCOPY_FAILED (on the destination) -- emitted * when migration fails in post-copy mode and it's unclear whether any * of the hosts has a complete guest state. * diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 7974cdb00b..51236b19f2 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -11105,6 +11105,9 @@ qemuDomainRunningReasonToResumeEvent(virDomainRunningReason reason) case VIR_DOMAIN_RUNNING_POSTCOPY: return VIR_DOMAIN_EVENT_RESUMED_POSTCOPY; + case VIR_DOMAIN_RUNNING_POSTCOPY_FAILED: + return VIR_DOMAIN_EVENT_RESUMED_POSTCOPY_FAILED; + case VIR_DOMAIN_RUNNING_UNKNOWN: case VIR_DOMAIN_RUNNING_SAVE_CANCELED: case VIR_DOMAIN_RUNNING_BOOTED: diff --git a/tools/virsh-domain-event.c b/tools/virsh-domain-event.c index 6dbb64a655..2d6db5f881 100644 --- a/tools/virsh-domain-event.c +++ b/tools/virsh-domain-event.c @@ -93,7 +93,8 @@ VIR_ENUM_IMPL(virshDomainEventResumed, N_("Unpaused"), N_("Migrated"), N_("Snapshot"), - N_("Post-copy")); + N_("Post-copy"), + N_("Post-copy Error")); VIR_ENUM_DECL(virshDomainEventStopped); VIR_ENUM_IMPL(virshDomainEventStopped, diff --git a/tools/virsh-domain-monitor.c b/tools/virsh-domain-monitor.c index 246e8a16c0..dc5fe13e49 100644 --- a/tools/virsh-domain-monitor.c +++ b/tools/virsh-domain-monitor.c @@ -166,6 +166,7 @@ VIR_ENUM_IMPL(virshDomainRunningReason, N_("event wakeup"), N_("crashed"), N_("post-copy"), + N_("post-copy failed"), ); VIR_ENUM_DECL(virshDomainBlockedReason); -- 2.35.1
On Tue, May 10, 2022 at 17:20:26 +0200, Jiri Denemark wrote:
This new "post-copy failed" reason for the running state will be used on the destination host when post-copy migration fails while the domain is already running there.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- examples/c/misc/event-test.c | 3 +++ include/libvirt/libvirt-domain.h | 2 ++ src/conf/domain_conf.c | 1 + src/libvirt-domain.c | 15 ++++++++++----- src/qemu/qemu_domain.c | 3 +++ tools/virsh-domain-event.c | 3 ++- tools/virsh-domain-monitor.c | 1 + 7 files changed, 22 insertions(+), 6 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 37 +++++++++++++++++++++++++++++++++---- src/qemu/qemu_migration.h | 6 ++++-- src/qemu/qemu_process.c | 8 ++++---- 3 files changed, 41 insertions(+), 10 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index b735bdb391..a5c7a27124 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1577,14 +1577,19 @@ qemuMigrationSrcIsSafe(virDomainDef *def, void -qemuMigrationAnyPostcopyFailed(virQEMUDriver *driver, - virDomainObj *vm) +qemuMigrationSrcPostcopyFailed(virDomainObj *vm) { + qemuDomainObjPrivate *priv = vm->privateData; + virQEMUDriver *driver = priv->driver; virDomainState state; int reason; state = virDomainObjGetState(vm, &reason); + VIR_DEBUG("%s/%s", + virDomainStateTypeToString(state), + virDomainStateReasonToString(state, reason)); + if (state != VIR_DOMAIN_PAUSED && state != VIR_DOMAIN_RUNNING) return; @@ -1608,6 +1613,30 @@ qemuMigrationAnyPostcopyFailed(virQEMUDriver *driver, } +void +qemuMigrationDstPostcopyFailed(virDomainObj *vm) +{ + virDomainState state; + int reason; + + state = virDomainObjGetState(vm, &reason); + + VIR_DEBUG("%s/%s", + virDomainStateTypeToString(state), + virDomainStateReasonToString(state, reason)); + + if (state != VIR_DOMAIN_RUNNING || + reason == VIR_DOMAIN_RUNNING_POSTCOPY_FAILED) + return; + + VIR_WARN("Incoming migration of domain %s failed during post-copy; " + "leaving the domain running in a degraded mode", vm->def->name); + + virDomainObjSetState(vm, VIR_DOMAIN_RUNNING, + VIR_DOMAIN_RUNNING_POSTCOPY_FAILED); +} + + static int qemuMigrationSrcWaitForSpice(virDomainObj *vm) { @@ -3470,7 +3499,7 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && reason == VIR_DOMAIN_PAUSED_POSTCOPY) - qemuMigrationAnyPostcopyFailed(driver, vm); + qemuMigrationSrcPostcopyFailed(vm); else qemuMigrationSrcRestoreDomainState(driver, vm); @@ -5847,7 +5876,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, VIR_DOMAIN_EVENT_STOPPED_FAILED); virObjectEventStateQueue(driver->domainEventState, event); } else { - qemuMigrationAnyPostcopyFailed(driver, vm); + qemuMigrationDstPostcopyFailed(vm); } } diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index a8afa66119..c4e4228282 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -251,8 +251,10 @@ qemuMigrationDstRun(virQEMUDriver *driver, virDomainAsyncJob asyncJob); void -qemuMigrationAnyPostcopyFailed(virQEMUDriver *driver, - virDomainObj *vm); +qemuMigrationSrcPostcopyFailed(virDomainObj *vm); + +void +qemuMigrationDstPostcopyFailed(virDomainObj *vm); int qemuMigrationSrcFetchMirrorStats(virQEMUDriver *driver, diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 1925559fad..a3192a7196 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3482,7 +3482,7 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, * confirm success or failure yet; killing it seems safest unless * we already started guest CPUs or we were in post-copy mode */ if (postcopy) { - qemuMigrationAnyPostcopyFailed(driver, vm); + qemuMigrationDstPostcopyFailed(vm); } else if (state != VIR_DOMAIN_RUNNING) { VIR_DEBUG("Killing migrated domain %s", vm->def->name); return -1; @@ -3533,7 +3533,7 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, * post-copy mode */ if (postcopy) { - qemuMigrationAnyPostcopyFailed(driver, vm); + qemuMigrationSrcPostcopyFailed(vm); } else { VIR_DEBUG("Cancelling unfinished migration of domain %s", vm->def->name); @@ -3551,7 +3551,7 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, * post-copy mode we can use PAUSED_POSTCOPY_FAILED state for this */ if (postcopy) - qemuMigrationAnyPostcopyFailed(driver, vm); + qemuMigrationSrcPostcopyFailed(vm); break; case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: @@ -3560,7 +3560,7 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, * as broken in that case */ if (postcopy) { - qemuMigrationAnyPostcopyFailed(driver, vm); + qemuMigrationSrcPostcopyFailed(vm); } else { VIR_DEBUG("Resuming domain %s after failed migration", vm->def->name); -- 2.35.1
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start. I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 37 +++++++++++++++++++++++++++++++++---- src/qemu/qemu_migration.h | 6 ++++-- src/qemu/qemu_process.c | 8 ++++---- 3 files changed, 41 insertions(+), 10 deletions(-)
The code looks okay, but I think this needs more justification if it's to be accepted in this state.
On Wed, May 11, 2022 at 10:48:10 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start.
I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
It depends how much was already migrated. In practise the guest may easily stop running anyway :-) So yeah, it was a needless policy decision which is being removed now. But the important reason behind it, which I should have mention in the commit message is the difference between libvirt and QEMU migration state. When libvirt connection breaks (between daemons for p2p migration or between a client and daemons) we consider migration as broken from the API point of view and return failure. However, the migration may still be running just fine if the connection between QEMU processes remains working. And since we're in post-copy phase, the migration can even finish just fine without libvirt. So a half-broken VM may magically become a fully working migrated VM after our migration API reported a failure. Keeping the domain running makes this situation easier to handle :-) Jirka
On Wed, May 11, 2022 at 12:26:52 +0200, Jiri Denemark wrote:
On Wed, May 11, 2022 at 10:48:10 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start.
I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
It depends how much was already migrated. In practise the guest may easily stop running anyway :-)
Well, I'd consider that behaviour to be very bad actually, but given the caveats below ...
So yeah, it was a needless policy decision which is being removed now. But the important reason behind it, which I should have mention in the commit message is the difference between libvirt and QEMU migration state. When libvirt connection breaks (between daemons for p2p migration or between a client and daemons) we consider migration as broken from the API point of view and return failure. However, the migration may still be running just fine if the connection between QEMU processes remains working. And since we're in post-copy phase, the migration can even finish just fine without libvirt. So a half-broken VM may magically become a fully working migrated VM after our migration API reported a failure. Keeping the domain running makes this situation easier to handle :-)
I see. Additionally if e.g. libvirtd isn't running at all (but that ties to the "connection broken" scenario) we wouldn't even pause it. So the caveats were there in fact always albeit less probable. IMO it should be documented somewhere. I agree that this patch is okay though. I think we can add the docs in a follow-up. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 12:42:08PM +0200, Peter Krempa wrote:
On Wed, May 11, 2022 at 12:26:52 +0200, Jiri Denemark wrote:
On Wed, May 11, 2022 at 10:48:10 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start.
I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
It depends how much was already migrated. In practise the guest may easily stop running anyway :-)
Well, I'd consider that behaviour to be very bad actually, but given the caveats below ...
So yeah, it was a needless policy decision which is being removed now. But the important reason behind it, which I should have mention in the commit message is the difference between libvirt and QEMU migration state. When libvirt connection breaks (between daemons for p2p migration or between a client and daemons) we consider migration as broken from the API point of view and return failure. However, the migration may still be running just fine if the connection between QEMU processes remains working. And since we're in post-copy phase, the migration can even finish just fine without libvirt. So a half-broken VM may magically become a fully working migrated VM after our migration API reported a failure. Keeping the domain running makes this situation easier to handle :-)
I see. Additionally if e.g. libvirtd isn't running at all (but that ties to the "connection broken" scenario) we wouldn't even pause it.
So the caveats were there in fact always albeit less probable.
There are two very different scenarios here though. Strictly from the QEMU scenario, migration only fails if there's a problem with QEMU's migration connection. This scenario can impact the guest, because processes get selectively blocked, which can ultimately lead to application timeouts and errors. If there's a failure at the QEMU level we want the guest to be paused, so that interaction between apps in the guest is not impacted. On the libvirt side, if our own libvirtd conenction fails, this does not impact the guest or QEMU's migration connection (I'll ignore tunnelled mig here). So there is no problem with the guest continuing to execute, and even complete the migration from QEMU POV. For robustness we want a way for QEMU to autonomously pause the guest when post-copy fails, so that this pausing happens even if libvirt's connection has also failed concurrently. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, May 11, 2022 at 11:54:24 +0100, Daniel P. Berrangé wrote:
On Wed, May 11, 2022 at 12:42:08PM +0200, Peter Krempa wrote:
On Wed, May 11, 2022 at 12:26:52 +0200, Jiri Denemark wrote:
On Wed, May 11, 2022 at 10:48:10 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start.
I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
It depends how much was already migrated. In practise the guest may easily stop running anyway :-)
Well, I'd consider that behaviour to be very bad actually, but given the caveats below ...
So yeah, it was a needless policy decision which is being removed now. But the important reason behind it, which I should have mention in the commit message is the difference between libvirt and QEMU migration state. When libvirt connection breaks (between daemons for p2p migration or between a client and daemons) we consider migration as broken from the API point of view and return failure. However, the migration may still be running just fine if the connection between QEMU processes remains working. And since we're in post-copy phase, the migration can even finish just fine without libvirt. So a half-broken VM may magically become a fully working migrated VM after our migration API reported a failure. Keeping the domain running makes this situation easier to handle :-)
I see. Additionally if e.g. libvirtd isn't running at all (but that ties to the "connection broken" scenario) we wouldn't even pause it.
So the caveats were there in fact always albeit less probable.
There are two very different scenarios here though.
Strictly from the QEMU scenario, migration only fails if there's a problem with QEMU's migration connection. This scenario can impact the guest, because processes get selectively blocked, which can ultimately lead to application timeouts and errors. If there's a failure at the QEMU level we want the guest to be paused, so that interaction between apps in the guest is not impacted.
Well, the same applies for an external communication to a paused domain. But I agree the timeouts between processes running inside a single domain are more serious, as this is not really expected in contrast to external communication where timeouts and broken connections are pretty common.
On the libvirt side, if our own libvirtd conenction fails, this does not impact the guest or QEMU's migration connection (I'll ignore tunnelled mig here).
Luckily we don't even support post-copy with tunnelled migration :-)
For robustness we want a way for QEMU to autonomously pause the guest when post-copy fails, so that this pausing happens even if libvirt's connection has also failed concurrently.
Yeah, if QEMU gets modified to stop vCPUs in postcopy-pause migration state (and resume them in postcopy-recover), we can keep this code to keep the domain running. Otherwise we'd need to check the state of QEMU migration and decide whether to pause the domain or keep it running. And pause it in postcopy-pause handler. But we could get into trouble if QEMU gets to this state while libvirtd is not running. So modifying QEMU looks would be a more robust solution. Jirka
On Wed, May 11, 2022 at 10:48:10AM +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start.
I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
It isn't even safe from a theoretical standpoint though. Consider 2 processes in a guest that are communicating with each other. 1 gets blocked on a page rea due to broken post copy, but we leave the guest running. The other process sees no progress from the blocked process and/or hits time timeout and throws an error. As a result the guest application workload ends up completely dead, even if we later recover the the postcopy migration. Migration needs to strive to be transparent to the guest workload and IMHO having the guest workload selectively executing and selectively blocked is not transparent enough. It should require a user opt-in for this kind of behaviour. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Wed, May 11, 2022 at 11:39:29 +0100, Daniel P. Berrangé wrote:
On Wed, May 11, 2022 at 10:48:10AM +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start.
I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
It isn't even safe from a theoretical standpoint though.
Consider 2 processes in a guest that are communicating with each other. 1 gets blocked on a page rea due to broken post copy, but we leave the guest running. The other process sees no progress from the blocked process and/or hits time timeout and throws an error. As a result the guest application workload ends up completely dead, even if we later recover the the postcopy migration.
IMO you have to deal with this scenario in a reduced scope anyways when opting into using post-copy. Each page transfer is vastly slower than the comparable access into memory, so if the 'timeout' portion is implied to be on the same order of magnitde of memory access latency then your software is going to have a very bad time when being migrated in post-copy mode. If the link gets congested ... then it's even worse. Obviously when the migration link breaks and you are getting unbounded wait for a page access it's worse even for other types of APPs. Anyways, does qemu support pausing the destination if the connection breaks? If no, the second best thing we can do is to pause it by libvirt, but it will still have caveats e.g. when libvirt is not around to pause it.
On Wed, May 11, 2022 at 01:03:43PM +0200, Peter Krempa wrote:
On Wed, May 11, 2022 at 11:39:29 +0100, Daniel P. Berrangé wrote:
On Wed, May 11, 2022 at 10:48:10AM +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:27 +0200, Jiri Denemark wrote:
There's no need to artificially pause a domain when post-copy fails. The virtual CPUs may continue running, only the guest tasks that decide to read a page which has not been migrated yet will get blocked.
IMO not pausing the VM is a policy decision (same way as pausing it was though) and should be user-configurable at migration start.
I can see that users might want to prevent a half-broken VM from executing until it gets attention needed to fix it, even when it's safe from a "theoretical" standpoint.
It isn't even safe from a theoretical standpoint though.
Consider 2 processes in a guest that are communicating with each other. 1 gets blocked on a page rea due to broken post copy, but we leave the guest running. The other process sees no progress from the blocked process and/or hits time timeout and throws an error. As a result the guest application workload ends up completely dead, even if we later recover the the postcopy migration.
IMO you have to deal with this scenario in a reduced scope anyways when opting into using post-copy.
Each page transfer is vastly slower than the comparable access into memory, so if the 'timeout' portion is implied to be on the same order of magnitde of memory access latency then your software is going to have a very bad time when being migrated in post-copy mode. If the link gets congested ... then it's even worse.
That's very different likely order of magnitudes though. A "slow" page access in post-copy is $LOW seconds. A blocked process due to a broken post-copy connection is potentially $HIGH minutes long if the infra takes a long time to fix. A page access taking a seconds rather than microseconds isn't going to trip up many app level timeouts IMHO. A process blocked for many minutes is highly likely to trigger app level timeouts. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
The events would normally be triggered only if we're changing domain state. But most of the time the domain is already in the right state and we're just changing its substate from {PAUSED,RUNNING}_POSTCOPY to *_POSTCOPY_FAILED. Let's emit lifecycle events explicitly when post-copy migration fails to make the failure visible without polling. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a5c7a27124..3e3203471a 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1581,6 +1581,7 @@ qemuMigrationSrcPostcopyFailed(virDomainObj *vm) { qemuDomainObjPrivate *priv = vm->privateData; virQEMUDriver *driver = priv->driver; + virObjectEvent *event = NULL; virDomainState state; int reason; @@ -1609,6 +1610,9 @@ qemuMigrationSrcPostcopyFailed(virDomainObj *vm) } else { virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, VIR_DOMAIN_PAUSED_POSTCOPY_FAILED); + event = virDomainEventLifecycleNewFromObj(vm, VIR_DOMAIN_EVENT_SUSPENDED, + VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY_FAILED); + virObjectEventStateQueue(driver->domainEventState, event); } } @@ -1616,6 +1620,9 @@ qemuMigrationSrcPostcopyFailed(virDomainObj *vm) void qemuMigrationDstPostcopyFailed(virDomainObj *vm) { + qemuDomainObjPrivate *priv = vm->privateData; + virQEMUDriver *driver = priv->driver; + virObjectEvent *event = NULL; virDomainState state; int reason; @@ -1634,6 +1641,9 @@ qemuMigrationDstPostcopyFailed(virDomainObj *vm) virDomainObjSetState(vm, VIR_DOMAIN_RUNNING, VIR_DOMAIN_RUNNING_POSTCOPY_FAILED); + event = virDomainEventLifecycleNewFromObj(vm, VIR_DOMAIN_EVENT_RESUMED, + VIR_DOMAIN_EVENT_RESUMED_POSTCOPY_FAILED); + virObjectEventStateQueue(driver->domainEventState, event); } -- 2.35.1
On Tue, May 10, 2022 at 17:20:28 +0200, Jiri Denemark wrote:
The events would normally be triggered only if we're changing domain state. But most of the time the domain is already in the right state and we're just changing its substate from {PAUSED,RUNNING}_POSTCOPY to *_POSTCOPY_FAILED. Let's emit lifecycle events explicitly when post-copy migration fails to make the failure visible without polling.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The function never returns anything but zero. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domain.c | 5 ++--- src/qemu/qemu_domain.h | 4 ++-- src/qemu/qemu_migration.c | 3 +-- 3 files changed, 5 insertions(+), 7 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 51236b19f2..7ad84f7080 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -7436,7 +7436,7 @@ qemuDomainCheckDiskStartupPolicy(virQEMUDriver *driver, * The vm must be locked when any of the following cleanup functions is * called. */ -int +void qemuDomainCleanupAdd(virDomainObj *vm, qemuDomainCleanupCallback cb) { @@ -7447,7 +7447,7 @@ qemuDomainCleanupAdd(virDomainObj *vm, for (i = 0; i < priv->ncleanupCallbacks; i++) { if (priv->cleanupCallbacks[i] == cb) - return 0; + return; } VIR_RESIZE_N(priv->cleanupCallbacks, @@ -7455,7 +7455,6 @@ qemuDomainCleanupAdd(virDomainObj *vm, priv->ncleanupCallbacks, 1); priv->cleanupCallbacks[priv->ncleanupCallbacks++] = cb; - return 0; } void diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index c7125722e0..3ea0426c7e 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -746,8 +746,8 @@ int qemuDomainPrepareStorageSourceBlockdev(virDomainDiskDef *disk, qemuDomainObjPrivate *priv, virQEMUDriverConfig *cfg); -int qemuDomainCleanupAdd(virDomainObj *vm, - qemuDomainCleanupCallback cb); +void qemuDomainCleanupAdd(virDomainObj *vm, + qemuDomainCleanupCallback cb); void qemuDomainCleanupRemove(virDomainObj *vm, qemuDomainCleanupCallback cb); void qemuDomainCleanupRun(virQEMUDriver *driver, diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 3e3203471a..19337878e4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3098,8 +3098,7 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, VIR_WARN("Unable to encode migration cookie"); } - if (qemuDomainCleanupAdd(vm, qemuMigrationDstPrepareCleanup) < 0) - goto stopjob; + qemuDomainCleanupAdd(vm, qemuMigrationDstPrepareCleanup); if (!(flags & VIR_MIGRATE_OFFLINE)) { virDomainAuditStart(vm, "migrated", true); -- 2.35.1
On Tue, May 10, 2022 at 17:20:29 +0200, Jiri Denemark wrote:
The function never returns anything but zero.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domain.c | 5 ++--- src/qemu/qemu_domain.h | 4 ++-- src/qemu/qemu_migration.c | 3 +-- 3 files changed, 5 insertions(+), 7 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/conf/domain_conf.c | 10 ++++++++++ src/conf/domain_conf.h | 4 ++++ src/libvirt_private.syms | 1 + 3 files changed, 15 insertions(+) diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index f8c6b78c92..bae03422ff 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -29052,6 +29052,16 @@ virDomainObjGetState(virDomainObj *dom, int *reason) } +bool +virDomainObjIsFailedPostcopy(virDomainObj *dom) +{ + return ((dom->state.state == VIR_DOMAIN_PAUSED && + dom->state.reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED) || + (dom->state.state == VIR_DOMAIN_RUNNING && + dom->state.reason == VIR_DOMAIN_RUNNING_POSTCOPY_FAILED)); +} + + void virDomainObjSetState(virDomainObj *dom, virDomainState state, int reason) { diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index 88a411d00c..2b1f86932f 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -3867,6 +3867,10 @@ virDomainState virDomainObjGetState(virDomainObj *obj, int *reason) ATTRIBUTE_NONNULL(1); +bool +virDomainObjIsFailedPostcopy(virDomainObj *obj) + ATTRIBUTE_NONNULL(1); + virSecurityLabelDef * virDomainDefGetSecurityLabelDef(const virDomainDef *def, const char *model); diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 97bfca906b..070551b773 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -579,6 +579,7 @@ virDomainObjGetOneDef; virDomainObjGetOneDefState; virDomainObjGetPersistentDef; virDomainObjGetState; +virDomainObjIsFailedPostcopy; virDomainObjNew; virDomainObjParseFile; virDomainObjParseNode; -- 2.35.1
On Tue, May 10, 2022 at 17:20:30 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/conf/domain_conf.c | 10 ++++++++++ src/conf/domain_conf.h | 4 ++++ src/libvirt_private.syms | 1 + 3 files changed, 15 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/conf/domain_conf.c | 22 ++++++++++++++++++++++ src/conf/domain_conf.h | 4 ++++ src/libvirt_private.syms | 1 + 3 files changed, 27 insertions(+) diff --git a/src/conf/domain_conf.c b/src/conf/domain_conf.c index bae03422ff..e6adfb1d90 100644 --- a/src/conf/domain_conf.c +++ b/src/conf/domain_conf.c @@ -29062,6 +29062,28 @@ virDomainObjIsFailedPostcopy(virDomainObj *dom) } +bool +virDomainObjIsPostcopy(virDomainObj *dom, + virDomainJobOperation op) +{ + if (op != VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN && + op != VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT) + return false; + + if (op == VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN) { + return (dom->state.state == VIR_DOMAIN_PAUSED && + dom->state.reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED) || + (dom->state.state == VIR_DOMAIN_RUNNING && + (dom->state.reason == VIR_DOMAIN_RUNNING_POSTCOPY || + dom->state.reason == VIR_DOMAIN_RUNNING_POSTCOPY_FAILED)); + } + + return dom->state.state == VIR_DOMAIN_PAUSED && + (dom->state.reason == VIR_DOMAIN_PAUSED_POSTCOPY || + dom->state.reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED); +} + + void virDomainObjSetState(virDomainObj *dom, virDomainState state, int reason) { diff --git a/src/conf/domain_conf.h b/src/conf/domain_conf.h index 2b1f86932f..3f2839bb82 100644 --- a/src/conf/domain_conf.h +++ b/src/conf/domain_conf.h @@ -3870,6 +3870,10 @@ virDomainObjGetState(virDomainObj *obj, int *reason) bool virDomainObjIsFailedPostcopy(virDomainObj *obj) ATTRIBUTE_NONNULL(1); +bool +virDomainObjIsPostcopy(virDomainObj *dom, + virDomainJobOperation op) + ATTRIBUTE_NONNULL(1); virSecurityLabelDef * virDomainDefGetSecurityLabelDef(const virDomainDef *def, const char *model); diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 070551b773..84e34c1648 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -580,6 +580,7 @@ virDomainObjGetOneDefState; virDomainObjGetPersistentDef; virDomainObjGetState; virDomainObjIsFailedPostcopy; +virDomainObjIsPostcopy; virDomainObjNew; virDomainObjParseFile; virDomainObjParseNode; -- 2.35.1
On Tue, May 10, 2022 at 17:20:31 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/conf/domain_conf.c | 22 ++++++++++++++++++++++ src/conf/domain_conf.h | 4 ++++ src/libvirt_private.syms | 1 +
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The function can be used as a callback for qemuDomainCleanupAdd to automatically clean up a migration job when a domain is destroyed. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 25 +++++++++++++++++++++++++ src/qemu/qemu_process.h | 3 +++ 2 files changed, 28 insertions(+) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index a3192a7196..dba3ad8c87 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3431,6 +3431,31 @@ qemuProcessUpdateState(virQEMUDriver *driver, virDomainObj *vm) return 0; } + +void +qemuProcessCleanupMigrationJob(virQEMUDriver *driver, + virDomainObj *vm) +{ + qemuDomainObjPrivate *priv = vm->privateData; + virDomainState state; + int reason; + + state = virDomainObjGetState(vm, &reason); + + VIR_DEBUG("driver=%p, vm=%s, asyncJob=%s, state=%s, reason=%s", + driver, vm->def->name, + virDomainAsyncJobTypeToString(priv->job.asyncJob), + virDomainStateTypeToString(state), + virDomainStateReasonToString(state, reason)); + + if (priv->job.asyncJob != VIR_ASYNC_JOB_MIGRATION_IN && + priv->job.asyncJob != VIR_ASYNC_JOB_MIGRATION_OUT) + return; + + qemuDomainObjDiscardAsyncJob(vm); +} + + static int qemuProcessRecoverMigrationIn(virQEMUDriver *driver, virDomainObj *vm, diff --git a/src/qemu/qemu_process.h b/src/qemu/qemu_process.h index f81bfd930a..73e6110eee 100644 --- a/src/qemu/qemu_process.h +++ b/src/qemu/qemu_process.h @@ -244,3 +244,6 @@ G_DEFINE_AUTOPTR_CLEANUP_FUNC(qemuProcessQMP, qemuProcessQMPFree); int qemuProcessQMPStart(qemuProcessQMP *proc); bool qemuProcessRebootAllowed(const virDomainDef *def); + +void qemuProcessCleanupMigrationJob(virQEMUDriver *driver, + virDomainObj *vm); -- 2.35.1
On Tue, May 10, 2022 at 17:20:32 +0200, Jiri Denemark wrote:
The function can be used as a callback for qemuDomainCleanupAdd to automatically clean up a migration job when a domain is destroyed.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 25 +++++++++++++++++++++++++ src/qemu/qemu_process.h | 3 +++ 2 files changed, 28 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
It is used for saving job out of domain object. Just like virErrorPreserveLast is used for errors. Let's make the naming consistent as Restore would suggest different semantics. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 14 ++++++++++++-- src/qemu/qemu_domainjob.h | 4 ++-- src/qemu/qemu_process.c | 2 +- 3 files changed, 15 insertions(+), 5 deletions(-) diff --git a/src/qemu/qemu_domainjob.c b/src/qemu/qemu_domainjob.c index cb20b798f7..1e5724b505 100644 --- a/src/qemu/qemu_domainjob.c +++ b/src/qemu/qemu_domainjob.c @@ -214,9 +214,19 @@ qemuDomainObjResetAsyncJob(qemuDomainJobObj *job) job->cb->resetJobPrivate(job->privateData); } + +/** + * qemuDomainObjPreserveJob + * @param obj domain with a job that needs to be preserved + * @param job structure where to store job details from @obj + * + * Saves the current job details from @obj to @job and resets the job in @obj. + * + * Returns 0 on success, -1 on failure. + */ int -qemuDomainObjRestoreJob(virDomainObj *obj, - qemuDomainJobObj *job) +qemuDomainObjPreserveJob(virDomainObj *obj, + qemuDomainJobObj *job) { qemuDomainObjPrivate *priv = obj->privateData; diff --git a/src/qemu/qemu_domainjob.h b/src/qemu/qemu_domainjob.h index f67eace36c..2bbccf6329 100644 --- a/src/qemu/qemu_domainjob.h +++ b/src/qemu/qemu_domainjob.h @@ -158,8 +158,8 @@ void qemuDomainObjSetJobPhase(virDomainObj *obj, int phase); void qemuDomainObjSetAsyncJobMask(virDomainObj *obj, unsigned long long allowedJobs); -int qemuDomainObjRestoreJob(virDomainObj *obj, - qemuDomainJobObj *job); +int qemuDomainObjPreserveJob(virDomainObj *obj, + qemuDomainJobObj *job); void qemuDomainObjDiscardAsyncJob(virDomainObj *obj); void qemuDomainObjReleaseAsyncJob(virDomainObj *obj); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index dba3ad8c87..db0d9935be 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -8775,7 +8775,7 @@ qemuProcessReconnect(void *opaque) cfg = virQEMUDriverGetConfig(driver); priv = obj->privateData; - qemuDomainObjRestoreJob(obj, &oldjob); + qemuDomainObjPreserveJob(obj, &oldjob); if (oldjob.asyncJob == VIR_ASYNC_JOB_MIGRATION_IN) stopFlags |= VIR_QEMU_PROCESS_STOP_MIGRATED; if (oldjob.asyncJob == VIR_ASYNC_JOB_BACKUP && priv->backup) -- 2.35.1
On Tue, May 10, 2022 at 17:20:33 +0200, Jiri Denemark wrote:
It is used for saving job out of domain object. Just like virErrorPreserveLast is used for errors. Let's make the naming consistent as Restore would suggest different semantics.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 14 ++++++++++++-- src/qemu/qemu_domainjob.h | 4 ++-- src/qemu/qemu_process.c | 2 +- 3 files changed, 15 insertions(+), 5 deletions(-)
I think this could be even moved to qemu_process.c, since it's used only in one place. Anyways: Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The code for setting up a previously active backup job in qemuProcessRecoverJob is generalized into a dedicated function so that it can be later reused in other places. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 35 +++++++++++++++++++++++++++++++++++ src/qemu/qemu_domainjob.h | 8 ++++++++ src/qemu/qemu_process.c | 29 +++++++---------------------- 3 files changed, 50 insertions(+), 22 deletions(-) diff --git a/src/qemu/qemu_domainjob.c b/src/qemu/qemu_domainjob.c index 1e5724b505..1f82457bd4 100644 --- a/src/qemu/qemu_domainjob.c +++ b/src/qemu/qemu_domainjob.c @@ -249,6 +249,41 @@ qemuDomainObjPreserveJob(virDomainObj *obj, return 0; } + +void +qemuDomainObjRestoreAsyncJob(virDomainObj *vm, + virDomainAsyncJob asyncJob, + int phase, + virDomainJobOperation operation, + qemuDomainJobStatsType statsType, + virDomainJobStatus status, + unsigned long long allowedJobs) +{ + qemuDomainObjPrivate *priv = vm->privateData; + qemuDomainJobObj *job = &priv->job; + unsigned long long now; + + VIR_DEBUG("Restoring %s async job for domain %s", + virDomainAsyncJobTypeToString(asyncJob), vm->def->name); + + ignore_value(virTimeMillisNow(&now)); + + priv->job.jobsQueued++; + job->asyncJob = asyncJob; + job->phase = phase; + job->asyncOwnerAPI = g_strdup(virThreadJobGet()); + job->asyncStarted = now; + + qemuDomainObjSetAsyncJobMask(vm, allowedJobs); + + job->current = virDomainJobDataInit(&qemuJobDataPrivateDataCallbacks); + qemuDomainJobSetStatsType(priv->job.current, statsType); + job->current->operation = operation; + job->current->status = status; + job->current->started = now; +} + + void qemuDomainObjClearJob(qemuDomainJobObj *job) { diff --git a/src/qemu/qemu_domainjob.h b/src/qemu/qemu_domainjob.h index 2bbccf6329..069bb9f8cb 100644 --- a/src/qemu/qemu_domainjob.h +++ b/src/qemu/qemu_domainjob.h @@ -160,6 +160,14 @@ void qemuDomainObjSetAsyncJobMask(virDomainObj *obj, unsigned long long allowedJobs); int qemuDomainObjPreserveJob(virDomainObj *obj, qemuDomainJobObj *job); +void +qemuDomainObjRestoreAsyncJob(virDomainObj *vm, + virDomainAsyncJob asyncJob, + int phase, + virDomainJobOperation operation, + qemuDomainJobStatsType statsType, + virDomainJobStatus status, + unsigned long long allowedJobs); void qemuDomainObjDiscardAsyncJob(virDomainObj *obj); void qemuDomainObjReleaseAsyncJob(virDomainObj *obj); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index db0d9935be..590e989126 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3628,7 +3628,6 @@ qemuProcessRecoverJob(virQEMUDriver *driver, qemuDomainObjPrivate *priv = vm->privateData; virDomainState state; int reason; - unsigned long long now; state = virDomainObjGetState(vm, &reason); @@ -3685,28 +3684,14 @@ qemuProcessRecoverJob(virQEMUDriver *driver, break; case VIR_ASYNC_JOB_BACKUP: - ignore_value(virTimeMillisNow(&now)); - /* Restore the config of the async job which is not persisted */ - priv->job.jobsQueued++; - priv->job.asyncJob = VIR_ASYNC_JOB_BACKUP; - priv->job.asyncOwnerAPI = g_strdup(virThreadJobGet()); - priv->job.asyncStarted = now; - - qemuDomainObjSetAsyncJobMask(vm, (VIR_JOB_DEFAULT_MASK | - JOB_MASK(VIR_JOB_SUSPEND) | - JOB_MASK(VIR_JOB_MODIFY))); - - /* We reset the job parameters for backup so that the job will look - * active. This is possible because we are able to recover the state - * of blockjobs and also the backup job allows all sub-job types */ - priv->job.current = virDomainJobDataInit(&qemuJobDataPrivateDataCallbacks); - - qemuDomainJobSetStatsType(priv->job.current, - QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP); - priv->job.current->operation = VIR_DOMAIN_JOB_OPERATION_BACKUP; - priv->job.current->status = VIR_DOMAIN_JOB_STATUS_ACTIVE; - priv->job.current->started = now; + qemuDomainObjRestoreAsyncJob(vm, VIR_ASYNC_JOB_BACKUP, 0, + VIR_DOMAIN_JOB_OPERATION_BACKUP, + QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP, + VIR_DOMAIN_JOB_STATUS_ACTIVE, + (VIR_JOB_DEFAULT_MASK | + JOB_MASK(VIR_JOB_SUSPEND) | + JOB_MASK(VIR_JOB_MODIFY))); break; case VIR_ASYNC_JOB_NONE: -- 2.35.1
On Tue, May 10, 2022 at 05:20:34PM +0200, Jiri Denemark wrote:
The code for setting up a previously active backup job in qemuProcessRecoverJob is generalized into a dedicated function so that it can be later reused in other places.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 35 +++++++++++++++++++++++++++++++++++ src/qemu/qemu_domainjob.h | 8 ++++++++ src/qemu/qemu_process.c | 29 +++++++---------------------- 3 files changed, 50 insertions(+), 22 deletions(-)
diff --git a/src/qemu/qemu_domainjob.c b/src/qemu/qemu_domainjob.c index 1e5724b505..1f82457bd4 100644 --- a/src/qemu/qemu_domainjob.c +++ b/src/qemu/qemu_domainjob.c @@ -249,6 +249,41 @@ qemuDomainObjPreserveJob(virDomainObj *obj, return 0; }
+ +void +qemuDomainObjRestoreAsyncJob(virDomainObj *vm, + virDomainAsyncJob asyncJob, + int phase, + virDomainJobOperation operation, + qemuDomainJobStatsType statsType, + virDomainJobStatus status, + unsigned long long allowedJobs) +{ + qemuDomainObjPrivate *priv = vm->privateData; + qemuDomainJobObj *job = &priv->job; + unsigned long long now; + + VIR_DEBUG("Restoring %s async job for domain %s", + virDomainAsyncJobTypeToString(asyncJob), vm->def->name); + + ignore_value(virTimeMillisNow(&now)); + + priv->job.jobsQueued++;
s/priv->job./job->/
+ job->asyncJob = asyncJob; + job->phase = phase; + job->asyncOwnerAPI = g_strdup(virThreadJobGet()); + job->asyncStarted = now; + + qemuDomainObjSetAsyncJobMask(vm, allowedJobs); + + job->current = virDomainJobDataInit(&qemuJobDataPrivateDataCallbacks); + qemuDomainJobSetStatsType(priv->job.current, statsType); + job->current->operation = operation; + job->current->status = status; + job->current->started = now; +} + + void qemuDomainObjClearJob(qemuDomainJobObj *job)
When post-copy migration fails, we can't just abort the migration and resume the domain on the source host as it is already running on the destination host and no host has a complete state of the domain memory. Instead of the current approach of just marking the domain on both ends as paused/running with a post-copy failed sub state, we will keep the migration job active (even though the migration API will return failure) so that the state is more visible and we can better control what APIs can be called on the domains and even allow for resuming the migration. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 94 ++++++++++++++++++++++++++++----------- 1 file changed, 68 insertions(+), 26 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 19337878e4..532a9300b6 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2249,10 +2249,17 @@ qemuMigrationSrcCleanup(virDomainObj *vm, VIR_WARN("Migration of domain %s finished but we don't know if the" " domain was successfully started on destination or not", vm->def->name); - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, - jobPriv->migParams, priv->job.apiFlags); - /* clear the job and let higher levels decide what to do */ - qemuMigrationJobFinish(vm); + + if (virDomainObjIsPostcopy(vm, VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT)) { + qemuMigrationSrcPostcopyFailed(vm); + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + } else { + qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + jobPriv->migParams, priv->job.apiFlags); + /* clear the job and let higher levels decide what to do */ + qemuMigrationJobFinish(vm); + } break; case QEMU_MIGRATION_PHASE_PERFORM3: @@ -3427,6 +3434,7 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobPrivate *jobPriv = priv->job.privateData; virDomainJobData *jobData = NULL; + qemuMigrationJobPhase phase; VIR_DEBUG("driver=%p, vm=%p, cookiein=%s, cookieinlen=%d, " "flags=0x%x, retcode=%d", @@ -3435,10 +3443,17 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, virCheckFlags(QEMU_MIGRATION_FLAGS, -1); - qemuMigrationJobSetPhase(vm, - retcode == 0 - ? QEMU_MIGRATION_PHASE_CONFIRM3 - : QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED); + /* Keep the original migration phase in case post-copy failed as the job + * will stay active even though migration API finishes with an error. + */ + if (virDomainObjIsFailedPostcopy(vm)) + phase = priv->job.phase; + else if (retcode == 0) + phase = QEMU_MIGRATION_PHASE_CONFIRM3; + else + phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; + + qemuMigrationJobSetPhase(vm, phase); if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, cookiein, cookieinlen, @@ -3507,13 +3522,14 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, virErrorRestore(&orig_err); if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && - reason == VIR_DOMAIN_PAUSED_POSTCOPY) + reason == VIR_DOMAIN_PAUSED_POSTCOPY) { qemuMigrationSrcPostcopyFailed(vm); - else + } else if (!virDomainObjIsFailedPostcopy(vm)) { qemuMigrationSrcRestoreDomainState(driver, vm); - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, - jobPriv->migParams, priv->job.apiFlags); + qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + jobPriv->migParams, priv->job.apiFlags); + } qemuDomainSaveStatus(vm); } @@ -3531,12 +3547,18 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, { qemuMigrationJobPhase phase; g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver); + qemuDomainObjPrivate *priv = vm->privateData; int ret = -1; if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_OUT)) goto cleanup; - if (cancelled) + /* Keep the original migration phase in case post-copy failed as the job + * will stay active even though migration API finishes with an error. + */ + if (virDomainObjIsFailedPostcopy(vm)) + phase = priv->job.phase; + else if (cancelled) phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; else phase = QEMU_MIGRATION_PHASE_CONFIRM3; @@ -3549,7 +3571,13 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, cookiein, cookieinlen, flags, cancelled); - qemuMigrationJobFinish(vm); + if (virDomainObjIsFailedPostcopy(vm)) { + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + } else { + qemuMigrationJobFinish(vm); + } + if (!virDomainObjIsActive(vm)) { if (!cancelled && ret == 0 && flags & VIR_MIGRATE_UNDEFINE_SOURCE) { virDomainDeleteConfig(cfg->configDir, cfg->autostartDir, vm); @@ -5365,16 +5393,22 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, if (ret < 0) virErrorPreserveLast(&orig_err); - /* v2 proto has no confirm phase so we need to reset migration parameters - * here - */ - if (!v3proto && ret < 0) - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, - jobPriv->migParams, priv->job.apiFlags); + if (virDomainObjIsFailedPostcopy(vm)) { + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + } else { + /* v2 proto has no confirm phase so we need to reset migration parameters + * here + */ + if (!v3proto && ret < 0) + qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + jobPriv->migParams, priv->job.apiFlags); - qemuMigrationSrcRestoreDomainState(driver, vm); + qemuMigrationSrcRestoreDomainState(driver, vm); + + qemuMigrationJobFinish(vm); + } - qemuMigrationJobFinish(vm); if (!virDomainObjIsActive(vm) && ret == 0) { if (flags & VIR_MIGRATE_UNDEFINE_SOURCE) { virDomainDeleteConfig(cfg->configDir, cfg->autostartDir, vm); @@ -5445,11 +5479,12 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto endjob; endjob: - if (ret < 0) { + if (ret < 0 && !virDomainObjIsFailedPostcopy(vm)) { qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, jobPriv->migParams, priv->job.apiFlags); qemuMigrationJobFinish(vm); } else { + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); } @@ -5910,10 +5945,17 @@ qemuMigrationDstFinish(virQEMUDriver *driver, g_clear_pointer(&priv->job.completed, virDomainJobDataFree); } - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, - jobPriv->migParams, priv->job.apiFlags); + if (virDomainObjIsFailedPostcopy(vm)) { + qemuProcessAutoDestroyRemove(driver, vm); + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + } else { + qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + jobPriv->migParams, priv->job.apiFlags); + + qemuMigrationJobFinish(vm); + } - qemuMigrationJobFinish(vm); if (!virDomainObjIsActive(vm)) qemuDomainRemoveInactive(driver, vm); -- 2.35.1
On Tue, May 10, 2022 at 17:20:35 +0200, Jiri Denemark wrote:
When post-copy migration fails, we can't just abort the migration and resume the domain on the source host as it is already running on the destination host and no host has a complete state of the domain memory. Instead of the current approach of just marking the domain on both ends as paused/running with a post-copy failed sub state, we will keep the migration job active (even though the migration API will return failure) so that the state is more visible and we can better control what APIs can be called on the domains and even allow for resuming the migration.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 94 ++++++++++++++++++++++++++++----------- 1 file changed, 68 insertions(+), 26 deletions(-)
@@ -5445,11 +5479,12 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto endjob;
endjob: - if (ret < 0) { + if (ret < 0 && !virDomainObjIsFailedPostcopy(vm)) { qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, jobPriv->migParams, priv->job.apiFlags); qemuMigrationJobFinish(vm); } else { + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); }
This logic change is a bit obscure and IMO would benefit from a comment stating that we want to continue all post-copy migration jobs and all successful other migrations. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 14:35:22 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:35 +0200, Jiri Denemark wrote:
When post-copy migration fails, we can't just abort the migration and resume the domain on the source host as it is already running on the destination host and no host has a complete state of the domain memory. Instead of the current approach of just marking the domain on both ends as paused/running with a post-copy failed sub state, we will keep the migration job active (even though the migration API will return failure) so that the state is more visible and we can better control what APIs can be called on the domains and even allow for resuming the migration.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 94 ++++++++++++++++++++++++++++----------- 1 file changed, 68 insertions(+), 26 deletions(-)
@@ -5445,11 +5479,12 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto endjob;
endjob: - if (ret < 0) { + if (ret < 0 && !virDomainObjIsFailedPostcopy(vm)) { qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, jobPriv->migParams, priv->job.apiFlags); qemuMigrationJobFinish(vm); } else { + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); }
This logic change is a bit obscure and IMO would benefit from a comment stating that we want to continue all post-copy migration jobs and all successful other migrations.
I'm not sure what is so special about this hunk to make it look obscure. It's the same change done everywhere in this patch, which is about continuing the job if it succeeded (existing) or failed in post-copy (new). Jirka
When migration fails after it already switched to post-copy phase on the source, but early enough that we haven't called Finish on the destination yet, we know the vCPUs were not started on the destination and the source host still has a complete state of the domain. Thus we can just ignore the fact post-copy phase started and normally abort the migration and resume vCPUs on the source. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 9 +++++++++ src/qemu/qemu_process.c | 20 ++++++++------------ 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 532a9300b6..e892a09885 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4442,6 +4442,15 @@ qemuMigrationSrcRun(virQEMUDriver *driver, virErrorPreserveLast(&orig_err); if (virDomainObjIsActive(vm)) { + int reason; + virDomainState state = virDomainObjGetState(vm, &reason); + + if (state == VIR_DOMAIN_PAUSED && reason == VIR_DOMAIN_PAUSED_POSTCOPY) { + VIR_DEBUG("Aborting failed post-copy migration as the destination " + "is not running yet"); + virDomainObjSetState(vm, state, VIR_DOMAIN_PAUSED_MIGRATION); + } + if (cancel && priv->job.current->status != VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED && qemuDomainObjEnterMonitorAsync(driver, vm, diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 590e989126..e83668e088 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3554,20 +3554,16 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, case QEMU_MIGRATION_PHASE_PERFORM2: case QEMU_MIGRATION_PHASE_PERFORM3: /* migration is still in progress, let's cancel it and resume the - * domain; however we can only do that before migration enters - * post-copy mode + * domain; we can do so even in post-copy phase as the domain was not + * resumed on the destination host yet */ - if (postcopy) { - qemuMigrationSrcPostcopyFailed(vm); - } else { - VIR_DEBUG("Cancelling unfinished migration of domain %s", - vm->def->name); - if (qemuMigrationSrcCancel(driver, vm) < 0) { - VIR_WARN("Could not cancel ongoing migration of domain %s", - vm->def->name); - } - resume = true; + VIR_DEBUG("Cancelling unfinished migration of domain %s", + vm->def->name); + if (qemuMigrationSrcCancel(driver, vm) < 0) { + VIR_WARN("Could not cancel ongoing migration of domain %s", + vm->def->name); } + resume = true; break; case QEMU_MIGRATION_PHASE_PERFORM3_DONE: -- 2.35.1
On Tue, May 10, 2022 at 17:20:36 +0200, Jiri Denemark wrote:
When migration fails after it already switched to post-copy phase on the source, but early enough that we haven't called Finish on the destination yet, we know the vCPUs were not started on the destination and the source host still has a complete state of the domain. Thus we can just ignore the fact post-copy phase started and normally abort the migration and resume vCPUs on the source.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 9 +++++++++ src/qemu/qemu_process.c | 20 ++++++++------------ 2 files changed, 17 insertions(+), 12 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 532a9300b6..e892a09885 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4442,6 +4442,15 @@ qemuMigrationSrcRun(virQEMUDriver *driver, virErrorPreserveLast(&orig_err);
if (virDomainObjIsActive(vm)) { + int reason; + virDomainState state = virDomainObjGetState(vm, &reason); + + if (state == VIR_DOMAIN_PAUSED && reason == VIR_DOMAIN_PAUSED_POSTCOPY) { + VIR_DEBUG("Aborting failed post-copy migration as the destination " + "is not running yet");
Preferrably no linebreaks in new diagnostic strings. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Since we keep the migration job active when post-copy migration fails, we need to restore it when reconnecting to running domains. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 128 ++++++++++++++++++++++++++++++---------- 1 file changed, 96 insertions(+), 32 deletions(-) diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index e83668e088..3d73c716f1 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3456,20 +3456,48 @@ qemuProcessCleanupMigrationJob(virQEMUDriver *driver, } +static void +qemuProcessRestoreMigrationJob(virDomainObj *vm, + qemuDomainJobObj *job) +{ + qemuDomainObjPrivate *priv = vm->privateData; + qemuDomainJobPrivate *jobPriv = job->privateData; + virDomainJobOperation op; + unsigned long long allowedJobs; + + if (job->asyncJob == VIR_ASYNC_JOB_MIGRATION_IN) { + op = VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN; + allowedJobs = VIR_JOB_NONE; + } else { + op = VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT; + allowedJobs = VIR_JOB_DEFAULT_MASK | JOB_MASK(VIR_JOB_MIGRATION_OP); + } + + qemuDomainObjRestoreAsyncJob(vm, job->asyncJob, job->phase, op, + QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION, + VIR_DOMAIN_JOB_STATUS_PAUSED, + allowedJobs); + + job->privateData = g_steal_pointer(&priv->job.privateData); + priv->job.privateData = jobPriv; + priv->job.apiFlags = job->apiFlags; + + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); +} + + +/* + * Returns + * -1 on error, the domain will be killed, + * 0 the domain should remain running with the migration job discarded, + * 1 the daemon was restarted during post-copy phase + */ static int qemuProcessRecoverMigrationIn(virQEMUDriver *driver, virDomainObj *vm, - const qemuDomainJobObj *job, - virDomainState state, - int reason) + qemuDomainJobObj *job, + virDomainState state) { - - qemuDomainJobPrivate *jobPriv = job->privateData; - bool postcopy = (state == VIR_DOMAIN_PAUSED && - reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED) || - (state == VIR_DOMAIN_RUNNING && - reason == VIR_DOMAIN_RUNNING_POSTCOPY); - VIR_DEBUG("Active incoming migration in phase %s", qemuMigrationJobPhaseTypeToString(job->phase)); @@ -3506,32 +3534,37 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, /* migration finished, we started resuming the domain but didn't * confirm success or failure yet; killing it seems safest unless * we already started guest CPUs or we were in post-copy mode */ - if (postcopy) { + if (virDomainObjIsPostcopy(vm, VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN)) { qemuMigrationDstPostcopyFailed(vm); - } else if (state != VIR_DOMAIN_RUNNING) { + return 1; + } + + if (state != VIR_DOMAIN_RUNNING) { VIR_DEBUG("Killing migrated domain %s", vm->def->name); return -1; } break; } - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_NONE, - jobPriv->migParams, job->apiFlags); return 0; } + +/* + * Returns + * -1 on error, the domain will be killed, + * 0 the domain should remain running with the migration job discarded, + * 1 the daemon was restarted during post-copy phase + */ static int qemuProcessRecoverMigrationOut(virQEMUDriver *driver, virDomainObj *vm, - const qemuDomainJobObj *job, + qemuDomainJobObj *job, virDomainState state, int reason, unsigned int *stopFlags) { - qemuDomainJobPrivate *jobPriv = job->privateData; - bool postcopy = state == VIR_DOMAIN_PAUSED && - (reason == VIR_DOMAIN_PAUSED_POSTCOPY || - reason == VIR_DOMAIN_PAUSED_POSTCOPY_FAILED); + bool postcopy = virDomainObjIsPostcopy(vm, VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT); bool resume = false; VIR_DEBUG("Active outgoing migration in phase %s", @@ -3571,8 +3604,10 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, * of Finish3 step; third party needs to check what to do next; in * post-copy mode we can use PAUSED_POSTCOPY_FAILED state for this */ - if (postcopy) + if (postcopy) { qemuMigrationSrcPostcopyFailed(vm); + return 1; + } break; case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: @@ -3582,11 +3617,12 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, */ if (postcopy) { qemuMigrationSrcPostcopyFailed(vm); - } else { - VIR_DEBUG("Resuming domain %s after failed migration", - vm->def->name); - resume = true; + return 1; } + + VIR_DEBUG("Resuming domain %s after failed migration", + vm->def->name); + resume = true; break; case QEMU_MIGRATION_PHASE_CONFIRM3: @@ -3610,15 +3646,49 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, } } + return 0; +} + + +static int +qemuProcessRecoverMigration(virQEMUDriver *driver, + virDomainObj *vm, + qemuDomainJobObj *job, + unsigned int *stopFlags) +{ + qemuDomainJobPrivate *jobPriv = job->privateData; + virDomainState state; + int reason; + int rc; + + state = virDomainObjGetState(vm, &reason); + + if (job->asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) { + rc = qemuProcessRecoverMigrationOut(driver, vm, job, + state, reason, stopFlags); + } else { + rc = qemuProcessRecoverMigrationIn(driver, vm, job, state); + } + + if (rc < 0) + return -1; + + if (rc > 0) { + qemuProcessRestoreMigrationJob(vm, job); + return 0; + } + qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_NONE, jobPriv->migParams, job->apiFlags); + return 0; } + static int qemuProcessRecoverJob(virQEMUDriver *driver, virDomainObj *vm, - const qemuDomainJobObj *job, + qemuDomainJobObj *job, unsigned int *stopFlags) { qemuDomainObjPrivate *priv = vm->privateData; @@ -3636,14 +3706,8 @@ qemuProcessRecoverJob(virQEMUDriver *driver, switch (job->asyncJob) { case VIR_ASYNC_JOB_MIGRATION_OUT: - if (qemuProcessRecoverMigrationOut(driver, vm, job, - state, reason, stopFlags) < 0) - return -1; - break; - case VIR_ASYNC_JOB_MIGRATION_IN: - if (qemuProcessRecoverMigrationIn(driver, vm, job, - state, reason) < 0) + if (qemuProcessRecoverMigration(driver, vm, job, stopFlags) < 0) return -1; break; -- 2.35.1
On Tue, May 10, 2022 at 17:20:37 +0200, Jiri Denemark wrote:
Since we keep the migration job active when post-copy migration fails, we need to restore it when reconnecting to running domains.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_process.c | 128 ++++++++++++++++++++++++++++++---------- 1 file changed, 96 insertions(+), 32 deletions(-)
The git diff algorithm did a very bad job in terms of making this patch reviewable from just the diffs. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Jobs that are supposed to remain active even when libvirt daemon restarts were reported as started at the time the daemon was restarted. This is not very helpful, we should restore the original timestamp. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 20 +++++++++++++------ src/qemu/qemu_domainjob.h | 1 + src/qemu/qemu_process.c | 4 +++- .../migration-in-params-in.xml | 2 +- .../migration-out-nbd-bitmaps-in.xml | 2 +- .../migration-out-nbd-out.xml | 2 +- .../migration-out-nbd-tls-out.xml | 2 +- .../migration-out-params-in.xml | 2 +- 8 files changed, 23 insertions(+), 12 deletions(-) diff --git a/src/qemu/qemu_domainjob.c b/src/qemu/qemu_domainjob.c index 1f82457bd4..8e8d229afe 100644 --- a/src/qemu/qemu_domainjob.c +++ b/src/qemu/qemu_domainjob.c @@ -235,6 +235,7 @@ qemuDomainObjPreserveJob(virDomainObj *obj, job->owner = priv->job.owner; job->asyncJob = priv->job.asyncJob; job->asyncOwner = priv->job.asyncOwner; + job->asyncStarted = priv->job.asyncStarted; job->phase = priv->job.phase; job->privateData = g_steal_pointer(&priv->job.privateData); job->apiFlags = priv->job.apiFlags; @@ -254,6 +255,7 @@ void qemuDomainObjRestoreAsyncJob(virDomainObj *vm, virDomainAsyncJob asyncJob, int phase, + unsigned long long started, virDomainJobOperation operation, qemuDomainJobStatsType statsType, virDomainJobStatus status, @@ -261,18 +263,15 @@ qemuDomainObjRestoreAsyncJob(virDomainObj *vm, { qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobObj *job = &priv->job; - unsigned long long now; VIR_DEBUG("Restoring %s async job for domain %s", virDomainAsyncJobTypeToString(asyncJob), vm->def->name); - ignore_value(virTimeMillisNow(&now)); - priv->job.jobsQueued++; job->asyncJob = asyncJob; job->phase = phase; job->asyncOwnerAPI = g_strdup(virThreadJobGet()); - job->asyncStarted = now; + job->asyncStarted = started; qemuDomainObjSetAsyncJobMask(vm, allowedJobs); @@ -280,7 +279,7 @@ qemuDomainObjRestoreAsyncJob(virDomainObj *vm, qemuDomainJobSetStatsType(priv->job.current, statsType); job->current->operation = operation; job->current->status = status; - job->current->started = now; + job->current->started = started; } @@ -1250,8 +1249,10 @@ qemuDomainObjPrivateXMLFormatJob(virBuffer *buf, priv->job.phase)); } - if (priv->job.asyncJob != VIR_ASYNC_JOB_NONE) + if (priv->job.asyncJob != VIR_ASYNC_JOB_NONE) { virBufferAsprintf(&attrBuf, " flags='0x%lx'", priv->job.apiFlags); + virBufferAsprintf(&attrBuf, " asyncStarted='%llu'", priv->job.asyncStarted); + } if (priv->job.cb && priv->job.cb->formatJob(&childBuf, &priv->job, vm) < 0) @@ -1307,6 +1308,13 @@ qemuDomainObjPrivateXMLParseJob(virDomainObj *vm, } VIR_FREE(tmp); } + + if (virXPathULongLong("string(@asyncStarted)", ctxt, + &priv->job.asyncStarted) == -2) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Invalid async job start")); + return -1; + } } if (virXPathULongHex("string(@flags)", ctxt, &priv->job.apiFlags) == -2) { diff --git a/src/qemu/qemu_domainjob.h b/src/qemu/qemu_domainjob.h index 069bb9f8cb..707d4e91ed 100644 --- a/src/qemu/qemu_domainjob.h +++ b/src/qemu/qemu_domainjob.h @@ -164,6 +164,7 @@ void qemuDomainObjRestoreAsyncJob(virDomainObj *vm, virDomainAsyncJob asyncJob, int phase, + unsigned long long started, virDomainJobOperation operation, qemuDomainJobStatsType statsType, virDomainJobStatus status, diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 3d73c716f1..53221ea9ff 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3473,7 +3473,8 @@ qemuProcessRestoreMigrationJob(virDomainObj *vm, allowedJobs = VIR_JOB_DEFAULT_MASK | JOB_MASK(VIR_JOB_MIGRATION_OP); } - qemuDomainObjRestoreAsyncJob(vm, job->asyncJob, job->phase, op, + qemuDomainObjRestoreAsyncJob(vm, job->asyncJob, job->phase, + job->asyncStarted, op, QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION, VIR_DOMAIN_JOB_STATUS_PAUSED, allowedJobs); @@ -3746,6 +3747,7 @@ qemuProcessRecoverJob(virQEMUDriver *driver, case VIR_ASYNC_JOB_BACKUP: /* Restore the config of the async job which is not persisted */ qemuDomainObjRestoreAsyncJob(vm, VIR_ASYNC_JOB_BACKUP, 0, + job->asyncStarted, VIR_DOMAIN_JOB_OPERATION_BACKUP, QEMU_DOMAIN_JOB_STATS_TYPE_BACKUP, VIR_DOMAIN_JOB_STATUS_ACTIVE, diff --git a/tests/qemustatusxml2xmldata/migration-in-params-in.xml b/tests/qemustatusxml2xmldata/migration-in-params-in.xml index f4bc5753c4..9e9c2deac6 100644 --- a/tests/qemustatusxml2xmldata/migration-in-params-in.xml +++ b/tests/qemustatusxml2xmldata/migration-in-params-in.xml @@ -238,7 +238,7 @@ <flag name='dump-completed'/> <flag name='hda-output'/> </qemuCaps> - <job type='none' async='migration in' phase='prepare' flags='0x900'> + <job type='none' async='migration in' phase='prepare' flags='0x900' asyncStarted='0'> <migParams> <param name='compress-level' value='1'/> <param name='compress-threads' value='8'/> diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-bitmaps-in.xml b/tests/qemustatusxml2xmldata/migration-out-nbd-bitmaps-in.xml index c88996f923..1e14efe79a 100644 --- a/tests/qemustatusxml2xmldata/migration-out-nbd-bitmaps-in.xml +++ b/tests/qemustatusxml2xmldata/migration-out-nbd-bitmaps-in.xml @@ -262,7 +262,7 @@ <flag name='cpu-max'/> <flag name='migration-param.block-bitmap-mapping'/> </qemuCaps> - <job type='none' async='migration out' phase='perform3' flags='0x42'> + <job type='none' async='migration out' phase='perform3' flags='0x42' asyncStarted='0'> <disk dev='hda' migrating='no'/> <disk dev='vda' migrating='yes'> <migrationSource type='network' format='raw'> diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml b/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml index 039dcacc58..12abd0c81a 100644 --- a/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml +++ b/tests/qemustatusxml2xmldata/migration-out-nbd-out.xml @@ -231,7 +231,7 @@ <flag name='dump-completed'/> <flag name='hda-output'/> </qemuCaps> - <job type='none' async='migration out' phase='perform3' flags='0x0'> + <job type='none' async='migration out' phase='perform3' flags='0x0' asyncStarted='0'> <disk dev='vdb' migrating='yes'/> <disk dev='hda' migrating='no'/> </job> diff --git a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml index 3d1ddd5771..75c656d6c4 100644 --- a/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml +++ b/tests/qemustatusxml2xmldata/migration-out-nbd-tls-out.xml @@ -235,7 +235,7 @@ <flag name='nbd-tls'/> <flag name='blockdev-del'/> </qemuCaps> - <job type='none' async='migration out' phase='perform3' flags='0x0'> + <job type='none' async='migration out' phase='perform3' flags='0x0' asyncStarted='0'> <disk dev='vdb' migrating='yes'> <migrationSource type='network' format='raw'> <source protocol='nbd' name='drive-virtio-disk1' tlsFromConfig='0'> diff --git a/tests/qemustatusxml2xmldata/migration-out-params-in.xml b/tests/qemustatusxml2xmldata/migration-out-params-in.xml index cd9dbccd3a..c23c3c3083 100644 --- a/tests/qemustatusxml2xmldata/migration-out-params-in.xml +++ b/tests/qemustatusxml2xmldata/migration-out-params-in.xml @@ -238,7 +238,7 @@ <flag name='dump-completed'/> <flag name='hda-output'/> </qemuCaps> - <job type='none' async='migration out' phase='perform3' flags='0x802'> + <job type='none' async='migration out' phase='perform3' flags='0x802' asyncStarted='0'> <disk dev='vda' migrating='no'/> <migParams> <param name='compress-level' value='1'/> -- 2.35.1
On Tue, May 10, 2022 at 17:20:38 +0200, Jiri Denemark wrote:
Jobs that are supposed to remain active even when libvirt daemon restarts were reported as started at the time the daemon was restarted. This is not very helpful, we should restore the original timestamp.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 20 +++++++++++++------ src/qemu/qemu_domainjob.h | 1 + src/qemu/qemu_process.c | 4 +++- .../migration-in-params-in.xml | 2 +- .../migration-out-nbd-bitmaps-in.xml | 2 +- .../migration-out-nbd-out.xml | 2 +- .../migration-out-nbd-tls-out.xml | 2 +- .../migration-out-params-in.xml | 2 +- 8 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/src/qemu/qemu_domainjob.c b/src/qemu/qemu_domainjob.c index 1f82457bd4..8e8d229afe 100644 --- a/src/qemu/qemu_domainjob.c +++ b/src/qemu/qemu_domainjob.c
[...]
@@ -261,18 +263,15 @@ qemuDomainObjRestoreAsyncJob(virDomainObj *vm, { qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobObj *job = &priv->job; - unsigned long long now;
VIR_DEBUG("Restoring %s async job for domain %s", virDomainAsyncJobTypeToString(asyncJob), vm->def->name);
- ignore_value(virTimeMillisNow(&now)); - priv->job.jobsQueued++; job->asyncJob = asyncJob; job->phase = phase; job->asyncOwnerAPI = g_strdup(virThreadJobGet()); - job->asyncStarted = now; + job->asyncStarted = started;
qemuDomainObjSetAsyncJobMask(vm, allowedJobs);
@@ -280,7 +279,7 @@ qemuDomainObjRestoreAsyncJob(virDomainObj *vm, qemuDomainJobSetStatsType(priv->job.current, statsType); job->current->operation = operation; job->current->status = status; - job->current->started = now; + job->current->started = started; }
You don't seem to have any fallback when reading back older status XML where ...
@@ -1250,8 +1249,10 @@ qemuDomainObjPrivateXMLFormatJob(virBuffer *buf, priv->job.phase)); }
- if (priv->job.asyncJob != VIR_ASYNC_JOB_NONE) + if (priv->job.asyncJob != VIR_ASYNC_JOB_NONE) { virBufferAsprintf(&attrBuf, " flags='0x%lx'", priv->job.apiFlags); + virBufferAsprintf(&attrBuf, " asyncStarted='%llu'", priv->job.asyncStarted);
... the start timestamp is not printed, so ...
+ }
if (priv->job.cb && priv->job.cb->formatJob(&childBuf, &priv->job, vm) < 0) @@ -1307,6 +1308,13 @@ qemuDomainObjPrivateXMLParseJob(virDomainObj *vm, } VIR_FREE(tmp); } + + if (virXPathULongLong("string(@asyncStarted)", ctxt, + &priv->job.asyncStarted) == -2) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Invalid async job start"));
... this will keep it initialized to 0 ...
+ return -1; + } }
if (virXPathULongHex("string(@flags)", ctxt, &priv->job.apiFlags) == -2) {
[...]
diff --git a/tests/qemustatusxml2xmldata/migration-in-params-in.xml b/tests/qemustatusxml2xmldata/migration-in-params-in.xml index f4bc5753c4..9e9c2deac6 100644 --- a/tests/qemustatusxml2xmldata/migration-in-params-in.xml +++ b/tests/qemustatusxml2xmldata/migration-in-params-in.xml @@ -238,7 +238,7 @@ <flag name='dump-completed'/> <flag name='hda-output'/> </qemuCaps> - <job type='none' async='migration in' phase='prepare' flags='0x900'>
... so existing backup jobs will seem to be started at the beginning of epoch:
+ <job type='none' async='migration in' phase='prepare' flags='0x900' asyncStarted='0'> <migParams> <param name='compress-level' value='1'/> <param name='compress-threads' value='8'/>
Probably not a big problem but we IMO should initialize it to current time. That will break the test though unless you mock the time getting function.
On Wed, May 11, 2022 at 15:04:53 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:38 +0200, Jiri Denemark wrote:
Jobs that are supposed to remain active even when libvirt daemon restarts were reported as started at the time the daemon was restarted. This is not very helpful, we should restore the original timestamp.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 20 +++++++++++++------ src/qemu/qemu_domainjob.h | 1 + src/qemu/qemu_process.c | 4 +++- .../migration-in-params-in.xml | 2 +- .../migration-out-nbd-bitmaps-in.xml | 2 +- .../migration-out-nbd-out.xml | 2 +- .../migration-out-nbd-tls-out.xml | 2 +- .../migration-out-params-in.xml | 2 +- 8 files changed, 23 insertions(+), 12 deletions(-)
diff --git a/src/qemu/qemu_domainjob.c b/src/qemu/qemu_domainjob.c index 1f82457bd4..8e8d229afe 100644 --- a/src/qemu/qemu_domainjob.c +++ b/src/qemu/qemu_domainjob.c
[...]
@@ -261,18 +263,15 @@ qemuDomainObjRestoreAsyncJob(virDomainObj *vm, { qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobObj *job = &priv->job; - unsigned long long now;
VIR_DEBUG("Restoring %s async job for domain %s", virDomainAsyncJobTypeToString(asyncJob), vm->def->name);
- ignore_value(virTimeMillisNow(&now)); - priv->job.jobsQueued++; job->asyncJob = asyncJob; job->phase = phase; job->asyncOwnerAPI = g_strdup(virThreadJobGet()); - job->asyncStarted = now; + job->asyncStarted = started;
qemuDomainObjSetAsyncJobMask(vm, allowedJobs);
@@ -280,7 +279,7 @@ qemuDomainObjRestoreAsyncJob(virDomainObj *vm, qemuDomainJobSetStatsType(priv->job.current, statsType); job->current->operation = operation; job->current->status = status; - job->current->started = now; + job->current->started = started; }
You don't seem to have any fallback when reading back older status XML where ...
@@ -1250,8 +1249,10 @@ qemuDomainObjPrivateXMLFormatJob(virBuffer *buf, priv->job.phase)); }
- if (priv->job.asyncJob != VIR_ASYNC_JOB_NONE) + if (priv->job.asyncJob != VIR_ASYNC_JOB_NONE) { virBufferAsprintf(&attrBuf, " flags='0x%lx'", priv->job.apiFlags); + virBufferAsprintf(&attrBuf, " asyncStarted='%llu'", priv->job.asyncStarted);
... the start timestamp is not printed, so ...
+ }
if (priv->job.cb && priv->job.cb->formatJob(&childBuf, &priv->job, vm) < 0) @@ -1307,6 +1308,13 @@ qemuDomainObjPrivateXMLParseJob(virDomainObj *vm, } VIR_FREE(tmp); } + + if (virXPathULongLong("string(@asyncStarted)", ctxt, + &priv->job.asyncStarted) == -2) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("Invalid async job start"));
... this will keep it initialized to 0 ...
+ return -1; + } }
if (virXPathULongHex("string(@flags)", ctxt, &priv->job.apiFlags) == -2) {
[...]
diff --git a/tests/qemustatusxml2xmldata/migration-in-params-in.xml b/tests/qemustatusxml2xmldata/migration-in-params-in.xml index f4bc5753c4..9e9c2deac6 100644 --- a/tests/qemustatusxml2xmldata/migration-in-params-in.xml +++ b/tests/qemustatusxml2xmldata/migration-in-params-in.xml @@ -238,7 +238,7 @@ <flag name='dump-completed'/> <flag name='hda-output'/> </qemuCaps> - <job type='none' async='migration in' phase='prepare' flags='0x900'>
... so existing backup jobs will seem to be started at the beginning of epoch:
+ <job type='none' async='migration in' phase='prepare' flags='0x900' asyncStarted='0'> <migParams> <param name='compress-level' value='1'/> <param name='compress-threads' value='8'/>
Probably not a big problem but we IMO should initialize it to current time. That will break the test though unless you mock the time getting function.
We can keep this fallback in qemuDomainObjRestoreAsyncJob instead of putting it in the parser to avoid test issues. Jirka
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 176 ++++++++++++++++---------------------- 1 file changed, 76 insertions(+), 100 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index e892a09885..63ebc15f65 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -81,35 +81,97 @@ VIR_ENUM_IMPL(qemuMigrationJobPhase, "finish3", ); -static int + +static int ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) G_GNUC_WARN_UNUSED_RESULT qemuMigrationJobStart(virQEMUDriver *driver, virDomainObj *vm, virDomainAsyncJob job, unsigned long apiFlags) - ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) G_GNUC_WARN_UNUSED_RESULT; +{ + qemuDomainObjPrivate *priv = vm->privateData; + virDomainJobOperation op; + unsigned long long mask; -static void + if (job == VIR_ASYNC_JOB_MIGRATION_IN) { + op = VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN; + mask = VIR_JOB_NONE; + } else { + op = VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT; + mask = VIR_JOB_DEFAULT_MASK | + JOB_MASK(VIR_JOB_SUSPEND) | + JOB_MASK(VIR_JOB_MIGRATION_OP); + } + + if (qemuDomainObjBeginAsyncJob(driver, vm, job, op, apiFlags) < 0) + return -1; + + qemuDomainJobSetStatsType(priv->job.current, + QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); + + qemuDomainObjSetAsyncJobMask(vm, mask); + return 0; +} + + +static void ATTRIBUTE_NONNULL(1) qemuMigrationJobSetPhase(virDomainObj *vm, qemuMigrationJobPhase phase) - ATTRIBUTE_NONNULL(1); +{ + qemuDomainObjPrivate *priv = vm->privateData; -static void + if (phase < priv->job.phase) { + VIR_ERROR(_("migration protocol going backwards %s => %s"), + qemuMigrationJobPhaseTypeToString(priv->job.phase), + qemuMigrationJobPhaseTypeToString(phase)); + return; + } + + qemuDomainObjSetJobPhase(vm, phase); +} + + +static void ATTRIBUTE_NONNULL(1) qemuMigrationJobStartPhase(virDomainObj *vm, qemuMigrationJobPhase phase) - ATTRIBUTE_NONNULL(1); +{ + qemuMigrationJobSetPhase(vm, phase); +} -static void -qemuMigrationJobContinue(virDomainObj *obj) - ATTRIBUTE_NONNULL(1); -static bool +static void ATTRIBUTE_NONNULL(1) +qemuMigrationJobContinue(virDomainObj *vm) +{ + qemuDomainObjReleaseAsyncJob(vm); +} + + +static bool ATTRIBUTE_NONNULL(1) qemuMigrationJobIsActive(virDomainObj *vm, virDomainAsyncJob job) - ATTRIBUTE_NONNULL(1); +{ + qemuDomainObjPrivate *priv = vm->privateData; + + if (priv->job.asyncJob != job) { + const char *msg; + + if (job == VIR_ASYNC_JOB_MIGRATION_IN) + msg = _("domain '%s' is not processing incoming migration"); + else + msg = _("domain '%s' is not being migrated"); + + virReportError(VIR_ERR_OPERATION_INVALID, msg, vm->def->name); + return false; + } + return true; +} + + +static void ATTRIBUTE_NONNULL(1) +qemuMigrationJobFinish(virDomainObj *vm) +{ + qemuDomainObjEndAsyncJob(vm); +} -static void -qemuMigrationJobFinish(virDomainObj *obj) - ATTRIBUTE_NONNULL(1); static void qemuMigrationSrcStoreDomainState(virDomainObj *vm) @@ -6180,92 +6242,6 @@ qemuMigrationSrcCancel(virQEMUDriver *driver, } -static int -qemuMigrationJobStart(virQEMUDriver *driver, - virDomainObj *vm, - virDomainAsyncJob job, - unsigned long apiFlags) -{ - qemuDomainObjPrivate *priv = vm->privateData; - virDomainJobOperation op; - unsigned long long mask; - - if (job == VIR_ASYNC_JOB_MIGRATION_IN) { - op = VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN; - mask = VIR_JOB_NONE; - } else { - op = VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT; - mask = VIR_JOB_DEFAULT_MASK | - JOB_MASK(VIR_JOB_SUSPEND) | - JOB_MASK(VIR_JOB_MIGRATION_OP); - } - - if (qemuDomainObjBeginAsyncJob(driver, vm, job, op, apiFlags) < 0) - return -1; - - qemuDomainJobSetStatsType(priv->job.current, - QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); - - qemuDomainObjSetAsyncJobMask(vm, mask); - return 0; -} - -static void -qemuMigrationJobSetPhase(virDomainObj *vm, - qemuMigrationJobPhase phase) -{ - qemuDomainObjPrivate *priv = vm->privateData; - - if (phase < priv->job.phase) { - VIR_ERROR(_("migration protocol going backwards %s => %s"), - qemuMigrationJobPhaseTypeToString(priv->job.phase), - qemuMigrationJobPhaseTypeToString(phase)); - return; - } - - qemuDomainObjSetJobPhase(vm, phase); -} - -static void -qemuMigrationJobStartPhase(virDomainObj *vm, - qemuMigrationJobPhase phase) -{ - qemuMigrationJobSetPhase(vm, phase); -} - -static void -qemuMigrationJobContinue(virDomainObj *vm) -{ - qemuDomainObjReleaseAsyncJob(vm); -} - -static bool -qemuMigrationJobIsActive(virDomainObj *vm, - virDomainAsyncJob job) -{ - qemuDomainObjPrivate *priv = vm->privateData; - - if (priv->job.asyncJob != job) { - const char *msg; - - if (job == VIR_ASYNC_JOB_MIGRATION_IN) - msg = _("domain '%s' is not processing incoming migration"); - else - msg = _("domain '%s' is not being migrated"); - - virReportError(VIR_ERR_OPERATION_INVALID, msg, vm->def->name); - return false; - } - return true; -} - -static void -qemuMigrationJobFinish(virDomainObj *vm) -{ - qemuDomainObjEndAsyncJob(vm); -} - - static void qemuMigrationDstErrorFree(void *data) { -- 2.35.1
On Tue, May 10, 2022 at 17:20:39 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 176 ++++++++++++++++---------------------- 1 file changed, 76 insertions(+), 100 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Migration is a job which takes some time and if it succeeds, there's nothing to call another migration on. If a migration fails, it might make sense to rerun it with different arguments, but this would only be done once the first migration fails rather than while it is still running. If this was not enough, the migration job now stays active even if post-copy migration fails and anyone possibly retrying the migration would be waiting for the job timeout just to get a suboptimal error message. So let's special case getting a migration job when another one is already active. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 63ebc15f65..72a2846d1f 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -82,6 +82,23 @@ VIR_ENUM_IMPL(qemuMigrationJobPhase, ); +static bool ATTRIBUTE_NONNULL(1) +qemuMigrationJobIsAllowed(virDomainObj *vm) +{ + qemuDomainObjPrivate *priv = vm->privateData; + + if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_IN || + priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("another migration job is already running for domain '%s'"), + vm->def->name); + return false; + } + + return true; +} + + static int ATTRIBUTE_NONNULL(1) ATTRIBUTE_NONNULL(2) G_GNUC_WARN_UNUSED_RESULT qemuMigrationJobStart(virQEMUDriver *driver, virDomainObj *vm, @@ -92,6 +109,9 @@ qemuMigrationJobStart(virQEMUDriver *driver, virDomainJobOperation op; unsigned long long mask; + if (!qemuMigrationJobIsAllowed(vm)) + return -1; + if (job == VIR_ASYNC_JOB_MIGRATION_IN) { op = VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN; mask = VIR_JOB_NONE; @@ -2629,6 +2649,9 @@ qemuMigrationSrcBegin(virConnectPtr conn, goto cleanup; asyncJob = VIR_ASYNC_JOB_MIGRATION_OUT; } else { + if (!qemuMigrationJobIsAllowed(vm)) + goto cleanup; + if (qemuDomainObjBeginJob(driver, vm, VIR_JOB_MODIFY) < 0) goto cleanup; asyncJob = VIR_ASYNC_JOB_NONE; -- 2.35.1
On Tue, May 10, 2022 at 17:20:40 +0200, Jiri Denemark wrote:
Migration is a job which takes some time and if it succeeds, there's nothing to call another migration on. If a migration fails, it might make sense to rerun it with different arguments, but this would only be done once the first migration fails rather than while it is still running.
If this was not enough, the migration job now stays active even if post-copy migration fails and anyone possibly retrying the migration would be waiting for the job timeout just to get a suboptimal error message.
So let's special case getting a migration job when another one is already active.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
To make sure all job states are properly handled. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index e3582f62a7..85ccc446d7 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12508,10 +12508,11 @@ qemuDomainGetJobInfoMigrationStats(virQEMUDriver *driver, bool events = virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_EVENT); - if (jobData->status == VIR_DOMAIN_JOB_STATUS_ACTIVE || - jobData->status == VIR_DOMAIN_JOB_STATUS_MIGRATING || - jobData->status == VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED || - jobData->status == VIR_DOMAIN_JOB_STATUS_POSTCOPY) { + switch (jobData->status) { + case VIR_DOMAIN_JOB_STATUS_ACTIVE: + case VIR_DOMAIN_JOB_STATUS_MIGRATING: + case VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED: + case VIR_DOMAIN_JOB_STATUS_POSTCOPY: if (events && jobData->status != VIR_DOMAIN_JOB_STATUS_ACTIVE && qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_NONE, @@ -12526,6 +12527,17 @@ qemuDomainGetJobInfoMigrationStats(virQEMUDriver *driver, if (qemuDomainJobDataUpdateTime(jobData) < 0) return -1; + + break; + + case VIR_DOMAIN_JOB_STATUS_NONE: + case VIR_DOMAIN_JOB_STATUS_PAUSED: + case VIR_DOMAIN_JOB_STATUS_COMPLETED: + case VIR_DOMAIN_JOB_STATUS_FAILED: + case VIR_DOMAIN_JOB_STATUS_CANCELED: + default: + /* No migration stats to be fetched in these states. */ + break; } return 0; -- 2.35.1
On Tue, May 10, 2022 at 17:20:41 +0200, Jiri Denemark wrote:
To make sure all job states are properly handled.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 15:19:33 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:41 +0200, Jiri Denemark wrote:
To make sure all job states are properly handled.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
This patch was replaced with the "qemu: Drop QEMU_CAPS_MIGRATION_EVENT" series. Jirka
Even though a migration is paused, we still want to see the amount of data transferred so far and that the migration is indeed not progressing any further. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 85ccc446d7..9e6cf26001 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12513,6 +12513,7 @@ qemuDomainGetJobInfoMigrationStats(virQEMUDriver *driver, case VIR_DOMAIN_JOB_STATUS_MIGRATING: case VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED: case VIR_DOMAIN_JOB_STATUS_POSTCOPY: + case VIR_DOMAIN_JOB_STATUS_PAUSED: if (events && jobData->status != VIR_DOMAIN_JOB_STATUS_ACTIVE && qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_NONE, @@ -12531,7 +12532,6 @@ qemuDomainGetJobInfoMigrationStats(virQEMUDriver *driver, break; case VIR_DOMAIN_JOB_STATUS_NONE: - case VIR_DOMAIN_JOB_STATUS_PAUSED: case VIR_DOMAIN_JOB_STATUS_COMPLETED: case VIR_DOMAIN_JOB_STATUS_FAILED: case VIR_DOMAIN_JOB_STATUS_CANCELED: -- 2.35.1
On Tue, May 10, 2022 at 17:20:42 +0200, Jiri Denemark wrote:
Even though a migration is paused, we still want to see the amount of data transferred so far and that the migration is indeed not progressing any further.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
When connection breaks during post-copy migration, QEMU enters 'postcopy-paused' state. We need to handle this state and make the situation visible to upper layers. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/hypervisor/domain_job.c | 1 + src/hypervisor/domain_job.h | 1 + src/qemu/qemu_driver.c | 1 + src/qemu/qemu_migration.c | 11 +++++++++++ src/qemu/qemu_monitor.c | 1 + src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 1 + src/qemu/qemu_process.c | 33 ++++++++++++++++++++------------- 8 files changed, 37 insertions(+), 13 deletions(-) diff --git a/src/hypervisor/domain_job.c b/src/hypervisor/domain_job.c index ff4e008cb5..49867c3982 100644 --- a/src/hypervisor/domain_job.c +++ b/src/hypervisor/domain_job.c @@ -93,6 +93,7 @@ virDomainJobStatusToType(virDomainJobStatus status) case VIR_DOMAIN_JOB_STATUS_MIGRATING: case VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED: case VIR_DOMAIN_JOB_STATUS_POSTCOPY: + case VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED: case VIR_DOMAIN_JOB_STATUS_PAUSED: return VIR_DOMAIN_JOB_UNBOUNDED; diff --git a/src/hypervisor/domain_job.h b/src/hypervisor/domain_job.h index db8b8b1390..fce35ffbf5 100644 --- a/src/hypervisor/domain_job.h +++ b/src/hypervisor/domain_job.h @@ -78,6 +78,7 @@ typedef enum { VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED, VIR_DOMAIN_JOB_STATUS_PAUSED, VIR_DOMAIN_JOB_STATUS_POSTCOPY, + VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED, VIR_DOMAIN_JOB_STATUS_COMPLETED, VIR_DOMAIN_JOB_STATUS_FAILED, VIR_DOMAIN_JOB_STATUS_CANCELED, diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index 9e6cf26001..d19115473a 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12514,6 +12514,7 @@ qemuDomainGetJobInfoMigrationStats(virQEMUDriver *driver, case VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED: case VIR_DOMAIN_JOB_STATUS_POSTCOPY: case VIR_DOMAIN_JOB_STATUS_PAUSED: + case VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED: if (events && jobData->status != VIR_DOMAIN_JOB_STATUS_ACTIVE && qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_NONE, diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 72a2846d1f..e5e33556e3 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1757,6 +1757,10 @@ qemuMigrationUpdateJobType(virDomainJobData *jobData) jobData->status = VIR_DOMAIN_JOB_STATUS_POSTCOPY; break; + case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_PAUSED: + jobData->status = VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED; + break; + case QEMU_MONITOR_MIGRATION_STATUS_COMPLETED: jobData->status = VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED; break; @@ -1883,6 +1887,12 @@ qemuMigrationJobCheckStatus(virQEMUDriver *driver, qemuMigrationJobName(vm), _("canceled by client")); return -1; + case VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED: + virReportError(VIR_ERR_OPERATION_FAILED, _("%s: %s"), + qemuMigrationJobName(vm), + _("post-copy phase failed")); + return -1; + case VIR_DOMAIN_JOB_STATUS_COMPLETED: case VIR_DOMAIN_JOB_STATUS_ACTIVE: case VIR_DOMAIN_JOB_STATUS_MIGRATING: @@ -1985,6 +1995,7 @@ qemuMigrationAnyCompleted(virQEMUDriver *driver, case VIR_DOMAIN_JOB_STATUS_FAILED: case VIR_DOMAIN_JOB_STATUS_CANCELED: + case VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED: /* QEMU aborted the migration. */ return -1; diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 98cf1c949e..36cf4e57b5 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -149,6 +149,7 @@ VIR_ENUM_IMPL(qemuMonitorMigrationStatus, "inactive", "setup", "active", "pre-switchover", "device", "postcopy-active", + "postcopy-paused", "completed", "failed", "cancelling", "cancelled", "wait-unplug", diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index abc29eaf4c..46cdc04925 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -831,6 +831,7 @@ typedef enum { QEMU_MONITOR_MIGRATION_STATUS_PRE_SWITCHOVER, QEMU_MONITOR_MIGRATION_STATUS_DEVICE, QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY, + QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_PAUSED, QEMU_MONITOR_MIGRATION_STATUS_COMPLETED, QEMU_MONITOR_MIGRATION_STATUS_ERROR, QEMU_MONITOR_MIGRATION_STATUS_CANCELLING, diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 532aad348e..ac3ec42fdd 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3242,6 +3242,7 @@ qemuMonitorJSONGetMigrationStatsReply(virJSONValue *reply, case QEMU_MONITOR_MIGRATION_STATUS_ACTIVE: case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY: + case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_PAUSED: case QEMU_MONITOR_MIGRATION_STATUS_COMPLETED: case QEMU_MONITOR_MIGRATION_STATUS_CANCELLING: case QEMU_MONITOR_MIGRATION_STATUS_PRE_SWITCHOVER: diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 53221ea9ff..fc61aa71a0 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -1568,19 +1568,26 @@ qemuProcessHandleMigrationStatus(qemuMonitor *mon G_GNUC_UNUSED, privJob->stats.mig.status = status; virDomainObjBroadcast(vm); - if (status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY && - priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT && - virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && - reason == VIR_DOMAIN_PAUSED_MIGRATION) { - VIR_DEBUG("Correcting paused state reason for domain %s to %s", - vm->def->name, - virDomainPausedReasonTypeToString(VIR_DOMAIN_PAUSED_POSTCOPY)); - - virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, VIR_DOMAIN_PAUSED_POSTCOPY); - event = virDomainEventLifecycleNewFromObj(vm, - VIR_DOMAIN_EVENT_SUSPENDED, - VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY); - qemuDomainSaveStatus(vm); + if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT && + virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED) { + if (status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_PAUSED) { + /* At this point no thread is watching the migration progress on + * the source as it is just waiting for the Finish phase to end. + * Thus we need to handle the event here. */ + qemuMigrationSrcPostcopyFailed(vm); + qemuDomainSaveStatus(vm); + } else if (status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY && + reason == VIR_DOMAIN_PAUSED_MIGRATION) { + VIR_DEBUG("Correcting paused state reason for domain %s to %s", + vm->def->name, + virDomainPausedReasonTypeToString(VIR_DOMAIN_PAUSED_POSTCOPY)); + + virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, VIR_DOMAIN_PAUSED_POSTCOPY); + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_SUSPENDED, + VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY); + qemuDomainSaveStatus(vm); + } } cleanup: -- 2.35.1
On Tue, May 10, 2022 at 17:20:43 +0200, Jiri Denemark wrote:
When connection breaks during post-copy migration, QEMU enters 'postcopy-paused' state. We need to handle this state and make the situation visible to upper layers.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/hypervisor/domain_job.c | 1 + src/hypervisor/domain_job.h | 1 + src/qemu/qemu_driver.c | 1 + src/qemu/qemu_migration.c | 11 +++++++++++ src/qemu/qemu_monitor.c | 1 + src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 1 + src/qemu/qemu_process.c | 33 ++++++++++++++++++++------------- 8 files changed, 37 insertions(+), 13 deletions(-)
[...]
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 72a2846d1f..e5e33556e3 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c
[...]
@ -1883,6 +1887,12 @@ qemuMigrationJobCheckStatus(virQEMUDriver *driver, qemuMigrationJobName(vm), _("canceled by client")); return -1;
+ case VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED: + virReportError(VIR_ERR_OPERATION_FAILED, _("%s: %s"), + qemuMigrationJobName(vm), + _("post-copy phase failed"));
This is very translation unfriendly: https://www.libvirt.org/coding-style.html#error-message-format I see it's a common theme in this function though ... bleah.
+ return -1; + case VIR_DOMAIN_JOB_STATUS_COMPLETED: case VIR_DOMAIN_JOB_STATUS_ACTIVE: case VIR_DOMAIN_JOB_STATUS_MIGRATING:
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 15:27:53 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:43 +0200, Jiri Denemark wrote:
When connection breaks during post-copy migration, QEMU enters 'postcopy-paused' state. We need to handle this state and make the situation visible to upper layers.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/hypervisor/domain_job.c | 1 + src/hypervisor/domain_job.h | 1 + src/qemu/qemu_driver.c | 1 + src/qemu/qemu_migration.c | 11 +++++++++++ src/qemu/qemu_monitor.c | 1 + src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 1 + src/qemu/qemu_process.c | 33 ++++++++++++++++++++------------- 8 files changed, 37 insertions(+), 13 deletions(-)
[...]
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 72a2846d1f..e5e33556e3 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c
[...]
@ -1883,6 +1887,12 @@ qemuMigrationJobCheckStatus(virQEMUDriver *driver, qemuMigrationJobName(vm), _("canceled by client")); return -1;
+ case VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED: + virReportError(VIR_ERR_OPERATION_FAILED, _("%s: %s"), + qemuMigrationJobName(vm), + _("post-copy phase failed"));
This is very translation unfriendly:
https://www.libvirt.org/coding-style.html#error-message-format
I see it's a common theme in this function though ... bleah.
Yeah. I'll make a follow up patch to fix this. Jirka
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 1 + src/qemu/qemu_monitor.c | 2 +- src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 1 + src/qemu/qemu_process.c | 31 ++++++++++++++++++++++++++++++- 5 files changed, 34 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index e5e33556e3..5fbc930e7c 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -1754,6 +1754,7 @@ qemuMigrationUpdateJobType(virDomainJobData *jobData) switch ((qemuMonitorMigrationStatus) priv->stats.mig.status) { case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY: + case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_RECOVER: jobData->status = VIR_DOMAIN_JOB_STATUS_POSTCOPY; break; diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 36cf4e57b5..589fef4385 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -149,7 +149,7 @@ VIR_ENUM_IMPL(qemuMonitorMigrationStatus, "inactive", "setup", "active", "pre-switchover", "device", "postcopy-active", - "postcopy-paused", + "postcopy-paused", "postcopy-recover", "completed", "failed", "cancelling", "cancelled", "wait-unplug", diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 46cdc04925..5949e21a39 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -832,6 +832,7 @@ typedef enum { QEMU_MONITOR_MIGRATION_STATUS_DEVICE, QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY, QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_PAUSED, + QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_RECOVER, QEMU_MONITOR_MIGRATION_STATUS_COMPLETED, QEMU_MONITOR_MIGRATION_STATUS_ERROR, QEMU_MONITOR_MIGRATION_STATUS_CANCELLING, diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index ac3ec42fdd..3a497dceae 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3243,6 +3243,7 @@ qemuMonitorJSONGetMigrationStatsReply(virJSONValue *reply, case QEMU_MONITOR_MIGRATION_STATUS_ACTIVE: case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY: case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_PAUSED: + case QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_RECOVER: case QEMU_MONITOR_MIGRATION_STATUS_COMPLETED: case QEMU_MONITOR_MIGRATION_STATUS_CANCELLING: case QEMU_MONITOR_MIGRATION_STATUS_PRE_SWITCHOVER: diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index fc61aa71a0..c3a966983f 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -1549,6 +1549,7 @@ qemuProcessHandleMigrationStatus(qemuMonitor *mon G_GNUC_UNUSED, qemuDomainJobDataPrivate *privJob = NULL; virQEMUDriver *driver = opaque; virObjectEvent *event = NULL; + virDomainState state; int reason; virObjectLock(vm); @@ -1568,8 +1569,10 @@ qemuProcessHandleMigrationStatus(qemuMonitor *mon G_GNUC_UNUSED, privJob->stats.mig.status = status; virDomainObjBroadcast(vm); + state = virDomainObjGetState(vm, &reason); + if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT && - virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED) { + state == VIR_DOMAIN_PAUSED) { if (status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_PAUSED) { /* At this point no thread is watching the migration progress on * the source as it is just waiting for the Finish phase to end. @@ -1590,6 +1593,32 @@ qemuProcessHandleMigrationStatus(qemuMonitor *mon G_GNUC_UNUSED, } } + if ((status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY || + status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_RECOVER) && + virDomainObjIsFailedPostcopy(vm)) { + int eventType = -1; + int eventDetail = -1; + + if (state == VIR_DOMAIN_PAUSED) { + reason = VIR_DOMAIN_PAUSED_POSTCOPY; + eventType = VIR_DOMAIN_EVENT_SUSPENDED; + eventDetail = qemuDomainPausedReasonToSuspendedEvent(reason); + } else { + reason = VIR_DOMAIN_RUNNING_POSTCOPY; + eventType = VIR_DOMAIN_EVENT_RESUMED; + eventDetail = qemuDomainRunningReasonToResumeEvent(reason); + } + + VIR_DEBUG("Post-copy migration recovered; correcting state for domain " + "%s to %s/%s", + vm->def->name, + virDomainStateTypeToString(state), + NULLSTR(virDomainStateReasonToString(state, reason))); + virDomainObjSetState(vm, state, reason); + event = virDomainEventLifecycleNewFromObj(vm, eventType, eventDetail); + qemuDomainSaveStatus(vm); + } + cleanup: virObjectUnlock(vm); virObjectEventStateQueue(driver->domainEventState, event); -- 2.35.1
On Tue, May 10, 2022 at 17:20:44 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 1 + src/qemu/qemu_monitor.c | 2 +- src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 1 + src/qemu/qemu_process.c | 31 ++++++++++++++++++++++++++++++- 5 files changed, 34 insertions(+), 2 deletions(-)
[...]
@@ -1590,6 +1593,32 @@ qemuProcessHandleMigrationStatus(qemuMonitor *mon G_GNUC_UNUSED, } }
+ if ((status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY || + status == QEMU_MONITOR_MIGRATION_STATUS_POSTCOPY_RECOVER) && + virDomainObjIsFailedPostcopy(vm)) { + int eventType = -1; + int eventDetail = -1; + + if (state == VIR_DOMAIN_PAUSED) { + reason = VIR_DOMAIN_PAUSED_POSTCOPY; + eventType = VIR_DOMAIN_EVENT_SUSPENDED; + eventDetail = qemuDomainPausedReasonToSuspendedEvent(reason); + } else { + reason = VIR_DOMAIN_RUNNING_POSTCOPY; + eventType = VIR_DOMAIN_EVENT_RESUMED; + eventDetail = qemuDomainRunningReasonToResumeEvent(reason); + } + + VIR_DEBUG("Post-copy migration recovered; correcting state for domain " + "%s to %s/%s",
Preferrably conform to the error message coding style here: - no linebreaks - all substitutions should be somehow enclosed e.g. in '%s'
+ vm->def->name, + virDomainStateTypeToString(state), + NULLSTR(virDomainStateReasonToString(state, reason))); + virDomainObjSetState(vm, state, reason); + event = virDomainEventLifecycleNewFromObj(vm, eventType, eventDetail); + qemuDomainSaveStatus(vm); + }
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
We don't need the object until we get to the "endjob" label. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 5fbc930e7c..1f6f008ad9 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5980,11 +5980,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, VIR_DOMAIN_RUNNING, VIR_DOMAIN_RUNNING_MIGRATED); } - } - - dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id); - if (inPostCopy) { /* The only RESUME event during post-copy migration is triggered by * QEMU when the running domain moves from the source to the * destination host, but then the migration keeps running until all @@ -6012,6 +6008,8 @@ qemuMigrationDstFinish(virQEMUDriver *driver, /* Guest is successfully running, so cancel previous auto destroy */ qemuProcessAutoDestroyRemove(driver, vm); + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id); + endjob: if (!dom && !(flags & VIR_MIGRATE_OFFLINE) && -- 2.35.1
On Tue, May 10, 2022 at 17:20:45 +0200, Jiri Denemark wrote:
We don't need the object until we get to the "endjob" label.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Code executed only when dom != NULL can be moved before "endjob" label, to the only place where dom is set. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 50 ++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 22 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 1f6f008ad9..cb17cbd189 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5844,8 +5844,16 @@ qemuMigrationDstFinish(virQEMUDriver *driver, if (flags & VIR_MIGRATE_OFFLINE) { if (retcode == 0 && - qemuMigrationDstPersist(driver, vm, mig, false) == 0) + qemuMigrationDstPersist(driver, vm, mig, false) == 0) { dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1); + + if (dom && + qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) + VIR_WARN("Unable to encode migration cookie"); + } goto endjob; } @@ -6008,6 +6016,25 @@ qemuMigrationDstFinish(virQEMUDriver *driver, /* Guest is successfully running, so cancel previous auto destroy */ qemuProcessAutoDestroyRemove(driver, vm); + if (jobData) { + priv->job.completed = g_steal_pointer(&jobData); + priv->job.completed->status = VIR_DOMAIN_JOB_STATUS_COMPLETED; + qemuDomainJobSetStatsType(priv->job.completed, + QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); + } + + if (qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) + VIR_WARN("Unable to encode migration cookie"); + + /* Remove completed stats for post-copy, everything but timing fields + * is obsolete anyway. + */ + if (inPostCopy) + g_clear_pointer(&priv->job.completed, virDomainJobDataFree); + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id); endjob: @@ -6028,27 +6055,6 @@ qemuMigrationDstFinish(virQEMUDriver *driver, } } - if (dom) { - if (jobData) { - priv->job.completed = g_steal_pointer(&jobData); - priv->job.completed->status = VIR_DOMAIN_JOB_STATUS_COMPLETED; - qemuDomainJobSetStatsType(priv->job.completed, - QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); - } - - if (qemuMigrationCookieFormat(mig, driver, vm, - QEMU_MIGRATION_DESTINATION, - cookieout, cookieoutlen, - QEMU_MIGRATION_COOKIE_STATS) < 0) - VIR_WARN("Unable to encode migration cookie"); - - /* Remove completed stats for post-copy, everything but timing fields - * is obsolete anyway. - */ - if (inPostCopy) - g_clear_pointer(&priv->job.completed, virDomainJobDataFree); - } - if (virDomainObjIsFailedPostcopy(vm)) { qemuProcessAutoDestroyRemove(driver, vm); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); -- 2.35.1
On Tue, May 10, 2022 at 17:20:46 +0200, Jiri Denemark wrote:
Code executed only when dom != NULL can be moved before "endjob" label, to the only place where dom is set.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 50 ++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 22 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 1f6f008ad9..cb17cbd189 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5844,8 +5844,16 @@ qemuMigrationDstFinish(virQEMUDriver *driver,
if (flags & VIR_MIGRATE_OFFLINE) { if (retcode == 0 && - qemuMigrationDstPersist(driver, vm, mig, false) == 0) + qemuMigrationDstPersist(driver, vm, mig, false) == 0) { dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1);
IMO virGetDomain should be treated as that it can't fail ...
+ + if (dom &&
... so this check seems pointless.
+ qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0)
broken alignment
+ VIR_WARN("Unable to encode migration cookie"); + } goto endjob; }
@@ -6008,6 +6016,25 @@ qemuMigrationDstFinish(virQEMUDriver *driver, /* Guest is successfully running, so cancel previous auto destroy */ qemuProcessAutoDestroyRemove(driver, vm);
+ if (jobData) { + priv->job.completed = g_steal_pointer(&jobData); + priv->job.completed->status = VIR_DOMAIN_JOB_STATUS_COMPLETED; + qemuDomainJobSetStatsType(priv->job.completed, + QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); + } + + if (qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) + VIR_WARN("Unable to encode migration cookie");
E.g. here you don't check 'dom' because it's not yet allocated.
+ + /* Remove completed stats for post-copy, everything but timing fields + * is obsolete anyway. + */ + if (inPostCopy) + g_clear_pointer(&priv->job.completed, virDomainJobDataFree); + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id);
endjob:
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 15:40:58 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:46 +0200, Jiri Denemark wrote:
Code executed only when dom != NULL can be moved before "endjob" label, to the only place where dom is set.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 50 ++++++++++++++++++++++----------------- 1 file changed, 28 insertions(+), 22 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 1f6f008ad9..cb17cbd189 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5844,8 +5844,16 @@ qemuMigrationDstFinish(virQEMUDriver *driver,
if (flags & VIR_MIGRATE_OFFLINE) { if (retcode == 0 && - qemuMigrationDstPersist(driver, vm, mig, false) == 0) + qemuMigrationDstPersist(driver, vm, mig, false) == 0) { dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1);
IMO virGetDomain should be treated as that it can't fail ...
Well, it does not return void and we check the value elsewhere...
+ + if (dom &&
... so this check seems pointless.
Mostly, but we can avoid formatting the cookie in case we know we're going to fail anyway.
+ qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0)
broken alignment
Oops.
+ VIR_WARN("Unable to encode migration cookie"); + } goto endjob; }
Jirka
Most of the code in "endjob" label is executed only on failure. Let's duplicate the rest so that the label can be used only in error path making the success path easier to follow and refactor. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 45 ++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 19 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index cb17cbd189..0c6d2c041b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5854,7 +5854,9 @@ qemuMigrationDstFinish(virQEMUDriver *driver, QEMU_MIGRATION_COOKIE_STATS) < 0) VIR_WARN("Unable to encode migration cookie"); } - goto endjob; + + qemuMigrationJobFinish(vm); + goto cleanup; } if (retcode != 0) { @@ -6035,12 +6037,31 @@ qemuMigrationDstFinish(virQEMUDriver *driver, if (inPostCopy) g_clear_pointer(&priv->job.completed, virDomainJobDataFree); + qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + jobPriv->migParams, priv->job.apiFlags); + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id); + qemuMigrationJobFinish(vm); + + cleanup: + g_clear_pointer(&jobData, virDomainJobDataFree); + virPortAllocatorRelease(port); + if (priv->mon) + qemuMonitorSetDomainLog(priv->mon, NULL, NULL, NULL); + VIR_FREE(priv->origname); + virDomainObjEndAPI(&vm); + virErrorRestore(&orig_err); + + /* Set a special error if Finish is expected to return NULL as a result of + * successful call with retcode != 0 + */ + if (retcode != 0 && !dom && virGetLastErrorCode() == VIR_ERR_OK) + virReportError(VIR_ERR_MIGRATE_FINISH_OK, NULL); + return dom; + endjob: - if (!dom && - !(flags & VIR_MIGRATE_OFFLINE) && - virDomainObjIsActive(vm)) { + if (virDomainObjIsActive(vm)) { if (doKill) { qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, VIR_ASYNC_JOB_MIGRATION_IN, @@ -6069,21 +6090,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, if (!virDomainObjIsActive(vm)) qemuDomainRemoveInactive(driver, vm); - cleanup: - g_clear_pointer(&jobData, virDomainJobDataFree); - virPortAllocatorRelease(port); - if (priv->mon) - qemuMonitorSetDomainLog(priv->mon, NULL, NULL, NULL); - VIR_FREE(priv->origname); - virDomainObjEndAPI(&vm); - virErrorRestore(&orig_err); - - /* Set a special error if Finish is expected to return NULL as a result of - * successful call with retcode != 0 - */ - if (retcode != 0 && !dom && virGetLastErrorCode() == VIR_ERR_OK) - virReportError(VIR_ERR_MIGRATE_FINISH_OK, NULL); - return dom; + goto cleanup; } -- 2.35.1
On Tue, May 10, 2022 at 17:20:47 +0200, Jiri Denemark wrote:
Most of the code in "endjob" label is executed only on failure. Let's duplicate the rest so that the label can be used only in error path making the success path easier to follow and refactor.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 45 ++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 19 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Let's call it "error" so that it's clear the label is only used in failure path. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 0c6d2c041b..4096aba0a6 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5840,7 +5840,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, cookiein, cookieinlen, cookie_flags))) - goto endjob; + goto error; if (flags & VIR_MIGRATE_OFFLINE) { if (retcode == 0 && @@ -5864,31 +5864,31 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * earlier than monitor EOF handler got a chance to process the error */ qemuDomainCheckMonitor(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN); - goto endjob; + goto error; } if (!virDomainObjIsActive(vm)) { virReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("guest unexpectedly quit")); qemuMigrationDstErrorReport(driver, vm->def->name); - goto endjob; + goto error; } if (qemuMigrationDstVPAssociatePortProfiles(vm->def) < 0) - goto endjob; + goto error; if (mig->network && qemuMigrationDstOPDRelocate(driver, vm, mig) < 0) VIR_WARN("unable to provide network data for relocation"); if (qemuMigrationDstStopNBDServer(driver, vm, mig) < 0) - goto endjob; + goto error; if (qemuRefreshVirtioChannelState(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN) < 0) - goto endjob; + goto error; if (qemuConnectAgent(driver, vm) < 0) - goto endjob; + goto error; if (flags & VIR_MIGRATE_PERSIST_DEST) { if (qemuMigrationDstPersist(driver, vm, mig, !v3proto) < 0) { @@ -5905,7 +5905,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * to restart during confirm() step, so we kill it off now. */ if (v3proto) - goto endjob; + goto error; } } @@ -5919,7 +5919,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * original domain on the source host is already gone. */ if (v3proto) - goto endjob; + goto error; } /* Now that the state data was transferred we can refresh the actual state @@ -5928,7 +5928,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, /* Similarly to the case above v2 protocol will not be able to recover * from this. Let's ignore this and perhaps stuff will not break. */ if (v3proto) - goto endjob; + goto error; } if (priv->job.current->status == VIR_DOMAIN_JOB_STATUS_POSTCOPY) @@ -5960,7 +5960,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * things up. */ if (v3proto) - goto endjob; + goto error; } if (inPostCopy) @@ -5983,7 +5983,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, if (qemuMigrationDstWaitForCompletion(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, false) < 0) { - goto endjob; + goto error; } if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) { virDomainObjSetState(vm, @@ -6060,7 +6060,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, virReportError(VIR_ERR_MIGRATE_FINISH_OK, NULL); return dom; - endjob: + error: if (virDomainObjIsActive(vm)) { if (doKill) { qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, -- 2.35.1
On Tue, May 10, 2022 at 17:20:48 +0200, Jiri Denemark wrote:
Let's call it "error" so that it's clear the label is only used in failure path.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The final part of Finish phase will be refactored into a dedicated function and we don't want to generate the cookie there. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 4096aba0a6..85ab4257d4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5979,12 +5979,27 @@ qemuMigrationDstFinish(virQEMUDriver *driver, qemuDomainJobDataUpdateDowntime(jobData); } + if (inPostCopy && + qemuMigrationDstWaitForCompletion(driver, vm, + VIR_ASYNC_JOB_MIGRATION_IN, + false) < 0) { + goto error; + } + + if (jobData) { + priv->job.completed = g_steal_pointer(&jobData); + priv->job.completed->status = VIR_DOMAIN_JOB_STATUS_COMPLETED; + qemuDomainJobSetStatsType(priv->job.completed, + QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); + } + + if (qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) + VIR_WARN("Unable to encode migration cookie"); + if (inPostCopy) { - if (qemuMigrationDstWaitForCompletion(driver, vm, - VIR_ASYNC_JOB_MIGRATION_IN, - false) < 0) { - goto error; - } if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) { virDomainObjSetState(vm, VIR_DOMAIN_RUNNING, @@ -6018,19 +6033,6 @@ qemuMigrationDstFinish(virQEMUDriver *driver, /* Guest is successfully running, so cancel previous auto destroy */ qemuProcessAutoDestroyRemove(driver, vm); - if (jobData) { - priv->job.completed = g_steal_pointer(&jobData); - priv->job.completed->status = VIR_DOMAIN_JOB_STATUS_COMPLETED; - qemuDomainJobSetStatsType(priv->job.completed, - QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); - } - - if (qemuMigrationCookieFormat(mig, driver, vm, - QEMU_MIGRATION_DESTINATION, - cookieout, cookieoutlen, - QEMU_MIGRATION_COOKIE_STATS) < 0) - VIR_WARN("Unable to encode migration cookie"); - /* Remove completed stats for post-copy, everything but timing fields * is obsolete anyway. */ -- 2.35.1
On Tue, May 10, 2022 at 17:20:49 +0200, Jiri Denemark wrote:
The final part of Finish phase will be refactored into a dedicated function and we don't want to generate the cookie there.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 38 ++++++++++++++++++++------------------ 1 file changed, 20 insertions(+), 18 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
By separating it into a dedicated qemuMigrationDstComplete function which can be later called in other places. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 99 ++++++++++++++++++++++----------------- src/qemu/qemu_migration.h | 6 +++ 2 files changed, 63 insertions(+), 42 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 85ab4257d4..a3d55c8e8b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5784,6 +5784,61 @@ qemuMigrationDstPersist(virQEMUDriver *driver, } +void +qemuMigrationDstComplete(virQEMUDriver *driver, + virDomainObj *vm, + bool inPostCopy, + virDomainAsyncJob asyncJob) +{ + qemuDomainObjPrivate *priv = vm->privateData; + qemuDomainJobPrivate *jobPriv = priv->job.privateData; + virObjectEvent *event; + + if (inPostCopy) { + if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) { + virDomainObjSetState(vm, + VIR_DOMAIN_RUNNING, + VIR_DOMAIN_RUNNING_MIGRATED); + } + + /* The only RESUME event during post-copy migration is triggered by + * QEMU when the running domain moves from the source to the + * destination host, but then the migration keeps running until all + * modified memory is transferred from the source host. This will + * result in VIR_DOMAIN_EVENT_RESUMED with RESUMED_POSTCOPY detail. + * However, our API documentation says we need to fire another RESUMED + * event at the very end of migration with RESUMED_MIGRATED detail. + */ + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_RESUMED, + VIR_DOMAIN_EVENT_RESUMED_MIGRATED); + virObjectEventStateQueue(driver->domainEventState, event); + } + + if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_PAUSED) { + virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, VIR_DOMAIN_PAUSED_USER); + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_SUSPENDED, + VIR_DOMAIN_EVENT_SUSPENDED_PAUSED); + virObjectEventStateQueue(driver->domainEventState, event); + } + + qemuDomainSaveStatus(vm); + + /* Guest is successfully running, so cancel previous auto destroy */ + qemuProcessAutoDestroyRemove(driver, vm); + + /* Remove completed stats for post-copy, everything but timing fields + * is obsolete anyway. + */ + if (inPostCopy) + g_clear_pointer(&priv->job.completed, virDomainJobDataFree); + + qemuMigrationParamsReset(driver, vm, asyncJob, jobPriv->migParams, + priv->job.apiFlags); +} + + virDomainPtr qemuMigrationDstFinish(virQEMUDriver *driver, virConnectPtr dconn, @@ -5999,48 +6054,8 @@ qemuMigrationDstFinish(virQEMUDriver *driver, QEMU_MIGRATION_COOKIE_STATS) < 0) VIR_WARN("Unable to encode migration cookie"); - if (inPostCopy) { - if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_RUNNING) { - virDomainObjSetState(vm, - VIR_DOMAIN_RUNNING, - VIR_DOMAIN_RUNNING_MIGRATED); - } - - /* The only RESUME event during post-copy migration is triggered by - * QEMU when the running domain moves from the source to the - * destination host, but then the migration keeps running until all - * modified memory is transferred from the source host. This will - * result in VIR_DOMAIN_EVENT_RESUMED with RESUMED_POSTCOPY detail. - * However, our API documentation says we need to fire another RESUMED - * event at the very end of migration with RESUMED_MIGRATED detail. - */ - event = virDomainEventLifecycleNewFromObj(vm, - VIR_DOMAIN_EVENT_RESUMED, - VIR_DOMAIN_EVENT_RESUMED_MIGRATED); - virObjectEventStateQueue(driver->domainEventState, event); - } - - if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_PAUSED) { - virDomainObjSetState(vm, VIR_DOMAIN_PAUSED, VIR_DOMAIN_PAUSED_USER); - event = virDomainEventLifecycleNewFromObj(vm, - VIR_DOMAIN_EVENT_SUSPENDED, - VIR_DOMAIN_EVENT_SUSPENDED_PAUSED); - virObjectEventStateQueue(driver->domainEventState, event); - } - - qemuDomainSaveStatus(vm); - - /* Guest is successfully running, so cancel previous auto destroy */ - qemuProcessAutoDestroyRemove(driver, vm); - - /* Remove completed stats for post-copy, everything but timing fields - * is obsolete anyway. - */ - if (inPostCopy) - g_clear_pointer(&priv->job.completed, virDomainJobDataFree); - - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, - jobPriv->migParams, priv->job.apiFlags); + qemuMigrationDstComplete(driver, vm, inPostCopy, + VIR_ASYNC_JOB_MIGRATION_IN); dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id); diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index c4e4228282..ecb0ad6302 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -191,6 +191,12 @@ qemuMigrationDstFinish(virQEMUDriver *driver, int retcode, bool v3proto); +void +qemuMigrationDstComplete(virQEMUDriver *driver, + virDomainObj *vm, + bool inPostCopy, + virDomainAsyncJob asyncJob); + int qemuMigrationSrcConfirm(virQEMUDriver *driver, virDomainObj *vm, -- 2.35.1
On Tue, May 10, 2022 at 17:20:50 +0200, Jiri Denemark wrote:
By separating it into a dedicated qemuMigrationDstComplete function which can be later called in other places.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 99 ++++++++++++++++++++++----------------- src/qemu/qemu_migration.h | 6 +++
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The comment about QEMU < 0.10.6 has been irrelevant for years. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a3d55c8e8b..a6c8e53ac5 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5990,10 +5990,6 @@ qemuMigrationDstFinish(virQEMUDriver *driver, inPostCopy = true; if (!(flags & VIR_MIGRATE_PAUSED)) { - /* run 'cont' on the destination, which allows migration on qemu - * >= 0.10.6 to work properly. This isn't strictly necessary on - * older qemu's, but it also doesn't hurt anything there - */ if (qemuProcessStartCPUs(driver, vm, inPostCopy ? VIR_DOMAIN_RUNNING_POSTCOPY : VIR_DOMAIN_RUNNING_MIGRATED, -- 2.35.1
On Tue, May 10, 2022 at 17:20:51 +0200, Jiri Denemark wrote:
The comment about QEMU < 0.10.6 has been irrelevant for years.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 4 ----
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
We want to prevent our error path that can potentially kill the domain on the destination host from overwriting an error reported earlier, but we were only doing so in one specific path when starting vCPUs fails. Let's do it in all paths. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a6c8e53ac5..bbea2f6e0e 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5997,11 +5997,6 @@ qemuMigrationDstFinish(virQEMUDriver *driver, if (virGetLastErrorCode() == VIR_ERR_OK) virReportError(VIR_ERR_INTERNAL_ERROR, "%s", _("resume operation failed")); - /* Need to save the current error, in case shutting - * down the process overwrites it - */ - virErrorPreserveLast(&orig_err); - /* * In v3 protocol, the source VM is still available to * restart during confirm() step, so we kill it off @@ -6074,6 +6069,10 @@ qemuMigrationDstFinish(virQEMUDriver *driver, return dom; error: + /* Need to save the current error, in case shutting down the process + * overwrites it. */ + virErrorPreserveLast(&orig_err); + if (virDomainObjIsActive(vm)) { if (doKill) { qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, -- 2.35.1
On Tue, May 10, 2022 at 17:20:52 +0200, Jiri Denemark wrote:
We want to prevent our error path that can potentially kill the domain on the destination host from overwriting an error reported earlier, but we were only doing so in one specific path when starting vCPUs fails. Let's do it in all paths.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph: - qemuMigrationDstFinish - qemuMigrationDstFinishOffline - qemuMigrationDstFinishActive - qemuMigrationDstFinishFresh - qemuMigrationDstFinishResume Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 234 +++++++++++++++++++++----------------- 1 file changed, 129 insertions(+), 105 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index bbea2f6e0e..385bd91a6b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5839,111 +5839,38 @@ qemuMigrationDstComplete(virQEMUDriver *driver, } -virDomainPtr -qemuMigrationDstFinish(virQEMUDriver *driver, - virConnectPtr dconn, - virDomainObj *vm, - const char *cookiein, - int cookieinlen, - char **cookieout, - int *cookieoutlen, - unsigned long flags, - int retcode, - bool v3proto) +/* + * Perform Finish phase of a fresh (i.e., not recovery) migration of an active + * domain. + */ +static int +qemuMigrationDstFinishFresh(virQEMUDriver *driver, + virDomainObj *vm, + qemuMigrationCookie *mig, + unsigned long flags, + bool v3proto, + unsigned long long timeReceived, + bool *doKill, + bool *inPostCopy) { - virDomainPtr dom = NULL; - g_autoptr(qemuMigrationCookie) mig = NULL; - virErrorPtr orig_err = NULL; - int cookie_flags = 0; qemuDomainObjPrivate *priv = vm->privateData; - qemuDomainJobPrivate *jobPriv = priv->job.privateData; - unsigned short port; - unsigned long long timeReceived = 0; - virObjectEvent *event; - virDomainJobData *jobData = NULL; - bool inPostCopy = false; - bool doKill = true; - - VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " - "cookieout=%p, cookieoutlen=%p, flags=0x%lx, retcode=%d", - driver, dconn, vm, NULLSTR(cookiein), cookieinlen, - cookieout, cookieoutlen, flags, retcode); - - port = priv->migrationPort; - priv->migrationPort = 0; - - if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_IN)) { - qemuMigrationDstErrorReport(driver, vm->def->name); - goto cleanup; - } - - ignore_value(virTimeMillisNow(&timeReceived)); - - qemuMigrationJobStartPhase(vm, - v3proto ? QEMU_MIGRATION_PHASE_FINISH3 - : QEMU_MIGRATION_PHASE_FINISH2); - - qemuDomainCleanupRemove(vm, qemuMigrationDstPrepareCleanup); - g_clear_pointer(&priv->job.completed, virDomainJobDataFree); - - cookie_flags = QEMU_MIGRATION_COOKIE_NETWORK | - QEMU_MIGRATION_COOKIE_STATS | - QEMU_MIGRATION_COOKIE_NBD; - /* Some older versions of libvirt always send persistent XML in the cookie - * even though VIR_MIGRATE_PERSIST_DEST was not used. */ - cookie_flags |= QEMU_MIGRATION_COOKIE_PERSISTENT; - - if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, - cookiein, cookieinlen, cookie_flags))) - goto error; - - if (flags & VIR_MIGRATE_OFFLINE) { - if (retcode == 0 && - qemuMigrationDstPersist(driver, vm, mig, false) == 0) { - dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1); - - if (dom && - qemuMigrationCookieFormat(mig, driver, vm, - QEMU_MIGRATION_DESTINATION, - cookieout, cookieoutlen, - QEMU_MIGRATION_COOKIE_STATS) < 0) - VIR_WARN("Unable to encode migration cookie"); - } - - qemuMigrationJobFinish(vm); - goto cleanup; - } - - if (retcode != 0) { - /* Check for a possible error on the monitor in case Finish was called - * earlier than monitor EOF handler got a chance to process the error - */ - qemuDomainCheckMonitor(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN); - goto error; - } - - if (!virDomainObjIsActive(vm)) { - virReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("guest unexpectedly quit")); - qemuMigrationDstErrorReport(driver, vm->def->name); - goto error; - } + g_autoptr(virDomainJobData) jobData = NULL; if (qemuMigrationDstVPAssociatePortProfiles(vm->def) < 0) - goto error; + return -1; if (mig->network && qemuMigrationDstOPDRelocate(driver, vm, mig) < 0) VIR_WARN("unable to provide network data for relocation"); if (qemuMigrationDstStopNBDServer(driver, vm, mig) < 0) - goto error; + return -1; if (qemuRefreshVirtioChannelState(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN) < 0) - goto error; + return -1; if (qemuConnectAgent(driver, vm) < 0) - goto error; + return -1; if (flags & VIR_MIGRATE_PERSIST_DEST) { if (qemuMigrationDstPersist(driver, vm, mig, !v3proto) < 0) { @@ -5960,7 +5887,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * to restart during confirm() step, so we kill it off now. */ if (v3proto) - goto error; + return -1; } } @@ -5974,7 +5901,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * original domain on the source host is already gone. */ if (v3proto) - goto error; + return -1; } /* Now that the state data was transferred we can refresh the actual state @@ -5983,16 +5910,16 @@ qemuMigrationDstFinish(virQEMUDriver *driver, /* Similarly to the case above v2 protocol will not be able to recover * from this. Let's ignore this and perhaps stuff will not break. */ if (v3proto) - goto error; + return -1; } if (priv->job.current->status == VIR_DOMAIN_JOB_STATUS_POSTCOPY) - inPostCopy = true; + *inPostCopy = true; if (!(flags & VIR_MIGRATE_PAUSED)) { if (qemuProcessStartCPUs(driver, vm, - inPostCopy ? VIR_DOMAIN_RUNNING_POSTCOPY - : VIR_DOMAIN_RUNNING_MIGRATED, + *inPostCopy ? VIR_DOMAIN_RUNNING_POSTCOPY + : VIR_DOMAIN_RUNNING_MIGRATED, VIR_ASYNC_JOB_MIGRATION_IN) < 0) { if (virGetLastErrorCode() == VIR_ERR_OK) virReportError(VIR_ERR_INTERNAL_ERROR, @@ -6006,11 +5933,11 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * things up. */ if (v3proto) - goto error; + return -1; } - if (inPostCopy) - doKill = false; + if (*inPostCopy) + *doKill = false; } if (mig->jobData) { @@ -6025,11 +5952,11 @@ qemuMigrationDstFinish(virQEMUDriver *driver, qemuDomainJobDataUpdateDowntime(jobData); } - if (inPostCopy && + if (*inPostCopy && qemuMigrationDstWaitForCompletion(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, false) < 0) { - goto error; + return -1; } if (jobData) { @@ -6039,6 +5966,106 @@ qemuMigrationDstFinish(virQEMUDriver *driver, QEMU_DOMAIN_JOB_STATS_TYPE_MIGRATION); } + return 0; +} + + +virDomainPtr +qemuMigrationDstFinish(virQEMUDriver *driver, + virConnectPtr dconn, + virDomainObj *vm, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + unsigned long flags, + int retcode, + bool v3proto) +{ + virDomainPtr dom = NULL; + g_autoptr(qemuMigrationCookie) mig = NULL; + virErrorPtr orig_err = NULL; + int cookie_flags = 0; + qemuDomainObjPrivate *priv = vm->privateData; + qemuDomainJobPrivate *jobPriv = priv->job.privateData; + unsigned short port; + unsigned long long timeReceived = 0; + virObjectEvent *event; + bool inPostCopy = false; + bool doKill = true; + int rc; + + VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " + "cookieout=%p, cookieoutlen=%p, flags=0x%lx, retcode=%d", + driver, dconn, vm, NULLSTR(cookiein), cookieinlen, + cookieout, cookieoutlen, flags, retcode); + + port = priv->migrationPort; + priv->migrationPort = 0; + + if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_IN)) { + qemuMigrationDstErrorReport(driver, vm->def->name); + goto cleanup; + } + + ignore_value(virTimeMillisNow(&timeReceived)); + + qemuMigrationJobStartPhase(vm, + v3proto ? QEMU_MIGRATION_PHASE_FINISH3 + : QEMU_MIGRATION_PHASE_FINISH2); + + qemuDomainCleanupRemove(vm, qemuMigrationDstPrepareCleanup); + g_clear_pointer(&priv->job.completed, virDomainJobDataFree); + + cookie_flags = QEMU_MIGRATION_COOKIE_NETWORK | + QEMU_MIGRATION_COOKIE_STATS | + QEMU_MIGRATION_COOKIE_NBD; + /* Some older versions of libvirt always send persistent XML in the cookie + * even though VIR_MIGRATE_PERSIST_DEST was not used. */ + cookie_flags |= QEMU_MIGRATION_COOKIE_PERSISTENT; + + if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, + cookiein, cookieinlen, cookie_flags))) + goto error; + + if (flags & VIR_MIGRATE_OFFLINE) { + if (retcode == 0 && + qemuMigrationDstPersist(driver, vm, mig, false) == 0) { + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1); + + if (dom && + qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) + VIR_WARN("Unable to encode migration cookie"); + } + + qemuMigrationJobFinish(vm); + goto cleanup; + } + + if (retcode != 0) { + /* Check for a possible error on the monitor in case Finish was called + * earlier than monitor EOF handler got a chance to process the error + */ + qemuDomainCheckMonitor(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN); + goto error; + } + + if (!virDomainObjIsActive(vm)) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("guest unexpectedly quit")); + qemuMigrationDstErrorReport(driver, vm->def->name); + goto error; + } + + rc = qemuMigrationDstFinishFresh(driver, vm, mig, flags, v3proto, + timeReceived, &doKill, &inPostCopy); + if (rc < 0 || + !(dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id))) + goto error; + if (qemuMigrationCookieFormat(mig, driver, vm, QEMU_MIGRATION_DESTINATION, cookieout, cookieoutlen, @@ -6048,12 +6075,9 @@ qemuMigrationDstFinish(virQEMUDriver *driver, qemuMigrationDstComplete(driver, vm, inPostCopy, VIR_ASYNC_JOB_MIGRATION_IN); - dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id); - qemuMigrationJobFinish(vm); cleanup: - g_clear_pointer(&jobData, virDomainJobDataFree); virPortAllocatorRelease(port); if (priv->mon) qemuMonitorSetDomainLog(priv->mon, NULL, NULL, NULL); -- 2.35.1
On Tue, May 10, 2022 at 17:20:53 +0200, Jiri Denemark wrote:
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph:
- qemuMigrationDstFinish - qemuMigrationDstFinishOffline - qemuMigrationDstFinishActive - qemuMigrationDstFinishFresh - qemuMigrationDstFinishResume
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 234 +++++++++++++++++++++----------------- 1 file changed, 129 insertions(+), 105 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph: - qemuMigrationDstFinish - qemuMigrationDstFinishOffline - qemuMigrationDstFinishActive - qemuMigrationDstFinishFresh - qemuMigrationDstFinishResume Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 385bd91a6b..dcd7ff3597 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5839,6 +5839,32 @@ qemuMigrationDstComplete(virQEMUDriver *driver, } +static virDomainPtr +qemuMigrationDstFinishOffline(virQEMUDriver *driver, + virConnectPtr dconn, + virDomainObj *vm, + qemuMigrationCookie *mig, + char **cookieout, + int *cookieoutlen) +{ + virDomainPtr dom = NULL; + + if (qemuMigrationDstPersist(driver, vm, mig, false) < 0) + return NULL; + + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1); + + if (dom && + qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) + VIR_WARN("Unable to encode migration cookie"); + + return dom; +} + + /* * Perform Finish phase of a fresh (i.e., not recovery) migration of an active * domain. @@ -6029,16 +6055,9 @@ qemuMigrationDstFinish(virQEMUDriver *driver, goto error; if (flags & VIR_MIGRATE_OFFLINE) { - if (retcode == 0 && - qemuMigrationDstPersist(driver, vm, mig, false) == 0) { - dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1); - - if (dom && - qemuMigrationCookieFormat(mig, driver, vm, - QEMU_MIGRATION_DESTINATION, - cookieout, cookieoutlen, - QEMU_MIGRATION_COOKIE_STATS) < 0) - VIR_WARN("Unable to encode migration cookie"); + if (retcode == 0) { + dom = qemuMigrationDstFinishOffline(driver, dconn, vm, mig, + cookieout, cookieoutlen); } qemuMigrationJobFinish(vm); -- 2.35.1
On Tue, May 10, 2022 at 17:20:54 +0200, Jiri Denemark wrote:
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph:
- qemuMigrationDstFinish - qemuMigrationDstFinishOffline - qemuMigrationDstFinishActive - qemuMigrationDstFinishFresh - qemuMigrationDstFinishResume
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 385bd91a6b..dcd7ff3597 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5839,6 +5839,32 @@ qemuMigrationDstComplete(virQEMUDriver *driver, }
+static virDomainPtr +qemuMigrationDstFinishOffline(virQEMUDriver *driver, + virConnectPtr dconn, + virDomainObj *vm, + qemuMigrationCookie *mig, + char **cookieout, + int *cookieoutlen) +{ + virDomainPtr dom = NULL; + + if (qemuMigrationDstPersist(driver, vm, mig, false) < 0) + return NULL; + + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1); + + if (dom &&
Earlier I've commented about uselessness of this check. Other code formats the cookie unconditionally.
+ qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) + VIR_WARN("Unable to encode migration cookie"); + + return dom; +}
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Wed, May 11, 2022 at 16:53:48 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:20:54 +0200, Jiri Denemark wrote:
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph:
- qemuMigrationDstFinish - qemuMigrationDstFinishOffline - qemuMigrationDstFinishActive - qemuMigrationDstFinishFresh - qemuMigrationDstFinishResume
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 39 +++++++++++++++++++++++++++++---------- 1 file changed, 29 insertions(+), 10 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 385bd91a6b..dcd7ff3597 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5839,6 +5839,32 @@ qemuMigrationDstComplete(virQEMUDriver *driver, }
+static virDomainPtr +qemuMigrationDstFinishOffline(virQEMUDriver *driver, + virConnectPtr dconn, + virDomainObj *vm, + qemuMigrationCookie *mig, + char **cookieout, + int *cookieoutlen) +{ + virDomainPtr dom = NULL; + + if (qemuMigrationDstPersist(driver, vm, mig, false) < 0) + return NULL; + + dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, -1); + + if (dom &&
Earlier I've commented about uselessness of this check. Other code formats the cookie unconditionally.
That's because the cookie is formatted before calling virGetDomain there. We could swap it here too, or drop the check, but we can as well keep it as is and avoid formatting the cookie when virGetDomain fails (even though the failure is mostly theoretical). Jirka
To keep all cookie handling (parsing and formatting) in the same function. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index dcd7ff3597..170d99d789 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5843,11 +5843,19 @@ static virDomainPtr qemuMigrationDstFinishOffline(virQEMUDriver *driver, virConnectPtr dconn, virDomainObj *vm, - qemuMigrationCookie *mig, + int cookie_flags, + const char *cookiein, + int cookieinlen, char **cookieout, int *cookieoutlen) { virDomainPtr dom = NULL; + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(qemuMigrationCookie) mig = NULL; + + if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, + cookiein, cookieinlen, cookie_flags))) + return NULL; if (qemuMigrationDstPersist(driver, vm, mig, false) < 0) return NULL; @@ -6050,13 +6058,11 @@ qemuMigrationDstFinish(virQEMUDriver *driver, * even though VIR_MIGRATE_PERSIST_DEST was not used. */ cookie_flags |= QEMU_MIGRATION_COOKIE_PERSISTENT; - if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, - cookiein, cookieinlen, cookie_flags))) - goto error; - if (flags & VIR_MIGRATE_OFFLINE) { if (retcode == 0) { - dom = qemuMigrationDstFinishOffline(driver, dconn, vm, mig, + dom = qemuMigrationDstFinishOffline(driver, dconn, vm, + cookie_flags, + cookiein, cookieinlen, cookieout, cookieoutlen); } @@ -6064,6 +6070,10 @@ qemuMigrationDstFinish(virQEMUDriver *driver, goto cleanup; } + if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, + cookiein, cookieinlen, cookie_flags))) + goto error; + if (retcode != 0) { /* Check for a possible error on the monitor in case Finish was called * earlier than monitor EOF handler got a chance to process the error -- 2.35.1
On Tue, May 10, 2022 at 17:20:55 +0200, Jiri Denemark wrote:
To keep all cookie handling (parsing and formatting) in the same function.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph: - qemuMigrationDstFinish - qemuMigrationDstFinishOffline - qemuMigrationDstFinishActive - qemuMigrationDstFinishFresh - qemuMigrationDstFinishResume Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 173 +++++++++++++++++++++++--------------- 1 file changed, 103 insertions(+), 70 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 170d99d789..d02e8132e4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -6004,71 +6004,32 @@ qemuMigrationDstFinishFresh(virQEMUDriver *driver, } -virDomainPtr -qemuMigrationDstFinish(virQEMUDriver *driver, - virConnectPtr dconn, - virDomainObj *vm, - const char *cookiein, - int cookieinlen, - char **cookieout, - int *cookieoutlen, - unsigned long flags, - int retcode, - bool v3proto) +static virDomainPtr +qemuMigrationDstFinishActive(virQEMUDriver *driver, + virConnectPtr dconn, + virDomainObj *vm, + int cookie_flags, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + unsigned long flags, + int retcode, + bool v3proto, + unsigned long long timeReceived, + virErrorPtr *orig_err) { virDomainPtr dom = NULL; g_autoptr(qemuMigrationCookie) mig = NULL; - virErrorPtr orig_err = NULL; - int cookie_flags = 0; qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobPrivate *jobPriv = priv->job.privateData; - unsigned short port; - unsigned long long timeReceived = 0; virObjectEvent *event; bool inPostCopy = false; bool doKill = true; int rc; - VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " - "cookieout=%p, cookieoutlen=%p, flags=0x%lx, retcode=%d", - driver, dconn, vm, NULLSTR(cookiein), cookieinlen, - cookieout, cookieoutlen, flags, retcode); - - port = priv->migrationPort; - priv->migrationPort = 0; - - if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_IN)) { - qemuMigrationDstErrorReport(driver, vm->def->name); - goto cleanup; - } - - ignore_value(virTimeMillisNow(&timeReceived)); - - qemuMigrationJobStartPhase(vm, - v3proto ? QEMU_MIGRATION_PHASE_FINISH3 - : QEMU_MIGRATION_PHASE_FINISH2); - - qemuDomainCleanupRemove(vm, qemuMigrationDstPrepareCleanup); - g_clear_pointer(&priv->job.completed, virDomainJobDataFree); - - cookie_flags = QEMU_MIGRATION_COOKIE_NETWORK | - QEMU_MIGRATION_COOKIE_STATS | - QEMU_MIGRATION_COOKIE_NBD; - /* Some older versions of libvirt always send persistent XML in the cookie - * even though VIR_MIGRATE_PERSIST_DEST was not used. */ - cookie_flags |= QEMU_MIGRATION_COOKIE_PERSISTENT; - - if (flags & VIR_MIGRATE_OFFLINE) { - if (retcode == 0) { - dom = qemuMigrationDstFinishOffline(driver, dconn, vm, - cookie_flags, - cookiein, cookieinlen, - cookieout, cookieoutlen); - } - - qemuMigrationJobFinish(vm); - goto cleanup; - } + VIR_DEBUG("vm=%p, flags=0x%lx, retcode=%d", + vm, flags, retcode); if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, cookiein, cookieinlen, cookie_flags))) @@ -6106,25 +6067,12 @@ qemuMigrationDstFinish(virQEMUDriver *driver, qemuMigrationJobFinish(vm); - cleanup: - virPortAllocatorRelease(port); - if (priv->mon) - qemuMonitorSetDomainLog(priv->mon, NULL, NULL, NULL); - VIR_FREE(priv->origname); - virDomainObjEndAPI(&vm); - virErrorRestore(&orig_err); - - /* Set a special error if Finish is expected to return NULL as a result of - * successful call with retcode != 0 - */ - if (retcode != 0 && !dom && virGetLastErrorCode() == VIR_ERR_OK) - virReportError(VIR_ERR_MIGRATE_FINISH_OK, NULL); return dom; error: /* Need to save the current error, in case shutting down the process * overwrites it. */ - virErrorPreserveLast(&orig_err); + virErrorPreserveLast(orig_err); if (virDomainObjIsActive(vm)) { if (doKill) { @@ -6155,7 +6103,92 @@ qemuMigrationDstFinish(virQEMUDriver *driver, if (!virDomainObjIsActive(vm)) qemuDomainRemoveInactive(driver, vm); - goto cleanup; + return NULL; +} + + +virDomainPtr +qemuMigrationDstFinish(virQEMUDriver *driver, + virConnectPtr dconn, + virDomainObj *vm, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + unsigned long flags, + int retcode, + bool v3proto) +{ + virDomainPtr dom = NULL; + virErrorPtr orig_err = NULL; + int cookie_flags = 0; + qemuDomainObjPrivate *priv = vm->privateData; + unsigned short port; + unsigned long long timeReceived = 0; + + VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " + "cookieout=%p, cookieoutlen=%p, flags=0x%lx, retcode=%d", + driver, dconn, vm, NULLSTR(cookiein), cookieinlen, + cookieout, cookieoutlen, flags, retcode); + + port = priv->migrationPort; + priv->migrationPort = 0; + + if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_IN)) { + qemuMigrationDstErrorReport(driver, vm->def->name); + goto cleanup; + } + + ignore_value(virTimeMillisNow(&timeReceived)); + + qemuMigrationJobStartPhase(vm, + v3proto ? QEMU_MIGRATION_PHASE_FINISH3 + : QEMU_MIGRATION_PHASE_FINISH2); + + qemuDomainCleanupRemove(vm, qemuMigrationDstPrepareCleanup); + g_clear_pointer(&priv->job.completed, virDomainJobDataFree); + + cookie_flags = QEMU_MIGRATION_COOKIE_NETWORK | + QEMU_MIGRATION_COOKIE_STATS | + QEMU_MIGRATION_COOKIE_NBD; + /* Some older versions of libvirt always send persistent XML in the cookie + * even though VIR_MIGRATE_PERSIST_DEST was not used. */ + cookie_flags |= QEMU_MIGRATION_COOKIE_PERSISTENT; + + if (flags & VIR_MIGRATE_OFFLINE) { + if (retcode == 0) { + dom = qemuMigrationDstFinishOffline(driver, dconn, vm, + cookie_flags, + cookiein, cookieinlen, + cookieout, cookieoutlen); + } + + qemuMigrationJobFinish(vm); + goto cleanup; + } + + dom = qemuMigrationDstFinishActive(driver, dconn, vm, cookie_flags, + cookiein, cookieinlen, + cookieout, cookieoutlen, + flags, retcode, v3proto, timeReceived, + &orig_err); + if (!dom) + goto cleanup; + + cleanup: + virPortAllocatorRelease(port); + if (priv->mon) + qemuMonitorSetDomainLog(priv->mon, NULL, NULL, NULL); + VIR_FREE(priv->origname); + virDomainObjEndAPI(&vm); + virErrorRestore(&orig_err); + + /* Set a special error if Finish is expected to return NULL as a result of + * successful call with retcode != 0 + */ + if (retcode != 0 && !dom && virGetLastErrorCode() == VIR_ERR_OK) + virReportError(VIR_ERR_MIGRATE_FINISH_OK, NULL); + return dom; } -- 2.35.1
On Tue, May 10, 2022 at 17:20:56 +0200, Jiri Denemark wrote:
Refactors qemuMigrationDstFinish by moving some parts to a dedicated function for easier introduction of postcopy resume code without duplicating common parts of the Finish phase. The goal is to have the following call graph:
- qemuMigrationDstFinish - qemuMigrationDstFinishOffline - qemuMigrationDstFinishActive - qemuMigrationDstFinishFresh - qemuMigrationDstFinishResume
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 173 +++++++++++++++++++++++--------------- 1 file changed, 103 insertions(+), 70 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 170d99d789..d02e8132e4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -6004,71 +6004,32 @@ qemuMigrationDstFinishFresh(virQEMUDriver *driver, }
-virDomainPtr -qemuMigrationDstFinish(virQEMUDriver *driver, - virConnectPtr dconn, - virDomainObj *vm, - const char *cookiein, - int cookieinlen, - char **cookieout, - int *cookieoutlen, - unsigned long flags, - int retcode, - bool v3proto) +static virDomainPtr +qemuMigrationDstFinishActive(virQEMUDriver *driver, + virConnectPtr dconn, + virDomainObj *vm, + int cookie_flags, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + unsigned long flags, + int retcode, + bool v3proto, + unsigned long long timeReceived, + virErrorPtr *orig_err)
The argument list is getting a bit riddiculous. Especially weird is that you pass 'orig_err' in. The only error that could overwrite it is from virPortAllocatorRelease and that one IMNSHO should not report any errrors or at least have a quiet version. Please don't pass the error out, but re-set it and deal with the caller separately. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The function which started a migration phase should also finish it by calling qemuMigrationJobFinish/qemuMigrationJobContinue so that the code is easier to follow. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index d02e8132e4..b62f7256c4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -6017,7 +6017,8 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, int retcode, bool v3proto, unsigned long long timeReceived, - virErrorPtr *orig_err) + virErrorPtr *orig_err, + bool *finishJob) { virDomainPtr dom = NULL; g_autoptr(qemuMigrationCookie) mig = NULL; @@ -6065,8 +6066,6 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, qemuMigrationDstComplete(driver, vm, inPostCopy, VIR_ASYNC_JOB_MIGRATION_IN); - qemuMigrationJobFinish(vm); - return dom; error: @@ -6092,12 +6091,10 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, if (virDomainObjIsFailedPostcopy(vm)) { qemuProcessAutoDestroyRemove(driver, vm); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + *finishJob = false; } else { qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, jobPriv->migParams, priv->job.apiFlags); - - qemuMigrationJobFinish(vm); } if (!virDomainObjIsActive(vm)) @@ -6162,18 +6159,21 @@ qemuMigrationDstFinish(virQEMUDriver *driver, cookiein, cookieinlen, cookieout, cookieoutlen); } + } else { + bool finishJob = true; - qemuMigrationJobFinish(vm); - goto cleanup; + dom = qemuMigrationDstFinishActive(driver, dconn, vm, cookie_flags, + cookiein, cookieinlen, + cookieout, cookieoutlen, + flags, retcode, v3proto, timeReceived, + &orig_err, &finishJob); + if (!finishJob) { + qemuMigrationJobContinue(vm); + goto cleanup; + } } - dom = qemuMigrationDstFinishActive(driver, dconn, vm, cookie_flags, - cookiein, cookieinlen, - cookieout, cookieoutlen, - flags, retcode, v3proto, timeReceived, - &orig_err); - if (!dom) - goto cleanup; + qemuMigrationJobFinish(vm); cleanup: virPortAllocatorRelease(port); -- 2.35.1
On Tue, May 10, 2022 at 17:20:57 +0200, Jiri Denemark wrote:
The function which started a migration phase should also finish it by calling qemuMigrationJobFinish/qemuMigrationJobContinue so that the code is easier to follow.
[1]
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index d02e8132e4..b62f7256c4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -6017,7 +6017,8 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, int retcode, bool v3proto, unsigned long long timeReceived, - virErrorPtr *orig_err) + virErrorPtr *orig_err, + bool *finishJob)
Come on! Really?!? [1]
{ virDomainPtr dom = NULL; g_autoptr(qemuMigrationCookie) mig = NULL;
[....]
@@ -6162,18 +6159,21 @@ qemuMigrationDstFinish(virQEMUDriver *driver, cookiein, cookieinlen, cookieout, cookieoutlen); } + } else { + bool finishJob = true;
- qemuMigrationJobFinish(vm); - goto cleanup; + dom = qemuMigrationDstFinishActive(driver, dconn, vm, cookie_flags, + cookiein, cookieinlen, + cookieout, cookieoutlen, + flags, retcode, v3proto, timeReceived, + &orig_err, &finishJob); + if (!finishJob) { + qemuMigrationJobContinue(vm); + goto cleanup; + } }
If I overlook the fact that you have to remember what qemuMigrationDstFinishActive you used probably the least easy variant of seeing what is happening with that extra jump. Do it without that jump: if (finishJob) qemuMigrationJobFinish(vm); else qemuMigrationJobContinue(vm); With that I'll maybe buy the argument from the commit message, thus: Reviewed-by: Peter Krempa <pkrempa@redhat.com>
- dom = qemuMigrationDstFinishActive(driver, dconn, vm, cookie_flags, - cookiein, cookieinlen, - cookieout, cookieoutlen, - flags, retcode, v3proto, timeReceived, - &orig_err); - if (!dom) - goto cleanup; + qemuMigrationJobFinish(vm);
cleanup: virPortAllocatorRelease(port); -- 2.35.1
By separating it into a dedicated qemuMigrationSrcComplete function which can be later called in other places. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 70 ++++++++++++++++++++++++--------------- src/qemu/qemu_migration.h | 5 +++ 2 files changed, 48 insertions(+), 27 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index b62f7256c4..903c1dbf6b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3518,6 +3518,48 @@ qemuMigrationAnyPrepareDef(virQEMUDriver *driver, } +void +qemuMigrationSrcComplete(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob asyncJob) +{ + qemuDomainObjPrivate *priv = vm->privateData; + virDomainJobData *jobData = priv->job.completed; + virObjectEvent *event; + int reason; + + if (jobData) { + /* We need to refresh migration statistics after a completed post-copy + * migration since jobData contains obsolete data from the time we + * switched to post-copy mode. + */ + if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && + reason == VIR_DOMAIN_PAUSED_POSTCOPY) { + VIR_DEBUG("Refreshing migration statistics"); + if (qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + jobData, NULL) < 0) + VIR_WARN("Could not refresh migration statistics"); + } + + qemuDomainJobDataUpdateTime(jobData); + } + + /* If guest uses SPICE and supports seamless migration we have to hold + * up domain shutdown until SPICE server transfers its data */ + qemuMigrationSrcWaitForSpice(vm); + + qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_MIGRATED, asyncJob, + VIR_QEMU_PROCESS_STOP_MIGRATED); + virDomainAuditStop(vm, "migrated"); + + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_STOPPED, + VIR_DOMAIN_EVENT_STOPPED_MIGRATED); + virObjectEventStateQueue(driver->domainEventState, event); + qemuDomainEventEmitJobCompleted(driver, vm); +} + + static int qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, virDomainObj *vm, @@ -3527,7 +3569,6 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, int retcode) { g_autoptr(qemuMigrationCookie) mig = NULL; - virObjectEvent *event; qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobPrivate *jobPriv = priv->job.privateData; virDomainJobData *jobData = NULL; @@ -3564,21 +3605,9 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, /* Update times with the values sent by the destination daemon */ if (mig->jobData && jobData) { - int reason; qemuDomainJobDataPrivate *privJob = jobData->privateData; qemuDomainJobDataPrivate *privMigJob = mig->jobData->privateData; - /* We need to refresh migration statistics after a completed post-copy - * migration since priv->job.completed contains obsolete data from the - * time we switched to post-copy mode. - */ - if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && - reason == VIR_DOMAIN_PAUSED_POSTCOPY && - qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, - jobData, NULL) < 0) - VIR_WARN("Could not refresh migration statistics"); - - qemuDomainJobDataUpdateTime(jobData); jobData->timeDeltaSet = mig->jobData->timeDeltaSet; jobData->timeDelta = mig->jobData->timeDelta; privJob->stats.mig.downtime_set = privMigJob->stats.mig.downtime_set; @@ -3592,20 +3621,7 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, * If something failed, resume CPUs, but only if we didn't use post-copy. */ if (retcode == 0) { - /* If guest uses SPICE and supports seamless migration we have to hold - * up domain shutdown until SPICE server transfers its data */ - qemuMigrationSrcWaitForSpice(vm); - - qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_MIGRATED, - VIR_ASYNC_JOB_MIGRATION_OUT, - VIR_QEMU_PROCESS_STOP_MIGRATED); - virDomainAuditStop(vm, "migrated"); - - event = virDomainEventLifecycleNewFromObj(vm, - VIR_DOMAIN_EVENT_STOPPED, - VIR_DOMAIN_EVENT_STOPPED_MIGRATED); - virObjectEventStateQueue(driver->domainEventState, event); - qemuDomainEventEmitJobCompleted(driver, vm); + qemuMigrationSrcComplete(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT); } else { virErrorPtr orig_err; int reason; diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index ecb0ad6302..1d6051859b 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -205,6 +205,11 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, unsigned int flags, int cancelled); +void +qemuMigrationSrcComplete(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob asyncJob); + bool qemuMigrationSrcIsAllowed(virQEMUDriver *driver, virDomainObj *vm, -- 2.35.1
On Tue, May 10, 2022 at 17:20:58 +0200, Jiri Denemark wrote:
By separating it into a dedicated qemuMigrationSrcComplete function which can be later called in other places.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 70 ++++++++++++++++++++++++--------------- src/qemu/qemu_migration.h | 5 +++ 2 files changed, 48 insertions(+), 27 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index b62f7256c4..903c1dbf6b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3518,6 +3518,48 @@ qemuMigrationAnyPrepareDef(virQEMUDriver *driver, }
+void +qemuMigrationSrcComplete(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob asyncJob) +{ + qemuDomainObjPrivate *priv = vm->privateData; + virDomainJobData *jobData = priv->job.completed; + virObjectEvent *event; + int reason; + + if (jobData) { + /* We need to refresh migration statistics after a completed post-copy + * migration since jobData contains obsolete data from the time we + * switched to post-copy mode. + */ + if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && + reason == VIR_DOMAIN_PAUSED_POSTCOPY) { + VIR_DEBUG("Refreshing migration statistics"); + if (qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + jobData, NULL) < 0) + VIR_WARN("Could not refresh migration statistics"); + } + + qemuDomainJobDataUpdateTime(jobData); + } + + /* If guest uses SPICE and supports seamless migration we have to hold + * up domain shutdown until SPICE server transfers its data */ + qemuMigrationSrcWaitForSpice(vm); + + qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_MIGRATED, asyncJob, + VIR_QEMU_PROCESS_STOP_MIGRATED); + virDomainAuditStop(vm, "migrated"); + + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_STOPPED, + VIR_DOMAIN_EVENT_STOPPED_MIGRATED);
Fix the alignment here. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Normally migrationPort is released in the Finish phase, but we need to make sure it is properly released also in case qemuMigrationDstFinish is not called at all. Currently the only callback which is called in this situation qemuMigrationDstPrepareCleanup which already releases migrationPort. This patch adds similar handling to additional callbacks which will be used in the future. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 3 +++ src/qemu/qemu_process.c | 2 ++ 2 files changed, 5 insertions(+) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 903c1dbf6b..ad9be3e68e 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5852,6 +5852,9 @@ qemuMigrationDstComplete(virQEMUDriver *driver, qemuMigrationParamsReset(driver, vm, asyncJob, jobPriv->migParams, priv->job.apiFlags); + + virPortAllocatorRelease(priv->migrationPort); + priv->migrationPort = 0; } diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index c3a966983f..d92cd8cb5e 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3488,6 +3488,8 @@ qemuProcessCleanupMigrationJob(virQEMUDriver *driver, priv->job.asyncJob != VIR_ASYNC_JOB_MIGRATION_OUT) return; + virPortAllocatorRelease(priv->migrationPort); + priv->migrationPort = 0; qemuDomainObjDiscardAsyncJob(vm); } -- 2.35.1
On Tue, May 10, 2022 at 17:20:59 +0200, Jiri Denemark wrote:
Normally migrationPort is released in the Finish phase, but we need to make sure it is properly released also in case qemuMigrationDstFinish is not called at all. Currently the only callback which is called in this situation qemuMigrationDstPrepareCleanup which already releases migrationPort. This patch adds similar handling to additional callbacks which will be used in the future.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 3 +++ src/qemu/qemu_process.c | 2 ++ 2 files changed, 5 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
When reconnecting to an active domains we need to use a different job structure than the one referenced from the VM object. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 11 ++++++----- src/qemu/qemu_migration.h | 3 ++- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index ad9be3e68e..6938e0fafc 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5804,10 +5804,11 @@ void qemuMigrationDstComplete(virQEMUDriver *driver, virDomainObj *vm, bool inPostCopy, - virDomainAsyncJob asyncJob) + virDomainAsyncJob asyncJob, + qemuDomainJobObj *job) { qemuDomainObjPrivate *priv = vm->privateData; - qemuDomainJobPrivate *jobPriv = priv->job.privateData; + qemuDomainJobPrivate *jobPriv = job->privateData; virObjectEvent *event; if (inPostCopy) { @@ -5848,10 +5849,10 @@ qemuMigrationDstComplete(virQEMUDriver *driver, * is obsolete anyway. */ if (inPostCopy) - g_clear_pointer(&priv->job.completed, virDomainJobDataFree); + g_clear_pointer(&job->completed, virDomainJobDataFree); qemuMigrationParamsReset(driver, vm, asyncJob, jobPriv->migParams, - priv->job.apiFlags); + job->apiFlags); virPortAllocatorRelease(priv->migrationPort); priv->migrationPort = 0; @@ -6083,7 +6084,7 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, VIR_WARN("Unable to encode migration cookie"); qemuMigrationDstComplete(driver, vm, inPostCopy, - VIR_ASYNC_JOB_MIGRATION_IN); + VIR_ASYNC_JOB_MIGRATION_IN, &priv->job); return dom; diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index 1d6051859b..c099cf99cf 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -195,7 +195,8 @@ void qemuMigrationDstComplete(virQEMUDriver *driver, virDomainObj *vm, bool inPostCopy, - virDomainAsyncJob asyncJob); + virDomainAsyncJob asyncJob, + qemuDomainJobObj *job); int qemuMigrationSrcConfirm(virQEMUDriver *driver, -- 2.35.1
On Tue, May 10, 2022 at 17:21:00 +0200, Jiri Denemark wrote:
When reconnecting to an active domains we need to use a different job
s/domains/domain/
structure than the one referenced from the VM object.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 11 ++++++----- src/qemu/qemu_migration.h | 3 ++- 2 files changed, 8 insertions(+), 6 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
So far migration could only be completed while a migration API was running and waiting for the migration to finish. In case such API could not be called (the connection that initiated the migration is broken) the migration would just be aborted or left in a "don't know what to do" state. But this will change soon and we will be able to successfully complete such migration once we get the corresponding event from QEMU. This is specific to post-copy migration when vCPUs are already running on the destination and we're only waiting for all memory pages to be transferred. Such post-copy migration (which no-one is actively watching) is called unattended migration. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domain.c | 1 + src/qemu/qemu_domain.h | 1 + src/qemu/qemu_driver.c | 5 +++++ src/qemu/qemu_migration.c | 43 +++++++++++++++++++++++++++++++++++++-- src/qemu/qemu_migration.h | 6 ++++++ src/qemu/qemu_process.c | 18 ++++++++++++++++ 6 files changed, 72 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_domain.c b/src/qemu/qemu_domain.c index 7ad84f7080..00e1f0c94e 100644 --- a/src/qemu/qemu_domain.c +++ b/src/qemu/qemu_domain.c @@ -11074,6 +11074,7 @@ qemuProcessEventFree(struct qemuProcessEvent *event) qemuMonitorMemoryDeviceSizeChangeFree(event->data); break; case QEMU_PROCESS_EVENT_PR_DISCONNECT: + case QEMU_PROCESS_EVENT_UNATTENDED_MIGRATION: case QEMU_PROCESS_EVENT_LAST: break; } diff --git a/src/qemu/qemu_domain.h b/src/qemu/qemu_domain.h index 3ea0426c7e..94520ac97e 100644 --- a/src/qemu/qemu_domain.h +++ b/src/qemu/qemu_domain.h @@ -410,6 +410,7 @@ typedef enum { QEMU_PROCESS_EVENT_RDMA_GID_STATUS_CHANGED, QEMU_PROCESS_EVENT_GUEST_CRASHLOADED, QEMU_PROCESS_EVENT_MEMORY_DEVICE_SIZE_CHANGE, + QEMU_PROCESS_EVENT_UNATTENDED_MIGRATION, QEMU_PROCESS_EVENT_LAST } qemuProcessEventType; diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index d19115473a..e268d3a478 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -4279,6 +4279,11 @@ static void qemuProcessEventHandler(void *data, void *opaque) case QEMU_PROCESS_EVENT_MEMORY_DEVICE_SIZE_CHANGE: processMemoryDeviceSizeChange(driver, vm, processEvent->data); break; + case QEMU_PROCESS_EVENT_UNATTENDED_MIGRATION: + qemuMigrationProcessUnattended(driver, vm, + processEvent->action, + processEvent->status); + break; case QEMU_PROCESS_EVENT_LAST: break; } diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 6938e0fafc..dacea63610 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5842,8 +5842,11 @@ qemuMigrationDstComplete(virQEMUDriver *driver, qemuDomainSaveStatus(vm); - /* Guest is successfully running, so cancel previous auto destroy */ - qemuProcessAutoDestroyRemove(driver, vm); + /* Guest is successfully running, so cancel previous auto destroy. There's + * nothing to remove when we are resuming post-copy migration. + */ + if (!virDomainObjIsFailedPostcopy(vm)) + qemuProcessAutoDestroyRemove(driver, vm); /* Remove completed stats for post-copy, everything but timing fields * is obsolete anyway. @@ -6212,6 +6215,42 @@ qemuMigrationDstFinish(virQEMUDriver *driver, } +void +qemuMigrationProcessUnattended(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob job, + qemuMonitorMigrationStatus status) +{ + qemuDomainObjPrivate *priv = vm->privateData; + qemuMigrationJobPhase phase; + + if (!qemuMigrationJobIsActive(vm, job) || + status != QEMU_MONITOR_MIGRATION_STATUS_COMPLETED) + return; + + VIR_DEBUG("Unattended %s migration of domain %s successfully finished", + job == VIR_ASYNC_JOB_MIGRATION_IN ? "incoming" : "outgoing", + vm->def->name); + + if (job == VIR_ASYNC_JOB_MIGRATION_IN) + phase = QEMU_MIGRATION_PHASE_FINISH3; + else + phase = QEMU_MIGRATION_PHASE_CONFIRM3; + + qemuMigrationJobStartPhase(vm, phase); + + if (job == VIR_ASYNC_JOB_MIGRATION_IN) + qemuMigrationDstComplete(driver, vm, true, job, &priv->job); + else + qemuMigrationSrcComplete(driver, vm, job); + + qemuMigrationJobFinish(vm); + + if (!virDomainObjIsActive(vm)) + qemuDomainRemoveInactive(driver, vm); +} + + /* Helper function called while vm is active. */ int qemuMigrationSrcToFile(virQEMUDriver *driver, virDomainObj *vm, diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index c099cf99cf..eeb69a52bf 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -211,6 +211,12 @@ qemuMigrationSrcComplete(virQEMUDriver *driver, virDomainObj *vm, virDomainAsyncJob asyncJob); +void +qemuMigrationProcessUnattended(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob job, + qemuMonitorMigrationStatus status); + bool qemuMigrationSrcIsAllowed(virQEMUDriver *driver, virDomainObj *vm, diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index d92cd8cb5e..7b347a9061 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -1619,6 +1619,24 @@ qemuProcessHandleMigrationStatus(qemuMonitor *mon G_GNUC_UNUSED, qemuDomainSaveStatus(vm); } + /* A post-copy migration marked as failed when reconnecting to a domain + * with running migration may actually still be running, but we're not + * watching it in any thread. Let's make sure the migration is properly + * finished in case we get a "completed" event. + */ + if (virDomainObjIsFailedPostcopy(vm) && + priv->job.asyncOwner == 0 && + status == QEMU_MONITOR_MIGRATION_STATUS_COMPLETED) { + struct qemuProcessEvent *proc = g_new0(struct qemuProcessEvent, 1); + + proc->eventType = QEMU_PROCESS_EVENT_UNATTENDED_MIGRATION; + proc->action = priv->job.asyncJob; + proc->status = status; + proc->vm = virObjectRef(vm); + + qemuProcessEventSubmit(driver, &proc); + } + cleanup: virObjectUnlock(vm); virObjectEventStateQueue(driver->domainEventState, event); -- 2.35.1
On Tue, May 10, 2022 at 17:21:01 +0200, Jiri Denemark wrote:
So far migration could only be completed while a migration API was running and waiting for the migration to finish. In case such API could not be called (the connection that initiated the migration is broken) the migration would just be aborted or left in a "don't know what to do" state. But this will change soon and we will be able to successfully complete such migration once we get the corresponding event from QEMU. This is specific to post-copy migration when vCPUs are already running on the destination and we're only waiting for all memory pages to be transferred. Such post-copy migration (which no-one is actively watching) is called unattended migration.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domain.c | 1 + src/qemu/qemu_domain.h | 1 + src/qemu/qemu_driver.c | 5 +++++ src/qemu/qemu_migration.c | 43 +++++++++++++++++++++++++++++++++++++-- src/qemu/qemu_migration.h | 6 ++++++ src/qemu/qemu_process.c | 18 ++++++++++++++++ 6 files changed, 72 insertions(+), 2 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
We want to use query-migrate QMP command to check the current migration state when reconnecting to active domains, but the reply we get to this command may not contain any statistics at all if called on the destination host. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_monitor_json.c | 88 +++++++++++++++++------------------- 1 file changed, 42 insertions(+), 46 deletions(-) diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 3a497dceae..df3fdfe4fb 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3249,56 +3249,52 @@ qemuMonitorJSONGetMigrationStatsReply(virJSONValue *reply, case QEMU_MONITOR_MIGRATION_STATUS_PRE_SWITCHOVER: case QEMU_MONITOR_MIGRATION_STATUS_DEVICE: ram = virJSONValueObjectGetObject(ret, "ram"); - if (!ram) { - virReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("migration was active, but no RAM info was set")); - return -1; - } + if (ram) { + if (virJSONValueObjectGetNumberUlong(ram, "transferred", + &stats->ram_transferred) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("migration was active, but RAM 'transferred' " + "data was missing")); + return -1; + } + if (virJSONValueObjectGetNumberUlong(ram, "remaining", + &stats->ram_remaining) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("migration was active, but RAM 'remaining' " + "data was missing")); + return -1; + } + if (virJSONValueObjectGetNumberUlong(ram, "total", + &stats->ram_total) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("migration was active, but RAM 'total' " + "data was missing")); + return -1; + } - if (virJSONValueObjectGetNumberUlong(ram, "transferred", - &stats->ram_transferred) < 0) { - virReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("migration was active, but RAM 'transferred' " - "data was missing")); - return -1; - } - if (virJSONValueObjectGetNumberUlong(ram, "remaining", - &stats->ram_remaining) < 0) { - virReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("migration was active, but RAM 'remaining' " - "data was missing")); - return -1; - } - if (virJSONValueObjectGetNumberUlong(ram, "total", - &stats->ram_total) < 0) { - virReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("migration was active, but RAM 'total' " - "data was missing")); - return -1; - } + if (virJSONValueObjectGetNumberDouble(ram, "mbps", &mbps) == 0 && + mbps > 0) { + /* mpbs from QEMU reports Mbits/s (M as in 10^6 not Mi as 2^20) */ + stats->ram_bps = mbps * (1000 * 1000 / 8); + } - if (virJSONValueObjectGetNumberDouble(ram, "mbps", &mbps) == 0 && - mbps > 0) { - /* mpbs from QEMU reports Mbits/s (M as in 10^6 not Mi as 2^20) */ - stats->ram_bps = mbps * (1000 * 1000 / 8); + if (virJSONValueObjectGetNumberUlong(ram, "duplicate", + &stats->ram_duplicate) == 0) + stats->ram_duplicate_set = true; + ignore_value(virJSONValueObjectGetNumberUlong(ram, "normal", + &stats->ram_normal)); + ignore_value(virJSONValueObjectGetNumberUlong(ram, "normal-bytes", + &stats->ram_normal_bytes)); + ignore_value(virJSONValueObjectGetNumberUlong(ram, "dirty-pages-rate", + &stats->ram_dirty_rate)); + ignore_value(virJSONValueObjectGetNumberUlong(ram, "page-size", + &stats->ram_page_size)); + ignore_value(virJSONValueObjectGetNumberUlong(ram, "dirty-sync-count", + &stats->ram_iteration)); + ignore_value(virJSONValueObjectGetNumberUlong(ram, "postcopy-requests", + &stats->ram_postcopy_reqs)); } - if (virJSONValueObjectGetNumberUlong(ram, "duplicate", - &stats->ram_duplicate) == 0) - stats->ram_duplicate_set = true; - ignore_value(virJSONValueObjectGetNumberUlong(ram, "normal", - &stats->ram_normal)); - ignore_value(virJSONValueObjectGetNumberUlong(ram, "normal-bytes", - &stats->ram_normal_bytes)); - ignore_value(virJSONValueObjectGetNumberUlong(ram, "dirty-pages-rate", - &stats->ram_dirty_rate)); - ignore_value(virJSONValueObjectGetNumberUlong(ram, "page-size", - &stats->ram_page_size)); - ignore_value(virJSONValueObjectGetNumberUlong(ram, "dirty-sync-count", - &stats->ram_iteration)); - ignore_value(virJSONValueObjectGetNumberUlong(ram, "postcopy-requests", - &stats->ram_postcopy_reqs)); - disk = virJSONValueObjectGetObject(ret, "disk"); if (disk) { rc = virJSONValueObjectGetNumberUlong(disk, "transferred", -- 2.35.1
On Tue, May 10, 2022 at 17:21:02 +0200, Jiri Denemark wrote:
We want to use query-migrate QMP command to check the current migration state when reconnecting to active domains, but the reply we get to this command may not contain any statistics at all if called on the destination host.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_monitor_json.c | 88 +++++++++++++++++------------------- 1 file changed, 42 insertions(+), 46 deletions(-)
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 3a497dceae..df3fdfe4fb 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3249,56 +3249,52 @@ qemuMonitorJSONGetMigrationStatsReply(virJSONValue *reply, case QEMU_MONITOR_MIGRATION_STATUS_PRE_SWITCHOVER: case QEMU_MONITOR_MIGRATION_STATUS_DEVICE: ram = virJSONValueObjectGetObject(ret, "ram"); - if (!ram) { - virReportError(VIR_ERR_INTERNAL_ERROR, "%s", - _("migration was active, but no RAM info was set")); - return -1; - } + if (ram) { + if (virJSONValueObjectGetNumberUlong(ram, "transferred", + &stats->ram_transferred) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("migration was active, but RAM 'transferred' " + "data was missing")); + return -1; + } + if (virJSONValueObjectGetNumberUlong(ram, "remaining", + &stats->ram_remaining) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("migration was active, but RAM 'remaining' " + "data was missing")); + return -1; + } + if (virJSONValueObjectGetNumberUlong(ram, "total", + &stats->ram_total) < 0) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("migration was active, but RAM 'total' " + "data was missing")); + return -1; + }
Please fix error messages to conform with coding style since you are touching them. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
When libvirt daemon is restarted during an active post-copy migration, we do not always mark the migration as broken. In this phase libvirt is not really needed for migration to finish successfully. In fact the migration could have even finished while libvirt was not running or it may still be happily running. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 27 +++++++++++++++++++++++++++ src/qemu/qemu_migration.h | 6 ++++++ src/qemu/qemu_process.c | 39 +++++++++++++++++++++++++++++---------- 3 files changed, 62 insertions(+), 10 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index dacea63610..854dfd43c1 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2460,6 +2460,33 @@ qemuMigrationSrcBeginPhaseBlockDirtyBitmaps(qemuMigrationCookie *mig, } +int +qemuMigrationAnyRefreshStatus(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob asyncJob, + virDomainJobStatus *status) +{ + g_autoptr(virDomainJobData) jobData = NULL; + qemuDomainJobDataPrivate *priv; + + jobData = virDomainJobDataInit(&qemuJobDataPrivateDataCallbacks); + priv = jobData->privateData; + + if (qemuMigrationAnyFetchStats(driver, vm, asyncJob, jobData, NULL) < 0) + return -1; + + qemuMigrationUpdateJobType(jobData); + VIR_DEBUG("QEMU reports domain '%s' is in '%s' migration state, " + "translated as %d", + vm->def->name, + qemuMonitorMigrationStatusTypeToString(priv->stats.mig.status), + jobData->status); + + *status = jobData->status; + return 0; +} + + /* The caller is supposed to lock the vm and start a migration job. */ static char * qemuMigrationSrcBeginPhase(virQEMUDriver *driver, diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index eeb69a52bf..9351d6ac51 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -279,3 +279,9 @@ qemuMigrationSrcFetchMirrorStats(virQEMUDriver *driver, virDomainObj *vm, virDomainAsyncJob asyncJob, virDomainJobData *jobData); + +int +qemuMigrationAnyRefreshStatus(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob asyncJob, + virDomainJobStatus *status); diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 7b347a9061..1cb00af6f1 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3591,10 +3591,8 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, /* migration finished, we started resuming the domain but didn't * confirm success or failure yet; killing it seems safest unless * we already started guest CPUs or we were in post-copy mode */ - if (virDomainObjIsPostcopy(vm, VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN)) { - qemuMigrationDstPostcopyFailed(vm); + if (virDomainObjIsPostcopy(vm, VIR_DOMAIN_JOB_OPERATION_MIGRATION_IN)) return 1; - } if (state != VIR_DOMAIN_RUNNING) { VIR_DEBUG("Killing migrated domain %s", vm->def->name); @@ -3661,10 +3659,8 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, * of Finish3 step; third party needs to check what to do next; in * post-copy mode we can use PAUSED_POSTCOPY_FAILED state for this */ - if (postcopy) { - qemuMigrationSrcPostcopyFailed(vm); + if (postcopy) return 1; - } break; case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: @@ -3672,10 +3668,8 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, * post-copy mode there's no way back, so let's just mark the domain * as broken in that case */ - if (postcopy) { - qemuMigrationSrcPostcopyFailed(vm); + if (postcopy) return 1; - } VIR_DEBUG("Resuming domain %s after failed migration", vm->def->name); @@ -3713,6 +3707,7 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, qemuDomainJobObj *job, unsigned int *stopFlags) { + virDomainJobStatus migStatus = VIR_DOMAIN_JOB_STATUS_NONE; qemuDomainJobPrivate *jobPriv = job->privateData; virDomainState state; int reason; @@ -3720,6 +3715,8 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, state = virDomainObjGetState(vm, &reason); + qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_NONE, &migStatus); + if (job->asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) { rc = qemuProcessRecoverMigrationOut(driver, vm, job, state, reason, stopFlags); @@ -3731,7 +3728,29 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, return -1; if (rc > 0) { - qemuProcessRestoreMigrationJob(vm, job); + if (migStatus == VIR_DOMAIN_JOB_STATUS_POSTCOPY) { + VIR_DEBUG("Post-copy migration of domain %s still running, it " + "will be handled as unattended", vm->def->name); + qemuProcessRestoreMigrationJob(vm, job); + return 0; + } + + if (migStatus != VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED) { + if (job->asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) + qemuMigrationSrcPostcopyFailed(vm); + else + qemuMigrationDstPostcopyFailed(vm); + + qemuProcessRestoreMigrationJob(vm, job); + return 0; + } + + VIR_DEBUG("Post-copy migration of domain %s already finished", + vm->def->name); + if (job->asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) + qemuMigrationSrcComplete(driver, vm, VIR_ASYNC_JOB_NONE); + else + qemuMigrationDstComplete(driver, vm, true, VIR_ASYNC_JOB_NONE, job); return 0; } -- 2.35.1
On Tue, May 10, 2022 at 17:21:03 +0200, Jiri Denemark wrote:
When libvirt daemon is restarted during an active post-copy migration, we do not always mark the migration as broken. In this phase libvirt is not really needed for migration to finish successfully. In fact the migration could have even finished while libvirt was not running or it may still be happily running.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 27 +++++++++++++++++++++++++++ src/qemu/qemu_migration.h | 6 ++++++ src/qemu/qemu_process.c | 39 +++++++++++++++++++++++++++++---------- 3 files changed, 62 insertions(+), 10 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index dacea63610..854dfd43c1 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2460,6 +2460,33 @@ qemuMigrationSrcBeginPhaseBlockDirtyBitmaps(qemuMigrationCookie *mig, }
+int +qemuMigrationAnyRefreshStatus(virQEMUDriver *driver, + virDomainObj *vm, + virDomainAsyncJob asyncJob, + virDomainJobStatus *status) +{ + g_autoptr(virDomainJobData) jobData = NULL; + qemuDomainJobDataPrivate *priv; + + jobData = virDomainJobDataInit(&qemuJobDataPrivateDataCallbacks); + priv = jobData->privateData; + + if (qemuMigrationAnyFetchStats(driver, vm, asyncJob, jobData, NULL) < 0) + return -1; + + qemuMigrationUpdateJobType(jobData); + VIR_DEBUG("QEMU reports domain '%s' is in '%s' migration state, " + "translated as %d",
Diagnostics on a single line please
+ vm->def->name, + qemuMonitorMigrationStatusTypeToString(priv->stats.mig.status), + jobData->status); + + *status = jobData->status; + return 0; +} +
[...]
@@ -3731,7 +3728,29 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, return -1;
if (rc > 0) { - qemuProcessRestoreMigrationJob(vm, job); + if (migStatus == VIR_DOMAIN_JOB_STATUS_POSTCOPY) { + VIR_DEBUG("Post-copy migration of domain %s still running, it " + "will be handled as unattended", vm->def->name);
This one too. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 44 +++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 22 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 854dfd43c1..180a760bca 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2541,6 +2541,28 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, return NULL; } + if (flags & VIR_MIGRATE_OFFLINE) { + if (flags & (VIR_MIGRATE_NON_SHARED_DISK | + VIR_MIGRATE_NON_SHARED_INC)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration cannot handle " + "non-shared storage")); + return NULL; + } + if (!(flags & VIR_MIGRATE_PERSIST_DEST)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration must be specified with " + "the persistent flag set")); + return NULL; + } + if (flags & VIR_MIGRATE_TUNNELLED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("tunnelled offline migration does not " + "make sense")); + return NULL; + } + } + if (flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC)) { if (flags & VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES && !virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_BLOCKDEV)) { @@ -2620,28 +2642,6 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, cookieFlags) < 0) return NULL; - if (flags & VIR_MIGRATE_OFFLINE) { - if (flags & (VIR_MIGRATE_NON_SHARED_DISK | - VIR_MIGRATE_NON_SHARED_INC)) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("offline migration cannot handle " - "non-shared storage")); - return NULL; - } - if (!(flags & VIR_MIGRATE_PERSIST_DEST)) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("offline migration must be specified with " - "the persistent flag set")); - return NULL; - } - if (flags & VIR_MIGRATE_TUNNELLED) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("tunnelled offline migration does not " - "make sense")); - return NULL; - } - } - if (xmlin) { if (!(def = virDomainDefParseString(xmlin, driver->xmlopt, priv->qemuCaps, VIR_DOMAIN_DEF_PARSE_INACTIVE | -- 2.35.1
On Tue, May 10, 2022 at 17:21:04 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 44 +++++++++++++++++++-------------------- 1 file changed, 22 insertions(+), 22 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 854dfd43c1..180a760bca 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2541,6 +2541,28 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, return NULL; }
+ if (flags & VIR_MIGRATE_OFFLINE) { + if (flags & (VIR_MIGRATE_NON_SHARED_DISK | + VIR_MIGRATE_NON_SHARED_INC)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration cannot handle " + "non-shared storage")); + return NULL; + } + if (!(flags & VIR_MIGRATE_PERSIST_DEST)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration must be specified with " + "the persistent flag set")); + return NULL; + } + if (flags & VIR_MIGRATE_TUNNELLED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("tunnelled offline migration does not " + "make sense")); + return NULL; + } + }
I'm aware that you are doing pure code movement, but remove the linebreaks in the error messages. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Turn the final part of Begin phase formatting a domain XML for migration into a reusable helper. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 95 ++++++++++++++++++++++++--------------- 1 file changed, 58 insertions(+), 37 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 180a760bca..7299bb6a0b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2487,6 +2487,60 @@ qemuMigrationAnyRefreshStatus(virQEMUDriver *driver, } +static char * +qemuMigrationSrcBeginXML(virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned int cookieFlags, + const char **migrate_disks, + size_t nmigrate_disks, + unsigned long flags) +{ + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(qemuMigrationCookie) mig = NULL; + + if (priv->origCPU) + cookieFlags |= QEMU_MIGRATION_COOKIE_CPU; + + if (!(flags & VIR_MIGRATE_OFFLINE)) + cookieFlags |= QEMU_MIGRATION_COOKIE_CAPS; + + if (!(mig = qemuMigrationCookieNew(vm->def, priv->origname))) + return NULL; + + if (cookieFlags & QEMU_MIGRATION_COOKIE_NBD && + virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_PARAM_BLOCK_BITMAP_MAPPING) && + qemuMigrationSrcBeginPhaseBlockDirtyBitmaps(mig, vm, migrate_disks, + nmigrate_disks) < 0) + return NULL; + + if (qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_SOURCE, + cookieout, cookieoutlen, + cookieFlags) < 0) + return NULL; + + if (xmlin) { + g_autoptr(virDomainDef) def = NULL; + + if (!(def = virDomainDefParseString(xmlin, driver->xmlopt, priv->qemuCaps, + VIR_DOMAIN_DEF_PARSE_INACTIVE | + VIR_DOMAIN_DEF_PARSE_SKIP_VALIDATE))) + return NULL; + + if (!qemuDomainCheckABIStability(driver, vm, def)) + return NULL; + + return qemuDomainDefFormatLive(driver, priv->qemuCaps, def, NULL, false, true); + } + + return qemuDomainDefFormatLive(driver, priv->qemuCaps, vm->def, priv->origCPU, + false, true); +} + + /* The caller is supposed to lock the vm and start a migration job. */ static char * qemuMigrationSrcBeginPhase(virQEMUDriver *driver, @@ -2499,8 +2553,6 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, const char **migrate_disks, unsigned long flags) { - g_autoptr(qemuMigrationCookie) mig = NULL; - g_autoptr(virDomainDef) def = NULL; qemuDomainObjPrivate *priv = vm->privateData; unsigned int cookieFlags = QEMU_MIGRATION_COOKIE_LOCKSTATE; @@ -2621,41 +2673,10 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, vm->newDef && !qemuDomainVcpuHotplugIsInOrder(vm->newDef))) cookieFlags |= QEMU_MIGRATION_COOKIE_CPU_HOTPLUG; - if (priv->origCPU) - cookieFlags |= QEMU_MIGRATION_COOKIE_CPU; - - if (!(flags & VIR_MIGRATE_OFFLINE)) - cookieFlags |= QEMU_MIGRATION_COOKIE_CAPS; - - if (!(mig = qemuMigrationCookieNew(vm->def, priv->origname))) - return NULL; - - if (cookieFlags & QEMU_MIGRATION_COOKIE_NBD && - virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_MIGRATION_PARAM_BLOCK_BITMAP_MAPPING) && - qemuMigrationSrcBeginPhaseBlockDirtyBitmaps(mig, vm, migrate_disks, - nmigrate_disks) < 0) - return NULL; - - if (qemuMigrationCookieFormat(mig, driver, vm, - QEMU_MIGRATION_SOURCE, - cookieout, cookieoutlen, - cookieFlags) < 0) - return NULL; - - if (xmlin) { - if (!(def = virDomainDefParseString(xmlin, driver->xmlopt, priv->qemuCaps, - VIR_DOMAIN_DEF_PARSE_INACTIVE | - VIR_DOMAIN_DEF_PARSE_SKIP_VALIDATE))) - return NULL; - - if (!qemuDomainCheckABIStability(driver, vm, def)) - return NULL; - - return qemuDomainDefFormatLive(driver, priv->qemuCaps, def, NULL, false, true); - } else { - return qemuDomainDefFormatLive(driver, priv->qemuCaps, vm->def, priv->origCPU, - false, true); - } + return qemuMigrationSrcBeginXML(driver, vm, xmlin, + cookieout, cookieoutlen, cookieFlags, + migrate_disks, nmigrate_disks, + flags); } char * -- 2.35.1
On Tue, May 10, 2022 at 17:21:05 +0200, Jiri Denemark wrote:
Turn the final part of Begin phase formatting a domain XML for migration into a reusable helper.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 95 ++++++++++++++++++++++++--------------- 1 file changed, 58 insertions(+), 37 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 180a760bca..7299bb6a0b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2487,6 +2487,60 @@ qemuMigrationAnyRefreshStatus(virQEMUDriver *driver, }
+static char * +qemuMigrationSrcBeginXML(virQEMUDriver *driver,
The name of this function is a bit too generic IMO, and doesn't imply that it's actually formatting the XML. Also note that 'driver' can be fetched from 'priv->driver' since you are refactoring the code.
+ virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned int cookieFlags, + const char **migrate_disks, + size_t nmigrate_disks, + unsigned long flags)
Preferrably remove the 'driver' argument. I can live with the name. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
When recovering from a failed post-copy migration, we need to go through all migration phases again, but don't need to repeat all the steps in each phase. Let's create a new set of migration phases dedicated to post-copy recovery so that we can easily distinguish between normal and recovery code. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 20 +++++++++++++++++++- src/qemu/qemu_migration.h | 6 ++++++ src/qemu/qemu_process.c | 25 ++++++++++++++++++++++++- 3 files changed, 49 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 7299bb6a0b..301d9db1d2 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -79,6 +79,12 @@ VIR_ENUM_IMPL(qemuMigrationJobPhase, "prepare", "finish2", "finish3", + "postcopy_failed", + "begin_resume", + "perform_resume", + "confirm_resume", + "prepare_resume", + "finish_resume", ); @@ -139,7 +145,8 @@ qemuMigrationJobSetPhase(virDomainObj *vm, { qemuDomainObjPrivate *priv = vm->privateData; - if (phase < priv->job.phase) { + if (phase < QEMU_MIGRATION_PHASE_POSTCOPY_FAILED && + phase < priv->job.phase) { VIR_ERROR(_("migration protocol going backwards %s => %s"), qemuMigrationJobPhaseTypeToString(priv->job.phase), qemuMigrationJobPhaseTypeToString(phase)); @@ -2356,18 +2363,29 @@ qemuMigrationSrcCleanup(virDomainObj *vm, } break; + case QEMU_MIGRATION_PHASE_BEGIN_RESUME: + case QEMU_MIGRATION_PHASE_PERFORM_RESUME: + qemuMigrationSrcPostcopyFailed(vm); + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + break; + case QEMU_MIGRATION_PHASE_PERFORM3: /* cannot be seen without an active migration API; unreachable */ case QEMU_MIGRATION_PHASE_CONFIRM3: case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: + case QEMU_MIGRATION_PHASE_CONFIRM_RESUME: /* all done; unreachable */ case QEMU_MIGRATION_PHASE_PREPARE: case QEMU_MIGRATION_PHASE_FINISH2: case QEMU_MIGRATION_PHASE_FINISH3: + case QEMU_MIGRATION_PHASE_PREPARE_RESUME: + case QEMU_MIGRATION_PHASE_FINISH_RESUME: /* incoming migration; unreachable */ case QEMU_MIGRATION_PHASE_PERFORM2: /* single phase outgoing migration; unreachable */ case QEMU_MIGRATION_PHASE_NONE: + case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_LAST: /* unreachable */ ; diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index 9351d6ac51..7eb0d4fe02 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -100,6 +100,12 @@ typedef enum { QEMU_MIGRATION_PHASE_PREPARE, QEMU_MIGRATION_PHASE_FINISH2, QEMU_MIGRATION_PHASE_FINISH3, + QEMU_MIGRATION_PHASE_POSTCOPY_FAILED, /* marker for resume phases */ + QEMU_MIGRATION_PHASE_BEGIN_RESUME, + QEMU_MIGRATION_PHASE_PERFORM_RESUME, + QEMU_MIGRATION_PHASE_CONFIRM_RESUME, + QEMU_MIGRATION_PHASE_PREPARE_RESUME, + QEMU_MIGRATION_PHASE_FINISH_RESUME, QEMU_MIGRATION_PHASE_LAST } qemuMigrationJobPhase; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index 1cb00af6f1..c7ed0a5c56 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -3566,6 +3566,10 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, case QEMU_MIGRATION_PHASE_PERFORM3_DONE: case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: case QEMU_MIGRATION_PHASE_CONFIRM3: + case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: + case QEMU_MIGRATION_PHASE_BEGIN_RESUME: + case QEMU_MIGRATION_PHASE_PERFORM_RESUME: + case QEMU_MIGRATION_PHASE_CONFIRM_RESUME: case QEMU_MIGRATION_PHASE_LAST: /* N/A for incoming migration */ break; @@ -3599,6 +3603,10 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, return -1; } break; + + case QEMU_MIGRATION_PHASE_PREPARE_RESUME: + case QEMU_MIGRATION_PHASE_FINISH_RESUME: + return 1; } return 0; @@ -3615,6 +3623,7 @@ static int qemuProcessRecoverMigrationOut(virQEMUDriver *driver, virDomainObj *vm, qemuDomainJobObj *job, + virDomainJobStatus migStatus, virDomainState state, int reason, unsigned int *stopFlags) @@ -3630,6 +3639,9 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, case QEMU_MIGRATION_PHASE_PREPARE: case QEMU_MIGRATION_PHASE_FINISH2: case QEMU_MIGRATION_PHASE_FINISH3: + case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: + case QEMU_MIGRATION_PHASE_PREPARE_RESUME: + case QEMU_MIGRATION_PHASE_FINISH_RESUME: case QEMU_MIGRATION_PHASE_LAST: /* N/A for outgoing migration */ break; @@ -3680,6 +3692,17 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, /* migration completed, we need to kill the domain here */ *stopFlags |= VIR_QEMU_PROCESS_STOP_MIGRATED; return -1; + + case QEMU_MIGRATION_PHASE_CONFIRM_RESUME: + if (migStatus == VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED) { + *stopFlags |= VIR_QEMU_PROCESS_STOP_MIGRATED; + return -1; + } + return 1; + + case QEMU_MIGRATION_PHASE_BEGIN_RESUME: + case QEMU_MIGRATION_PHASE_PERFORM_RESUME: + return 1; } if (resume) { @@ -3718,7 +3741,7 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_NONE, &migStatus); if (job->asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) { - rc = qemuProcessRecoverMigrationOut(driver, vm, job, + rc = qemuProcessRecoverMigrationOut(driver, vm, job, migStatus, state, reason, stopFlags); } else { rc = qemuProcessRecoverMigrationIn(driver, vm, job, state); -- 2.35.1
On Tue, May 10, 2022 at 17:21:06 +0200, Jiri Denemark wrote:
When recovering from a failed post-copy migration, we need to go through all migration phases again, but don't need to repeat all the steps in each phase. Let's create a new set of migration phases dedicated to post-copy recovery so that we can easily distinguish between normal and recovery code.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 20 +++++++++++++++++++- src/qemu/qemu_migration.h | 6 ++++++ src/qemu/qemu_process.c | 25 ++++++++++++++++++++++++- 3 files changed, 49 insertions(+), 2 deletions(-)
[...]
@@ -3680,6 +3692,17 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, /* migration completed, we need to kill the domain here */ *stopFlags |= VIR_QEMU_PROCESS_STOP_MIGRATED; return -1; + + case QEMU_MIGRATION_PHASE_CONFIRM_RESUME: + if (migStatus == VIR_DOMAIN_JOB_STATUS_HYPERVISOR_COMPLETED) { + *stopFlags |= VIR_QEMU_PROCESS_STOP_MIGRATED;
I'd consider this case to be success ...
+ return -1;
... so this technically violates the return values as you've previously declared that: * -1 on error, the domain will be killed, Please add a comment that makes the reader aware that this is success actually or modify the comment at the top of the function.
+ } + return 1; +
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Into a new qemuMigrationCheckPhase helper, which can be reused in other places. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 301d9db1d2..5b6073b963 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -139,9 +139,9 @@ qemuMigrationJobStart(virQEMUDriver *driver, } -static void ATTRIBUTE_NONNULL(1) -qemuMigrationJobSetPhase(virDomainObj *vm, - qemuMigrationJobPhase phase) +static int +qemuMigrationCheckPhase(virDomainObj *vm, + qemuMigrationJobPhase phase) { qemuDomainObjPrivate *priv = vm->privateData; @@ -150,9 +150,20 @@ qemuMigrationJobSetPhase(virDomainObj *vm, VIR_ERROR(_("migration protocol going backwards %s => %s"), qemuMigrationJobPhaseTypeToString(priv->job.phase), qemuMigrationJobPhaseTypeToString(phase)); - return; + return -1; } + return 0; +} + + +static void ATTRIBUTE_NONNULL(1) +qemuMigrationJobSetPhase(virDomainObj *vm, + qemuMigrationJobPhase phase) +{ + if (qemuMigrationCheckPhase(vm, phase) < 0) + return; + qemuDomainObjSetJobPhase(vm, phase); } -- 2.35.1
On Tue, May 10, 2022 at 17:21:07 +0200, Jiri Denemark wrote:
Into a new qemuMigrationCheckPhase helper, which can be reused in other places.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The check can reveal a serious bug in our migration code and we should not silently ignore it. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 58 ++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 22 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 5b6073b963..1c5dd9b391 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -147,9 +147,10 @@ qemuMigrationCheckPhase(virDomainObj *vm, if (phase < QEMU_MIGRATION_PHASE_POSTCOPY_FAILED && phase < priv->job.phase) { - VIR_ERROR(_("migration protocol going backwards %s => %s"), - qemuMigrationJobPhaseTypeToString(priv->job.phase), - qemuMigrationJobPhaseTypeToString(phase)); + virReportError(VIR_ERR_INTERNAL_ERROR, + _("migration protocol going backwards %s => %s"), + qemuMigrationJobPhaseTypeToString(priv->job.phase), + qemuMigrationJobPhaseTypeToString(phase)); return -1; } @@ -157,22 +158,23 @@ qemuMigrationCheckPhase(virDomainObj *vm, } -static void ATTRIBUTE_NONNULL(1) +static int G_GNUC_WARN_UNUSED_RESULT qemuMigrationJobSetPhase(virDomainObj *vm, qemuMigrationJobPhase phase) { if (qemuMigrationCheckPhase(vm, phase) < 0) - return; + return -1; qemuDomainObjSetJobPhase(vm, phase); + return 0; } -static void ATTRIBUTE_NONNULL(1) +static int G_GNUC_WARN_UNUSED_RESULT qemuMigrationJobStartPhase(virDomainObj *vm, qemuMigrationJobPhase phase) { - qemuMigrationJobSetPhase(vm, phase); + return qemuMigrationJobSetPhase(vm, phase); } @@ -2596,8 +2598,9 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, * Otherwise we will start the async job later in the perform phase losing * change protection. */ - if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_BEGIN3); + if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT && + qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_BEGIN3) < 0) + return NULL; if (!qemuMigrationSrcIsAllowed(driver, vm, true, flags)) return NULL; @@ -3148,7 +3151,9 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, if (qemuMigrationJobStart(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, flags) < 0) goto cleanup; - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PREPARE); + + if (qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PREPARE) < 0) + goto stopjob; /* Domain starts inactive, even if the domain XML had an id field. */ vm->def->id = -1; @@ -3668,7 +3673,8 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, else phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; - qemuMigrationJobSetPhase(vm, phase); + if (qemuMigrationJobSetPhase(vm, phase) < 0) + return -1; if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, cookiein, cookieinlen, @@ -3753,7 +3759,9 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, else phase = QEMU_MIGRATION_PHASE_CONFIRM3; - qemuMigrationJobStartPhase(vm, phase); + if (qemuMigrationJobStartPhase(vm, phase) < 0) + goto cleanup; + virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup); @@ -4920,7 +4928,7 @@ qemuMigrationSrcPerformPeer2Peer2(virQEMUDriver *driver, * until the migration is complete. */ VIR_DEBUG("Perform %p", sconn); - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2)); if (flags & VIR_MIGRATE_TUNNELLED) ret = qemuMigrationSrcPerformTunnel(driver, vm, st, NULL, NULL, 0, NULL, NULL, @@ -5164,7 +5172,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, * confirm migration completion. */ VIR_DEBUG("Perform3 %p uri=%s", sconn, NULLSTR(uri)); - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3)); VIR_FREE(cookiein); cookiein = g_steal_pointer(&cookieout); cookieinlen = cookieoutlen; @@ -5189,7 +5197,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, if (ret < 0) { virErrorPreserveLast(&orig_err); } else { - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); } /* If Perform returns < 0, then we need to cancel the VM @@ -5565,7 +5573,9 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, migParams, flags, dname, resource, &v3proto); } else { - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2); + if (qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2) < 0) + goto endjob; + ret = qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, cookieout, cookieoutlen, flags, resource, NULL, NULL, 0, NULL, @@ -5657,7 +5667,9 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, return ret; } - qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3); + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3) < 0) + goto endjob; + virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup); @@ -5671,7 +5683,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto endjob; } - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, qemuMigrationSrcCleanup) < 0) @@ -6238,9 +6250,10 @@ qemuMigrationDstFinish(virQEMUDriver *driver, ignore_value(virTimeMillisNow(&timeReceived)); - qemuMigrationJobStartPhase(vm, - v3proto ? QEMU_MIGRATION_PHASE_FINISH3 - : QEMU_MIGRATION_PHASE_FINISH2); + if (qemuMigrationJobStartPhase(vm, + v3proto ? QEMU_MIGRATION_PHASE_FINISH3 + : QEMU_MIGRATION_PHASE_FINISH2) < 0) + goto cleanup; qemuDomainCleanupRemove(vm, qemuMigrationDstPrepareCleanup); g_clear_pointer(&priv->job.completed, virDomainJobDataFree); @@ -6314,7 +6327,8 @@ qemuMigrationProcessUnattended(virQEMUDriver *driver, else phase = QEMU_MIGRATION_PHASE_CONFIRM3; - qemuMigrationJobStartPhase(vm, phase); + if (qemuMigrationJobStartPhase(vm, phase) < 0) + return; if (job == VIR_ASYNC_JOB_MIGRATION_IN) qemuMigrationDstComplete(driver, vm, true, job, &priv->job); -- 2.35.1
On Tue, May 10, 2022 at 17:21:08 +0200, Jiri Denemark wrote:
The check can reveal a serious bug in our migration code and we should not silently ignore it.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 58 ++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 22 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 5b6073b963..1c5dd9b391 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -147,9 +147,10 @@ qemuMigrationCheckPhase(virDomainObj *vm,
if (phase < QEMU_MIGRATION_PHASE_POSTCOPY_FAILED && phase < priv->job.phase) { - VIR_ERROR(_("migration protocol going backwards %s => %s"), - qemuMigrationJobPhaseTypeToString(priv->job.phase), - qemuMigrationJobPhaseTypeToString(phase)); + virReportError(VIR_ERR_INTERNAL_ERROR, + _("migration protocol going backwards %s => %s"), + qemuMigrationJobPhaseTypeToString(priv->job.phase), + qemuMigrationJobPhaseTypeToString(phase));
This bit seems to belongs to the previous commit actually, but since it's not used anywhere else ...
return -1; }
[...]
@@ -4920,7 +4928,7 @@ qemuMigrationSrcPerformPeer2Peer2(virQEMUDriver *driver, * until the migration is complete. */ VIR_DEBUG("Perform %p", sconn); - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2)); if (flags & VIR_MIGRATE_TUNNELLED) ret = qemuMigrationSrcPerformTunnel(driver, vm, st, NULL, NULL, 0, NULL, NULL, @@ -5164,7 +5172,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, * confirm migration completion. */ VIR_DEBUG("Perform3 %p uri=%s", sconn, NULLSTR(uri)); - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3)); VIR_FREE(cookiein); cookiein = g_steal_pointer(&cookieout); cookieinlen = cookieoutlen;
Any reason why you want to ignore this before the migration was performed?
@@ -5189,7 +5197,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, if (ret < 0) { virErrorPreserveLast(&orig_err); } else { - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); }
/* If Perform returns < 0, then we need to cancel the VM
I could somehwat understand it here after the migration is done, but a bug could be also in this code.
@@ -5657,7 +5667,9 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, return ret; }
- qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3); + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3) < 0) + goto endjob; + virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup);
@@ -5671,7 +5683,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto endjob; }
- qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE));
if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, qemuMigrationSrcCleanup) < 0)
Same here.
On Thu, May 12, 2022 at 10:39:50 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:08 +0200, Jiri Denemark wrote:
The check can reveal a serious bug in our migration code and we should not silently ignore it.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 58 ++++++++++++++++++++++++--------------- 1 file changed, 36 insertions(+), 22 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 5b6073b963..1c5dd9b391 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -147,9 +147,10 @@ qemuMigrationCheckPhase(virDomainObj *vm,
if (phase < QEMU_MIGRATION_PHASE_POSTCOPY_FAILED && phase < priv->job.phase) { - VIR_ERROR(_("migration protocol going backwards %s => %s"), - qemuMigrationJobPhaseTypeToString(priv->job.phase), - qemuMigrationJobPhaseTypeToString(phase)); + virReportError(VIR_ERR_INTERNAL_ERROR, + _("migration protocol going backwards %s => %s"), + qemuMigrationJobPhaseTypeToString(priv->job.phase), + qemuMigrationJobPhaseTypeToString(phase));
This bit seems to belongs to the previous commit actually, but since it's not used anywhere else ...
It is intentionally here as the previous commit is just a code movement with no functional change.
return -1; }
[...]
@@ -4920,7 +4928,7 @@ qemuMigrationSrcPerformPeer2Peer2(virQEMUDriver *driver, * until the migration is complete. */ VIR_DEBUG("Perform %p", sconn); - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2)); if (flags & VIR_MIGRATE_TUNNELLED) ret = qemuMigrationSrcPerformTunnel(driver, vm, st, NULL, NULL, 0, NULL, NULL, @@ -5164,7 +5172,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, * confirm migration completion. */ VIR_DEBUG("Perform3 %p uri=%s", sconn, NULLSTR(uri)); - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3)); VIR_FREE(cookiein); cookiein = g_steal_pointer(&cookieout); cookieinlen = cookieoutlen;
Any reason why you want to ignore this before the migration was performed?
@@ -5189,7 +5197,7 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, if (ret < 0) { virErrorPreserveLast(&orig_err); } else { - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); }
/* If Perform returns < 0, then we need to cancel the VM
I could somehwat understand it here after the migration is done, but a bug could be also in this code.
Mostly because we're in a p2p migration where everything is done within a single API call and thus it cannot really fail.
@@ -5657,7 +5667,9 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, return ret; }
- qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3); + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3) < 0) + goto endjob; + virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup);
@@ -5671,7 +5683,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto endjob; }
- qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE));
if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, qemuMigrationSrcCleanup) < 0)
Same here.
It is similar to the p2p case... we already started a phase in the same API just a few lines above and thus this call cannot really fail. I guess I could add actual handling in all the cases here for consistency even though it would effectively be a dead code. I chose ignore_value() as it is less work :-) Jirka
We will want to update migration phase without affecting job ownership. Either in the thread that already owns the job or from an event handler which only changes the phase (of a job no-one owns) without assuming it. Let's move the ownership change to a new qemuDomainObjStartJobPhase helper and let qemuDomainObjSetJobPhase set the phase without touching ownership. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 44 +++++++++++++++++++++++++++++++++++---- src/qemu/qemu_domainjob.h | 3 +++ src/qemu/qemu_migration.c | 2 +- 3 files changed, 44 insertions(+), 5 deletions(-) diff --git a/src/qemu/qemu_domainjob.c b/src/qemu/qemu_domainjob.c index 8e8d229afe..10381755a9 100644 --- a/src/qemu/qemu_domainjob.c +++ b/src/qemu/qemu_domainjob.c @@ -714,6 +714,10 @@ qemuDomainJobDataToParams(virDomainJobData *jobData, } +/* + * Sets the job phase without changing the job owner. The owner is supposed to + * be 0 or the current thread, a warning is issued otherwise. + */ void qemuDomainObjSetJobPhase(virDomainObj *obj, int phase) @@ -728,19 +732,51 @@ qemuDomainObjSetJobPhase(virDomainObj *obj, virDomainAsyncJobTypeToString(priv->job.asyncJob), qemuDomainAsyncJobPhaseToString(priv->job.asyncJob, phase)); + if (priv->job.asyncOwner != 0 && + priv->job.asyncOwner != me) { + VIR_WARN("'%s' async job is owned by thread %llu, API '%s'", + virDomainAsyncJobTypeToString(priv->job.asyncJob), + priv->job.asyncOwner, + NULLSTR(priv->job.asyncOwnerAPI)); + } + + priv->job.phase = phase; + qemuDomainSaveStatus(obj); +} + + +/* + * Changes the job owner and sets the job phase. The current owner is supposed + * to be 0 or the current thread, a warning is issued otherwise. + */ +void +qemuDomainObjStartJobPhase(virDomainObj *obj, + int phase) +{ + qemuDomainObjPrivate *priv = obj->privateData; + unsigned long long me = virThreadSelfID(); + + if (!priv->job.asyncJob) + return; + + VIR_DEBUG("Starting phase '%s' of '%s' job", + qemuDomainAsyncJobPhaseToString(priv->job.asyncJob, phase), + virDomainAsyncJobTypeToString(priv->job.asyncJob)); + if (priv->job.asyncOwner == 0) { priv->job.asyncOwnerAPI = g_strdup(virThreadJobGet()); } else if (me != priv->job.asyncOwner) { - VIR_WARN("'%s' async job is owned by thread %llu", + VIR_WARN("'%s' async job is owned by thread %llu, API '%s'", virDomainAsyncJobTypeToString(priv->job.asyncJob), - priv->job.asyncOwner); + priv->job.asyncOwner, + NULLSTR(priv->job.asyncOwnerAPI)); } - priv->job.phase = phase; priv->job.asyncOwner = me; - qemuDomainSaveStatus(obj); + qemuDomainObjSetJobPhase(obj, phase); } + void qemuDomainObjSetAsyncJobMask(virDomainObj *obj, unsigned long long allowedJobs) diff --git a/src/qemu/qemu_domainjob.h b/src/qemu/qemu_domainjob.h index 707d4e91ed..e8021a7f04 100644 --- a/src/qemu/qemu_domainjob.h +++ b/src/qemu/qemu_domainjob.h @@ -156,6 +156,9 @@ void qemuDomainObjEndAsyncJob(virDomainObj *obj); void qemuDomainObjAbortAsyncJob(virDomainObj *obj); void qemuDomainObjSetJobPhase(virDomainObj *obj, int phase); +void +qemuDomainObjStartJobPhase(virDomainObj *obj, + int phase); void qemuDomainObjSetAsyncJobMask(virDomainObj *obj, unsigned long long allowedJobs); int qemuDomainObjPreserveJob(virDomainObj *obj, diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 1c5dd9b391..e4c0c1c6f4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -165,7 +165,7 @@ qemuMigrationJobSetPhase(virDomainObj *vm, if (qemuMigrationCheckPhase(vm, phase) < 0) return -1; - qemuDomainObjSetJobPhase(vm, phase); + qemuDomainObjStartJobPhase(vm, phase); return 0; } -- 2.35.1
On Tue, May 10, 2022 at 17:21:09 +0200, Jiri Denemark wrote:
We will want to update migration phase without affecting job ownership. Either in the thread that already owns the job or from an event handler which only changes the phase (of a job no-one owns) without assuming it.
Let's move the ownership change to a new qemuDomainObjStartJobPhase helper and let qemuDomainObjSetJobPhase set the phase without touching ownership.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_domainjob.c | 44 +++++++++++++++++++++++++++++++++++---- src/qemu/qemu_domainjob.h | 3 +++ src/qemu/qemu_migration.c | 2 +- 3 files changed, 44 insertions(+), 5 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Both qemuMigrationJobSetPhase and qemuMigrationJobStartPhase were doing the same thing, which made little sense. Let's follow the difference between qemuDomainObjSetJobPhase and qemuDomainObjStartJobPhase and change job owner only in qemuMigrationJobStartPhase. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index e4c0c1c6f4..3f6921b4b2 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -165,7 +165,7 @@ qemuMigrationJobSetPhase(virDomainObj *vm, if (qemuMigrationCheckPhase(vm, phase) < 0) return -1; - qemuDomainObjStartJobPhase(vm, phase); + qemuDomainObjSetJobPhase(vm, phase); return 0; } @@ -174,7 +174,11 @@ static int G_GNUC_WARN_UNUSED_RESULT qemuMigrationJobStartPhase(virDomainObj *vm, qemuMigrationJobPhase phase) { - return qemuMigrationJobSetPhase(vm, phase); + if (qemuMigrationCheckPhase(vm, phase) < 0) + return -1; + + qemuDomainObjStartJobPhase(vm, phase); + return 0; } @@ -2599,7 +2603,7 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, * change protection. */ if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT && - qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_BEGIN3) < 0) + qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_BEGIN3) < 0) return NULL; if (!qemuMigrationSrcIsAllowed(driver, vm, true, flags)) @@ -3152,7 +3156,7 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, flags) < 0) goto cleanup; - if (qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PREPARE) < 0) + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PREPARE) < 0) goto stopjob; /* Domain starts inactive, even if the domain XML had an id field. */ @@ -3673,7 +3677,7 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, else phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; - if (qemuMigrationJobSetPhase(vm, phase) < 0) + if (qemuMigrationJobStartPhase(vm, phase) < 0) return -1; if (!(mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, @@ -4928,7 +4932,7 @@ qemuMigrationSrcPerformPeer2Peer2(virQEMUDriver *driver, * until the migration is complete. */ VIR_DEBUG("Perform %p", sconn); - ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2)); + ignore_value(qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2)); if (flags & VIR_MIGRATE_TUNNELLED) ret = qemuMigrationSrcPerformTunnel(driver, vm, st, NULL, NULL, 0, NULL, NULL, @@ -5573,7 +5577,7 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, migParams, flags, dname, resource, &v3proto); } else { - if (qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2) < 0) + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM2) < 0) goto endjob; ret = qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, -- 2.35.1
On Tue, May 10, 2022 at 17:21:10 +0200, Jiri Denemark wrote:
Both qemuMigrationJobSetPhase and qemuMigrationJobStartPhase were doing the same thing, which made little sense. Let's follow the difference between qemuDomainObjSetJobPhase and qemuDomainObjStartJobPhase and change job owner only in qemuMigrationJobStartPhase.
Okay so at this point qemuMigrationJobSetPhase will be used only when advancing the Phase after already using qemuMigrationJobStartPhase in the same function.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 18 +++++++++++------- 1 file changed, 11 insertions(+), 7 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Thu, May 12, 2022 at 13:26:01 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:10 +0200, Jiri Denemark wrote:
Both qemuMigrationJobSetPhase and qemuMigrationJobStartPhase were doing the same thing, which made little sense. Let's follow the difference between qemuDomainObjSetJobPhase and qemuDomainObjStartJobPhase and change job owner only in qemuMigrationJobStartPhase.
Okay so at this point qemuMigrationJobSetPhase will be used only when advancing the Phase after already using qemuMigrationJobStartPhase in the same function.
Mostly, qemuMigrationJobSetPhase may also be used in callbacks which do not want to take over the job ownership, but need to change the phase. Jirka
This phase marks a migration protocol as broken in a post-copy phase. Libvirt is no longer actively watching the migration in this phase as the migration API that started the migration failed. This may either happen when post-copy migration really fails (QEMU enters postcopy-paused migration state) or when the migration still progresses between both QEMU processes, but libvirt lost control of it because the connection between libvirt daemons (in p2p migration) or a daemon and client (non-p2p migration) was closed. For example, when one of the daemons was restarted. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 15 +++++++++++---- src/qemu/qemu_process.c | 16 +++++++++++++--- 2 files changed, 24 insertions(+), 7 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 3f6921b4b2..c111dd8686 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2369,6 +2369,7 @@ qemuMigrationSrcCleanup(virDomainObj *vm, vm->def->name); if (virDomainObjIsPostcopy(vm, VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT)) { + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuMigrationSrcPostcopyFailed(vm); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); @@ -2380,8 +2381,10 @@ qemuMigrationSrcCleanup(virDomainObj *vm, } break; + case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_BEGIN_RESUME: case QEMU_MIGRATION_PHASE_PERFORM_RESUME: + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuMigrationSrcPostcopyFailed(vm); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); @@ -2402,7 +2405,6 @@ qemuMigrationSrcCleanup(virDomainObj *vm, case QEMU_MIGRATION_PHASE_PERFORM2: /* single phase outgoing migration; unreachable */ case QEMU_MIGRATION_PHASE_NONE: - case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_LAST: /* unreachable */ ; @@ -3774,6 +3776,7 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, flags, cancelled); if (virDomainObjIsFailedPostcopy(vm)) { + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); } else { @@ -5607,6 +5610,7 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, virErrorPreserveLast(&orig_err); if (virDomainObjIsFailedPostcopy(vm)) { + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); } else { @@ -5699,6 +5703,8 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, jobPriv->migParams, priv->job.apiFlags); qemuMigrationJobFinish(vm); } else { + if (ret < 0) + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); } @@ -5938,7 +5944,7 @@ qemuMigrationDstComplete(virQEMUDriver *driver, /* Guest is successfully running, so cancel previous auto destroy. There's * nothing to remove when we are resuming post-copy migration. */ - if (!virDomainObjIsFailedPostcopy(vm)) + if (job->phase < QEMU_MIGRATION_PHASE_POSTCOPY_FAILED) qemuProcessAutoDestroyRemove(driver, vm); /* Remove completed stats for post-copy, everything but timing fields @@ -6205,6 +6211,7 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, } if (virDomainObjIsFailedPostcopy(vm)) { + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuProcessAutoDestroyRemove(driver, vm); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); *finishJob = false; @@ -6327,9 +6334,9 @@ qemuMigrationProcessUnattended(virQEMUDriver *driver, vm->def->name); if (job == VIR_ASYNC_JOB_MIGRATION_IN) - phase = QEMU_MIGRATION_PHASE_FINISH3; + phase = QEMU_MIGRATION_PHASE_FINISH_RESUME; else - phase = QEMU_MIGRATION_PHASE_CONFIRM3; + phase = QEMU_MIGRATION_PHASE_CONFIRM_RESUME; if (qemuMigrationJobStartPhase(vm, phase) < 0) return; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index c7ed0a5c56..f42c9a3018 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -1624,7 +1624,8 @@ qemuProcessHandleMigrationStatus(qemuMonitor *mon G_GNUC_UNUSED, * watching it in any thread. Let's make sure the migration is properly * finished in case we get a "completed" event. */ - if (virDomainObjIsFailedPostcopy(vm) && + if (virDomainObjIsPostcopy(vm, priv->job.current->operation) && + priv->job.phase == QEMU_MIGRATION_PHASE_POSTCOPY_FAILED && priv->job.asyncOwner == 0 && status == QEMU_MONITOR_MIGRATION_STATUS_COMPLETED) { struct qemuProcessEvent *proc = g_new0(struct qemuProcessEvent, 1); @@ -3566,7 +3567,6 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, case QEMU_MIGRATION_PHASE_PERFORM3_DONE: case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: case QEMU_MIGRATION_PHASE_CONFIRM3: - case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_BEGIN_RESUME: case QEMU_MIGRATION_PHASE_PERFORM_RESUME: case QEMU_MIGRATION_PHASE_CONFIRM_RESUME: @@ -3604,6 +3604,7 @@ qemuProcessRecoverMigrationIn(virQEMUDriver *driver, } break; + case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_PREPARE_RESUME: case QEMU_MIGRATION_PHASE_FINISH_RESUME: return 1; @@ -3639,7 +3640,6 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, case QEMU_MIGRATION_PHASE_PREPARE: case QEMU_MIGRATION_PHASE_FINISH2: case QEMU_MIGRATION_PHASE_FINISH3: - case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_PREPARE_RESUME: case QEMU_MIGRATION_PHASE_FINISH_RESUME: case QEMU_MIGRATION_PHASE_LAST: @@ -3700,6 +3700,7 @@ qemuProcessRecoverMigrationOut(virQEMUDriver *driver, } return 1; + case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_BEGIN_RESUME: case QEMU_MIGRATION_PHASE_PERFORM_RESUME: return 1; @@ -3751,9 +3752,18 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, return -1; if (rc > 0) { + job->phase = QEMU_MIGRATION_PHASE_POSTCOPY_FAILED; + if (migStatus == VIR_DOMAIN_JOB_STATUS_POSTCOPY) { VIR_DEBUG("Post-copy migration of domain %s still running, it " "will be handled as unattended", vm->def->name); + + if (state == VIR_DOMAIN_RUNNING) + reason = VIR_DOMAIN_RUNNING_POSTCOPY; + else + reason = VIR_DOMAIN_PAUSED_POSTCOPY; + + virDomainObjSetState(vm, state, reason); qemuProcessRestoreMigrationJob(vm, job); return 0; } -- 2.35.1
On Tue, May 10, 2022 at 17:21:11 +0200, Jiri Denemark wrote:
This phase marks a migration protocol as broken in a post-copy phase. Libvirt is no longer actively watching the migration in this phase as the migration API that started the migration failed.
This may either happen when post-copy migration really fails (QEMU enters postcopy-paused migration state) or when the migration still progresses between both QEMU processes, but libvirt lost control of it because the connection between libvirt daemons (in p2p migration) or a daemon and client (non-p2p migration) was closed. For example, when one of the daemons was restarted.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 15 +++++++++++---- src/qemu/qemu_process.c | 16 +++++++++++++--- 2 files changed, 24 insertions(+), 7 deletions(-)
[...]
@@ -6327,9 +6334,9 @@ qemuMigrationProcessUnattended(virQEMUDriver *driver, vm->def->name);
if (job == VIR_ASYNC_JOB_MIGRATION_IN) - phase = QEMU_MIGRATION_PHASE_FINISH3; + phase = QEMU_MIGRATION_PHASE_FINISH_RESUME; else - phase = QEMU_MIGRATION_PHASE_CONFIRM3; + phase = QEMU_MIGRATION_PHASE_CONFIRM_RESUME;
if (qemuMigrationJobStartPhase(vm, phase) < 0) return;
This hunk seems to be misplaced or at least doesn't really seem to be related to anything this patch is claiming to do. [..]
@@ -3751,9 +3752,18 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, return -1;
if (rc > 0) { + job->phase = QEMU_MIGRATION_PHASE_POSTCOPY_FAILED; + if (migStatus == VIR_DOMAIN_JOB_STATUS_POSTCOPY) { VIR_DEBUG("Post-copy migration of domain %s still running, it " "will be handled as unattended", vm->def->name); + + if (state == VIR_DOMAIN_RUNNING) + reason = VIR_DOMAIN_RUNNING_POSTCOPY; + else + reason = VIR_DOMAIN_PAUSED_POSTCOPY;
This bit also doesn't seem to be justified by what this patch is supposed to do.
+ + virDomainObjSetState(vm, state, reason);
The rest looks good so if you explain what's going on in those two cases you can use: Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Thu, May 12, 2022 at 13:50:08 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:11 +0200, Jiri Denemark wrote:
This phase marks a migration protocol as broken in a post-copy phase. Libvirt is no longer actively watching the migration in this phase as the migration API that started the migration failed.
This may either happen when post-copy migration really fails (QEMU enters postcopy-paused migration state) or when the migration still progresses between both QEMU processes, but libvirt lost control of it because the connection between libvirt daemons (in p2p migration) or a daemon and client (non-p2p migration) was closed. For example, when one of the daemons was restarted.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 15 +++++++++++---- src/qemu/qemu_process.c | 16 +++++++++++++--- 2 files changed, 24 insertions(+), 7 deletions(-)
[...]
@@ -6327,9 +6334,9 @@ qemuMigrationProcessUnattended(virQEMUDriver *driver, vm->def->name);
if (job == VIR_ASYNC_JOB_MIGRATION_IN) - phase = QEMU_MIGRATION_PHASE_FINISH3; + phase = QEMU_MIGRATION_PHASE_FINISH_RESUME; else - phase = QEMU_MIGRATION_PHASE_CONFIRM3; + phase = QEMU_MIGRATION_PHASE_CONFIRM_RESUME;
if (qemuMigrationJobStartPhase(vm, phase) < 0) return;
This hunk seems to be misplaced or at least doesn't really seem to be related to anything this patch is claiming to do.
Thanks to changes in this patch migration is in QEMU_MIGRATION_PHASE_POSTCOPY_FAILED phase when we get here so to make it all work, we need to use RESUME phases > QEMU_MIGRATION_PHASE_POSTCOPY_FAILED. Otherwise qemuMigrationCheckPhase would complain. Perhaps this could be moved to "Add new migration phases for post-copy recovery", but I haven't checked for sure.
[..]
@@ -3751,9 +3752,18 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, return -1;
if (rc > 0) { + job->phase = QEMU_MIGRATION_PHASE_POSTCOPY_FAILED; + if (migStatus == VIR_DOMAIN_JOB_STATUS_POSTCOPY) { VIR_DEBUG("Post-copy migration of domain %s still running, it " "will be handled as unattended", vm->def->name); + + if (state == VIR_DOMAIN_RUNNING) + reason = VIR_DOMAIN_RUNNING_POSTCOPY; + else + reason = VIR_DOMAIN_PAUSED_POSTCOPY;
This bit also doesn't seem to be justified by what this patch is supposed to do.
Until now broken migration protocol in post-copy phase has been indicated with VIR_DOMAIN_*_POSTCOPY_FAILED states, which changed in this patch and we can use the right state here depending on the QEMU state. That said, it could be moved into a separate patch. Jirka
On Wed, May 18, 2022 at 14:53:50 +0200, Jiri Denemark wrote:
On Thu, May 12, 2022 at 13:50:08 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:11 +0200, Jiri Denemark wrote:
This phase marks a migration protocol as broken in a post-copy phase. Libvirt is no longer actively watching the migration in this phase as the migration API that started the migration failed.
This may either happen when post-copy migration really fails (QEMU enters postcopy-paused migration state) or when the migration still progresses between both QEMU processes, but libvirt lost control of it because the connection between libvirt daemons (in p2p migration) or a daemon and client (non-p2p migration) was closed. For example, when one of the daemons was restarted.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 15 +++++++++++---- src/qemu/qemu_process.c | 16 +++++++++++++--- 2 files changed, 24 insertions(+), 7 deletions(-)
[...]
@@ -6327,9 +6334,9 @@ qemuMigrationProcessUnattended(virQEMUDriver *driver, vm->def->name);
if (job == VIR_ASYNC_JOB_MIGRATION_IN) - phase = QEMU_MIGRATION_PHASE_FINISH3; + phase = QEMU_MIGRATION_PHASE_FINISH_RESUME; else - phase = QEMU_MIGRATION_PHASE_CONFIRM3; + phase = QEMU_MIGRATION_PHASE_CONFIRM_RESUME;
if (qemuMigrationJobStartPhase(vm, phase) < 0) return;
This hunk seems to be misplaced or at least doesn't really seem to be related to anything this patch is claiming to do.
Thanks to changes in this patch migration is in QEMU_MIGRATION_PHASE_POSTCOPY_FAILED phase when we get here so to make it all work, we need to use RESUME phases > QEMU_MIGRATION_PHASE_POSTCOPY_FAILED. Otherwise qemuMigrationCheckPhase would complain. Perhaps this could be moved to "Add new migration phases for post-copy recovery", but I haven't checked for sure.
[..]
@@ -3751,9 +3752,18 @@ qemuProcessRecoverMigration(virQEMUDriver *driver, return -1;
if (rc > 0) { + job->phase = QEMU_MIGRATION_PHASE_POSTCOPY_FAILED; + if (migStatus == VIR_DOMAIN_JOB_STATUS_POSTCOPY) { VIR_DEBUG("Post-copy migration of domain %s still running, it " "will be handled as unattended", vm->def->name); + + if (state == VIR_DOMAIN_RUNNING) + reason = VIR_DOMAIN_RUNNING_POSTCOPY; + else + reason = VIR_DOMAIN_PAUSED_POSTCOPY;
This bit also doesn't seem to be justified by what this patch is supposed to do.
Until now broken migration protocol in post-copy phase has been indicated with VIR_DOMAIN_*_POSTCOPY_FAILED states, which changed in this patch and we can use the right state here depending on the QEMU state. That said, it could be moved into a separate patch.
Fair enough on both points. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
This flag can be used to restart post-copy migration once it failed because of a broken connection. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 5 +++++ src/libvirt-domain.c | 6 ++++++ 2 files changed, 11 insertions(+) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index 34b8adc2bf..aee2b49114 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -1084,6 +1084,11 @@ typedef enum { */ VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES = (1 << 18), + /* Resume migration which failed in post-copy phase. + * + * Since: 8.4.0 + */ + VIR_MIGRATE_POSTCOPY_RESUME = (1 << 19), } virDomainMigrateFlags; diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 2e39687e27..41ba4aa39c 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -9762,6 +9762,12 @@ virDomainMigrateGetMaxSpeed(virDomainPtr domain, * in such case. Because of this, libvirt will refuse to cancel post-copy * migration via virDomainAbortJob. * + * Failed post-copy migration can be recovered once the cause for the failure + * (e.g., a network issue) is resolved by repeating the migration with an + * additional VIR_MIGRATE_POSTCOPY_RESUME flag. This will recreate the + * connection and resume migration from the point where it failed. This step + * can be repeated in case the migration breaks again. + * * The following domain life cycle events are emitted during post-copy * migration: * VIR_DOMAIN_EVENT_SUSPENDED_POSTCOPY (on the source) -- migration entered -- 2.35.1
On Tue, May 10, 2022 at 17:21:12 +0200, Jiri Denemark wrote:
This flag can be used to restart post-copy migration once it failed because of a broken connection.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 5 +++++ src/libvirt-domain.c | 6 ++++++ 2 files changed, 11 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- docs/manpages/virsh.rst | 9 +++++++-- tools/virsh-domain.c | 8 ++++++++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst index e73e590754..e36d64c164 100644 --- a/docs/manpages/virsh.rst +++ b/docs/manpages/virsh.rst @@ -3246,7 +3246,8 @@ migrate migrate [--live] [--offline] [--direct] [--p2p [--tunnelled]] [--persistent] [--undefinesource] [--suspend] [--copy-storage-all] [--copy-storage-inc] [--change-protection] [--unsafe] [--verbose] - [--rdma-pin-all] [--abort-on-error] [--postcopy] [--postcopy-after-precopy] + [--rdma-pin-all] [--abort-on-error] [--postcopy] + [--postcopy-after-precopy] [--postcopy-resume] domain desturi [migrateuri] [graphicsuri] [listen-address] [dname] [--timeout seconds [--timeout-suspend | --timeout-postcopy]] [--xml file] [--migrate-disks disk-list] [--disks-port port] @@ -3302,7 +3303,11 @@ Once migration is running, the user may switch to post-copy using the automatically switch to post-copy after the first pass of pre-copy is finished. The maximum bandwidth consumed during the post-copy phase may be limited using *--postcopy-bandwidth*. The maximum bandwidth consumed during the pre-copy phase -may be limited using *--bandwidth*. +may be limited using *--bandwidth*. In case connection between the hosts breaks +while migration is in post-copy mode, the domain cannot be resumed on either +source or destination host and the ``migrate`` command will report an error +leaving the domain active on both hosts. To recover from such situation repeat +the original ``migrate`` command with an additional *--postcopy-resume* flag. *--auto-converge* forces convergence during live migration. The initial guest CPU throttling rate can be set with *auto-converge-initial*. If the diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index ba492e807e..b0d5b15dff 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -10807,6 +10807,10 @@ static const vshCmdOptDef opts_migrate[] = { .type = VSH_OT_BOOL, .help = N_("automatically switch to post-copy migration after one pass of pre-copy") }, + {.name = "postcopy-resume", + .type = VSH_OT_BOOL, + .help = N_("resume failed post-copy migration") + }, {.name = "migrateuri", .type = VSH_OT_STRING, .completer = virshCompleteEmpty, @@ -11210,6 +11214,9 @@ doMigrate(void *opaque) if (vshCommandOptBool(cmd, "postcopy")) flags |= VIR_MIGRATE_POSTCOPY; + if (vshCommandOptBool(cmd, "postcopy-resume")) + flags |= VIR_MIGRATE_POSTCOPY_RESUME; + if (vshCommandOptBool(cmd, "tls")) flags |= VIR_MIGRATE_TLS; @@ -11314,6 +11321,7 @@ cmdMigrate(vshControl *ctl, const vshCmd *cmd) VSH_EXCLUSIVE_OPTIONS("live", "offline"); VSH_EXCLUSIVE_OPTIONS("timeout-suspend", "timeout-postcopy"); VSH_REQUIRE_OPTION("postcopy-after-precopy", "postcopy"); + VSH_REQUIRE_OPTION("postcopy-resume", "postcopy"); VSH_REQUIRE_OPTION("timeout-postcopy", "postcopy"); VSH_REQUIRE_OPTION("persistent-xml", "persistent"); VSH_REQUIRE_OPTION("tls-destination", "tls"); -- 2.35.1
On Tue, May 10, 2022 at 17:21:13 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- docs/manpages/virsh.rst | 9 +++++++-- tools/virsh-domain.c | 8 ++++++++ 2 files changed, 15 insertions(+), 2 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
For historical reasons we automatically enabled VIR_MIGRATE_PAUSED flag when a migration was started for a paused domain. However, when resuming failed post-copy migration the domain on the source host will always be paused (as it is already running on the destination host). We must avoid enabling VIR_MIGRATE_PAUSED in this case. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/libvirt-domain.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 41ba4aa39c..07fb8119d9 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -3279,7 +3279,9 @@ virDomainMigrateVersion3Full(virDomainPtr domain, ret = virDomainGetInfo(domain, &info); state = info.state; } - if (ret == 0 && state == VIR_DOMAIN_PAUSED) + if (ret == 0 && + state == VIR_DOMAIN_PAUSED && + !(flags & VIR_MIGRATE_POSTCOPY_RESUME)) flags |= VIR_MIGRATE_PAUSED; destflags = flags & ~(VIR_MIGRATE_ABORT_ON_ERROR | -- 2.35.1
On Tue, May 10, 2022 at 17:21:14 +0200, Jiri Denemark wrote:
For historical reasons we automatically enabled VIR_MIGRATE_PAUSED flag when a migration was started for a paused domain. However, when resuming failed post-copy migration the domain on the source host will always be paused (as it is already running on the destination host). We must avoid enabling VIR_MIGRATE_PAUSED in this case.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/libvirt-domain.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Mostly we just need to check whether the domain is in a failed post-copy migration that can be resumed. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 143 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c111dd8686..99b1d4b88b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2717,6 +2717,143 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, flags); } + +static bool +qemuMigrationAnyCanResume(virDomainObj *vm, + virDomainAsyncJob job, + unsigned long flags, + qemuMigrationJobPhase expectedPhase) +{ + qemuDomainObjPrivate *priv = vm->privateData; + + VIR_DEBUG("vm=%p, job=%s, flags=0x%lx, expectedPhase=%s", + vm, virDomainAsyncJobTypeToString(job), flags, + qemuDomainAsyncJobPhaseToString(VIR_ASYNC_JOB_MIGRATION_OUT, + expectedPhase)); + + if (!(flags & VIR_MIGRATE_POSTCOPY)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "post-copy to be enabled")); + return false; + } + + /* This should never happen since POSTCOPY_RESUME is newer than + * CHANGE_PROTECTION, but let's check it anyway in case we're talking to + * a weired client. + */ + if (job == VIR_ASYNC_JOB_MIGRATION_OUT && + expectedPhase < QEMU_MIGRATION_PHASE_PERFORM_RESUME && + !(flags & VIR_MIGRATE_CHANGE_PROTECTION)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "change protection")); + return NULL; + } + + if (!qemuMigrationJobIsActive(vm, job)) + return false; + + if (priv->job.asyncOwner != 0 && + priv->job.asyncOwner != virThreadSelfID()) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("migration of domain %s is being actively monitored " + "by another thread"), + vm->def->name); + return false; + } + + if (!virDomainObjIsPostcopy(vm, priv->job.current->operation)) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("migration of domain %s is not in post-copy phase"), + vm->def->name); + return false; + } + + if (priv->job.phase < QEMU_MIGRATION_PHASE_POSTCOPY_FAILED && + !virDomainObjIsFailedPostcopy(vm)) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("post-copy migration of domain %s has not failed"), + vm->def->name); + return false; + } + + if (priv->job.phase > expectedPhase) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("resuming failed post-copy migration of domain %s " + "already in progress"), + vm->def->name); + return false; + } + + return true; +} + + +static char * +qemuMigrationSrcBeginResume(virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + virDomainJobStatus status; + + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + &status) < 0) + return NULL; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + return NULL; + } + + return qemuMigrationSrcBeginXML(driver, vm, xmlin, + cookieout, cookieoutlen, 0, NULL, 0, flags); +} + + +static char * +qemuMigrationSrcBeginResumePhase(virConnectPtr conn, + virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + g_autofree char *xml = NULL; + + VIR_DEBUG("vm=%p", vm); + + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_OUT, flags, + QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)) + return NULL; + + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_BEGIN_RESUME) < 0) + return NULL; + + virCloseCallbacksUnset(driver->closeCallbacks, vm, + qemuMigrationSrcCleanup); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); + + xml = qemuMigrationSrcBeginResume(driver, vm, xmlin, cookieout, cookieoutlen, flags); + + if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, + qemuMigrationSrcCleanup) < 0) + g_clear_pointer(&xml, g_free); + + if (!xml) + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); + + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + return g_steal_pointer(&xml); +} + + char * qemuMigrationSrcBegin(virConnectPtr conn, virDomainObj *vm, @@ -2741,6 +2878,12 @@ qemuMigrationSrcBegin(virConnectPtr conn, goto cleanup; } + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + xml = qemuMigrationSrcBeginResumePhase(conn, driver, vm, xmlin, + cookieout, cookieoutlen, flags); + goto cleanup; + } + if ((flags & VIR_MIGRATE_CHANGE_PROTECTION)) { if (qemuMigrationJobStart(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, flags) < 0) -- 2.35.1
On Tue, May 10, 2022 at 17:21:15 +0200, Jiri Denemark wrote:
Mostly we just need to check whether the domain is in a failed post-copy migration that can be resumed.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 143 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c111dd8686..99b1d4b88b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2717,6 +2717,143 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, flags); }
+ +static bool +qemuMigrationAnyCanResume(virDomainObj *vm, + virDomainAsyncJob job, + unsigned long flags, + qemuMigrationJobPhase expectedPhase) +{ + qemuDomainObjPrivate *priv = vm->privateData; + + VIR_DEBUG("vm=%p, job=%s, flags=0x%lx, expectedPhase=%s", + vm, virDomainAsyncJobTypeToString(job), flags, + qemuDomainAsyncJobPhaseToString(VIR_ASYNC_JOB_MIGRATION_OUT, + expectedPhase)); + + if (!(flags & VIR_MIGRATE_POSTCOPY)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "post-copy to be enabled"));
Error messages shall not have line breaks. Also ... requiring this flag seems to be a bit pointless and ... cosmetic at best. Since the previous migration would need to be started with the post-copy flag already which is checked below.
+ return false; + } + + /* This should never happen since POSTCOPY_RESUME is newer than + * CHANGE_PROTECTION, but let's check it anyway in case we're talking to + * a weired client. + */ + if (job == VIR_ASYNC_JOB_MIGRATION_OUT && + expectedPhase < QEMU_MIGRATION_PHASE_PERFORM_RESUME && + !(flags & VIR_MIGRATE_CHANGE_PROTECTION)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "change protection"));
Same here.
+ return NULL; + }
[...]
+static char * +qemuMigrationSrcBeginResume(virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + virDomainJobStatus status; + + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + &status) < 0) + return NULL; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + return NULL; + } + + return qemuMigrationSrcBeginXML(driver, vm, xmlin, + cookieout, cookieoutlen, 0, NULL, 0, flags);
Certain steps here around the non-shared-storage migration are redundant at this point and IMO should be skipped.
+} + + +static char * +qemuMigrationSrcBeginResumePhase(virConnectPtr conn, + virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + g_autofree char *xml = NULL; + + VIR_DEBUG("vm=%p", vm); + + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_OUT, flags, + QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)) + return NULL; + + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_BEGIN_RESUME) < 0) + return NULL; + + virCloseCallbacksUnset(driver->closeCallbacks, vm, + qemuMigrationSrcCleanup); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob);
Could you please add an explanation why we are removing the callbacks ...
+ + xml = qemuMigrationSrcBeginResume(driver, vm, xmlin, cookieout, cookieoutlen, flags); + + if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, + qemuMigrationSrcCleanup) < 0) + g_clear_pointer(&xml, g_free); + + if (!xml) + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); + + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob);
Just to add them here?
+ qemuMigrationJobContinue(vm); + return g_steal_pointer(&xml); +}
On Thu, May 12, 2022 at 14:14:15 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:15 +0200, Jiri Denemark wrote:
Mostly we just need to check whether the domain is in a failed post-copy migration that can be resumed.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 143 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c111dd8686..99b1d4b88b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2717,6 +2717,143 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, flags); }
+ +static bool +qemuMigrationAnyCanResume(virDomainObj *vm, + virDomainAsyncJob job, + unsigned long flags, + qemuMigrationJobPhase expectedPhase) +{ + qemuDomainObjPrivate *priv = vm->privateData; + + VIR_DEBUG("vm=%p, job=%s, flags=0x%lx, expectedPhase=%s", + vm, virDomainAsyncJobTypeToString(job), flags, + qemuDomainAsyncJobPhaseToString(VIR_ASYNC_JOB_MIGRATION_OUT, + expectedPhase)); + + if (!(flags & VIR_MIGRATE_POSTCOPY)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "post-copy to be enabled"));
Error messages shall not have line breaks.
Also ... requiring this flag seems to be a bit pointless and ... cosmetic at best. Since the previous migration would need to be started with the post-copy flag already which is checked below.
Right, the previous migration must have been started as post-copy, but resuming it is post-copy too. We could have implied the flag based on POSTCOPY_RESUME (which is ugly) or all relevant places checking POSTCOPY would need to check POSTCOPY_RESUME too (which is ugly as well). And with the expected use case (repeating the failed migration command with all the flags and arguments and adding an extra POSTCOPY_RESUME flag) in mind requiring POSTCOPY to be used with POSTCOPY_RESUME looks like the the best option to me.
+ return false; + } + + /* This should never happen since POSTCOPY_RESUME is newer than + * CHANGE_PROTECTION, but let's check it anyway in case we're talking to + * a weired client. + */ + if (job == VIR_ASYNC_JOB_MIGRATION_OUT && + expectedPhase < QEMU_MIGRATION_PHASE_PERFORM_RESUME && + !(flags & VIR_MIGRATE_CHANGE_PROTECTION)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "change protection"));
Same here.
+ return NULL; + }
[...]
+static char * +qemuMigrationSrcBeginResume(virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + virDomainJobStatus status; + + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + &status) < 0) + return NULL; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + return NULL; + } + + return qemuMigrationSrcBeginXML(driver, vm, xmlin, + cookieout, cookieoutlen, 0, NULL, 0, flags);
Certain steps here around the non-shared-storage migration are redundant at this point and IMO should be skipped.
Which is what happens in qemuMigrationSrcBegin: if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { xml = qemuMigrationSrcBeginResumePhase(...); goto cleanup; } ... Or did I miss anything?
+} + + +static char * +qemuMigrationSrcBeginResumePhase(virConnectPtr conn, + virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + g_autofree char *xml = NULL; + + VIR_DEBUG("vm=%p", vm); + + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_OUT, flags, + QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)) + return NULL; + + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_BEGIN_RESUME) < 0) + return NULL; + + virCloseCallbacksUnset(driver->closeCallbacks, vm, + qemuMigrationSrcCleanup); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob);
Could you please add an explanation why we are removing the callbacks ...
+ + xml = qemuMigrationSrcBeginResume(driver, vm, xmlin, cookieout, cookieoutlen, flags); + + if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, + qemuMigrationSrcCleanup) < 0) + g_clear_pointer(&xml, g_free); + + if (!xml) + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); + + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob);
Just to add them here?
That's what happens in all migration APIs. The goal is to have the callbacks active only when there is not API running any part of the migration. Otherwise the API itself will handle the situations cover by the callbacks. If this is not needed and we are guaranteed the callbacks cannot be run while a migration API is active or after it already handled the same situation, we could replace this pattern. Although it would deserve its own separate series. Also note that in some cases (not in this particular one, though), we're removing different callabcks then what we're going to add later.
+ qemuMigrationJobContinue(vm); + return g_steal_pointer(&xml); +}
Jirka
On Wed, May 18, 2022 at 15:40:18 +0200, Jiri Denemark wrote:
On Thu, May 12, 2022 at 14:14:15 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:15 +0200, Jiri Denemark wrote:
Mostly we just need to check whether the domain is in a failed post-copy migration that can be resumed.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 143 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c111dd8686..99b1d4b88b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2717,6 +2717,143 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, flags); }
+ +static bool +qemuMigrationAnyCanResume(virDomainObj *vm, + virDomainAsyncJob job, + unsigned long flags, + qemuMigrationJobPhase expectedPhase) +{ + qemuDomainObjPrivate *priv = vm->privateData; + + VIR_DEBUG("vm=%p, job=%s, flags=0x%lx, expectedPhase=%s", + vm, virDomainAsyncJobTypeToString(job), flags, + qemuDomainAsyncJobPhaseToString(VIR_ASYNC_JOB_MIGRATION_OUT, + expectedPhase)); + + if (!(flags & VIR_MIGRATE_POSTCOPY)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "post-copy to be enabled"));
Error messages shall not have line breaks.
Also ... requiring this flag seems to be a bit pointless and ... cosmetic at best. Since the previous migration would need to be started with the post-copy flag already which is checked below.
Right, the previous migration must have been started as post-copy, but resuming it is post-copy too. We could have implied the flag based on POSTCOPY_RESUME (which is ugly) or all relevant places checking POSTCOPY would need to check POSTCOPY_RESUME too (which is ugly as well). And with the expected use case (repeating the failed migration command with all the flags and arguments and adding an extra POSTCOPY_RESUME flag) in mind requiring POSTCOPY to be used with POSTCOPY_RESUME looks like the the best option to me.
I was kind of referring to the fact that it's implied that postcopy is being resumed with a POSTCOPY_RESUME flag and you need to check anyways that the migration is indeed postcopy, so this feels redundant, but I think it's okay.
+ return false; + } + + /* This should never happen since POSTCOPY_RESUME is newer than + * CHANGE_PROTECTION, but let's check it anyway in case we're talking to + * a weired client. + */ + if (job == VIR_ASYNC_JOB_MIGRATION_OUT && + expectedPhase < QEMU_MIGRATION_PHASE_PERFORM_RESUME && + !(flags & VIR_MIGRATE_CHANGE_PROTECTION)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "change protection"));
Same here.
+ return NULL; + }
[...]
+static char * +qemuMigrationSrcBeginResume(virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + virDomainJobStatus status; + + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + &status) < 0) + return NULL; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + return NULL; + } + + return qemuMigrationSrcBeginXML(driver, vm, xmlin, + cookieout, cookieoutlen, 0, NULL, 0, flags);
Certain steps here around the non-shared-storage migration are redundant at this point and IMO should be skipped.
Which is what happens in qemuMigrationSrcBegin:
if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { xml = qemuMigrationSrcBeginResumePhase(...); goto cleanup; }
...
Or did I miss anything?
In 'qemuMigrationSrcBeginXML' qemuMigrationSrcBeginPhaseBlockDirtyBitmaps is called which does a bunch of setup for bitmap migration which is pointless when recovering as storage is copied already if the CPUs are running on the destination.
On Wed, May 18, 2022 at 15:59:46 +0200, Peter Krempa wrote:
On Wed, May 18, 2022 at 15:40:18 +0200, Jiri Denemark wrote:
On Thu, May 12, 2022 at 14:14:15 +0200, Peter Krempa wrote:
+static char * +qemuMigrationSrcBeginResume(virQEMUDriver *driver, + virDomainObj *vm, + const char *xmlin, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + virDomainJobStatus status; + + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + &status) < 0) + return NULL; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + return NULL; + } + + return qemuMigrationSrcBeginXML(driver, vm, xmlin, + cookieout, cookieoutlen, 0, NULL, 0, flags);
Certain steps here around the non-shared-storage migration are redundant at this point and IMO should be skipped.
Which is what happens in qemuMigrationSrcBegin:
if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { xml = qemuMigrationSrcBeginResumePhase(...); goto cleanup; }
...
Or did I miss anything?
In 'qemuMigrationSrcBeginXML' qemuMigrationSrcBeginPhaseBlockDirtyBitmaps is called which does a bunch of setup for bitmap migration which is pointless when recovering as storage is copied already if the CPUs are running on the destination.
Oh I see. This part is actually skipped, although it is not easily visible. The function is only called when QEMU_MIGRATION_COOKIE_NBD is set in cookieFlags and qemuMigrationSrcBeginResume calls qemuMigrationSrcBeginXML with cookieFlags == 0. Jirka
On Tue, May 10, 2022 at 05:21:15PM +0200, Jiri Denemark wrote:
Mostly we just need to check whether the domain is in a failed post-copy migration that can be resumed.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 143 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 143 insertions(+)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c111dd8686..99b1d4b88b 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2717,6 +2717,143 @@ qemuMigrationSrcBeginPhase(virQEMUDriver *driver, flags); }
+ +static bool +qemuMigrationAnyCanResume(virDomainObj *vm, + virDomainAsyncJob job, + unsigned long flags, + qemuMigrationJobPhase expectedPhase) +{ + qemuDomainObjPrivate *priv = vm->privateData; + + VIR_DEBUG("vm=%p, job=%s, flags=0x%lx, expectedPhase=%s", + vm, virDomainAsyncJobTypeToString(job), flags, + qemuDomainAsyncJobPhaseToString(VIR_ASYNC_JOB_MIGRATION_OUT, + expectedPhase)); + + if (!(flags & VIR_MIGRATE_POSTCOPY)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "post-copy to be enabled")); + return false; + } + + /* This should never happen since POSTCOPY_RESUME is newer than + * CHANGE_PROTECTION, but let's check it anyway in case we're talking to + * a weired client. + */ + if (job == VIR_ASYNC_JOB_MIGRATION_OUT && + expectedPhase < QEMU_MIGRATION_PHASE_PERFORM_RESUME && + !(flags & VIR_MIGRATE_CHANGE_PROTECTION)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("resuming failed post-copy migration requires " + "change protection")); + return NULL;
s/return NULL/return false/
+ } + + if (!qemuMigrationJobIsActive(vm, job)) + return false; + + if (priv->job.asyncOwner != 0 && + priv->job.asyncOwner != virThreadSelfID()) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("migration of domain %s is being actively monitored " + "by another thread"), + vm->def->name); + return false; + } + + if (!virDomainObjIsPostcopy(vm, priv->job.current->operation)) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("migration of domain %s is not in post-copy phase"), + vm->def->name); + return false; + } + + if (priv->job.phase < QEMU_MIGRATION_PHASE_POSTCOPY_FAILED && + !virDomainObjIsFailedPostcopy(vm)) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("post-copy migration of domain %s has not failed"), + vm->def->name); + return false; + } + + if (priv->job.phase > expectedPhase) { + virReportError(VIR_ERR_OPERATION_INVALID, + _("resuming failed post-copy migration of domain %s " + "already in progress"), + vm->def->name); + return false; + } + + return true; +}
To make the code flow a bit more sensible. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 99b1d4b88b..dd18a4ad63 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5819,29 +5819,27 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, } if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3) < 0) - goto endjob; + goto cleanup; virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup); - ret = qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, - cookieout, cookieoutlen, - flags, resource, NULL, graphicsuri, - nmigrate_disks, migrate_disks, migParams, nbdURI); - - if (ret < 0) { - qemuMigrationSrcRestoreDomainState(driver, vm); - goto endjob; - } - - ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); + if (qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, + cookieout, cookieoutlen, + flags, resource, NULL, graphicsuri, + nmigrate_disks, migrate_disks, migParams, nbdURI) < 0) + goto cleanup; if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, qemuMigrationSrcCleanup) < 0) - goto endjob; + goto cleanup; - endjob: + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); + ret = 0; + + cleanup: if (ret < 0 && !virDomainObjIsFailedPostcopy(vm)) { + qemuMigrationSrcRestoreDomainState(driver, vm); qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, jobPriv->migParams, priv->job.apiFlags); qemuMigrationJobFinish(vm); -- 2.35.1
On Tue, May 10, 2022 at 17:21:16 +0200, Jiri Denemark wrote:
To make the code flow a bit more sensible.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 26 ++++++++++++-------------- 1 file changed, 12 insertions(+), 14 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 99b1d4b88b..dd18a4ad63 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -5819,29 +5819,27 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, }
if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3) < 0) - goto endjob; + goto cleanup;
virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup);
- ret = qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, - cookieout, cookieoutlen, - flags, resource, NULL, graphicsuri, - nmigrate_disks, migrate_disks, migParams, nbdURI); - - if (ret < 0) { - qemuMigrationSrcRestoreDomainState(driver, vm); - goto endjob; - } - - ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); + if (qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, + cookieout, cookieoutlen, + flags, resource, NULL, graphicsuri, + nmigrate_disks, migrate_disks, migParams, nbdURI) < 0) + goto cleanup;
if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, qemuMigrationSrcCleanup) < 0) - goto endjob; + goto cleanup;
So this failure now becomes an error from previously being technically ignored, and ...
- endjob: + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); + ret = 0; + + cleanup: if (ret < 0 && !virDomainObjIsFailedPostcopy(vm)) { + qemuMigrationSrcRestoreDomainState(driver, vm);
... this call will not be skipped in such case, thus restoring the VM, but since we fail anyways the migration will be cancelled. So I think this can be considered a "bugfix" for a extremely unlikely/impossible bug.
qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, jobPriv->migParams, priv->job.apiFlags); qemuMigrationJobFinish(vm); -- 2.35.1
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
qemuMigrationSrcRun does a lot of thing before and after telling QEMU to start the migration. Let's make the core reusable by moving it to a new qemuMigrationSrcStart function. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 118 +++++++++++++++++++++++--------------- 1 file changed, 71 insertions(+), 47 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index dd18a4ad63..e1c67c51ec 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4413,6 +4413,76 @@ qemuMigrationSrcRunPrepareBlockDirtyBitmaps(virDomainObj *vm, } +/* The caller is supposed to enter monitor before calling this. */ +static int +qemuMigrationSrcStart(virQEMUDriver *driver, + virDomainObj *vm, + qemuMigrationSpec *spec, + unsigned int migrateFlags, + int *tunnelFd) +{ + qemuDomainObjPrivate *priv = vm->privateData; + g_autofree char *timestamp = NULL; + int rc; + + /* connect to the destination qemu if needed */ + if ((spec->destType == MIGRATION_DEST_CONNECT_HOST || + spec->destType == MIGRATION_DEST_CONNECT_SOCKET) && + qemuMigrationSrcConnect(driver, vm, spec) < 0) { + return -1; + } + + /* log start of migration */ + if ((timestamp = virTimeStringNow()) != NULL) + qemuDomainLogAppendMessage(driver, vm, "%s: initiating migration\n", timestamp); + + switch (spec->destType) { + case MIGRATION_DEST_HOST: + if (STREQ(spec->dest.host.protocol, "rdma") && + vm->def->mem.hard_limit > 0 && + virProcessSetMaxMemLock(vm->pid, vm->def->mem.hard_limit << 10) < 0) { + return -1; + } + return qemuMonitorMigrateToHost(priv->mon, migrateFlags, + spec->dest.host.protocol, + spec->dest.host.name, + spec->dest.host.port); + + case MIGRATION_DEST_SOCKET: + qemuSecurityDomainSetPathLabel(driver, vm, spec->dest.socket.path, false); + return qemuMonitorMigrateToSocket(priv->mon, migrateFlags, + spec->dest.socket.path); + + case MIGRATION_DEST_CONNECT_HOST: + case MIGRATION_DEST_CONNECT_SOCKET: + /* handled above and transformed into MIGRATION_DEST_FD */ + break; + + case MIGRATION_DEST_FD: + if (spec->fwdType != MIGRATION_FWD_DIRECT) { + if (!tunnelFd) { + virReportError(VIR_ERR_INTERNAL_ERROR, "%s", + _("tunnelFD argument is required for tunnelled " + "migration")); + VIR_FORCE_CLOSE(spec->dest.fd.local); + VIR_FORCE_CLOSE(spec->dest.fd.qemu); + return -1; + } + *tunnelFd = spec->dest.fd.local; + spec->dest.fd.local = -1; + } + rc = qemuMonitorMigrateToFd(priv->mon, migrateFlags, + spec->dest.fd.qemu); + VIR_FORCE_CLOSE(spec->dest.fd.qemu); + return rc; + } + + virReportError(VIR_ERR_INTERNAL_ERROR, + _("unexpected migration schema: %d"), spec->destType); + return -1; +} + + static int qemuMigrationSrcRun(virQEMUDriver *driver, virDomainObj *vm, @@ -4448,7 +4518,6 @@ qemuMigrationSrcRun(virQEMUDriver *driver, bool cancel = false; unsigned int waitFlags; g_autoptr(virDomainDef) persistDef = NULL; - g_autofree char *timestamp = NULL; int rc; if (resource > 0) @@ -4624,52 +4693,7 @@ qemuMigrationSrcRun(virQEMUDriver *driver, qemuMonitorSetMigrationSpeed(priv->mon, priv->migMaxBandwidth) < 0) goto exit_monitor; - /* connect to the destination qemu if needed */ - if ((spec->destType == MIGRATION_DEST_CONNECT_HOST || - spec->destType == MIGRATION_DEST_CONNECT_SOCKET) && - qemuMigrationSrcConnect(driver, vm, spec) < 0) { - goto exit_monitor; - } - - /* log start of migration */ - if ((timestamp = virTimeStringNow()) != NULL) - qemuDomainLogAppendMessage(driver, vm, "%s: initiating migration\n", timestamp); - - rc = -1; - switch (spec->destType) { - case MIGRATION_DEST_HOST: - if (STREQ(spec->dest.host.protocol, "rdma") && - vm->def->mem.hard_limit > 0 && - virProcessSetMaxMemLock(vm->pid, vm->def->mem.hard_limit << 10) < 0) { - goto exit_monitor; - } - rc = qemuMonitorMigrateToHost(priv->mon, migrate_flags, - spec->dest.host.protocol, - spec->dest.host.name, - spec->dest.host.port); - break; - - case MIGRATION_DEST_SOCKET: - qemuSecurityDomainSetPathLabel(driver, vm, spec->dest.socket.path, false); - rc = qemuMonitorMigrateToSocket(priv->mon, migrate_flags, - spec->dest.socket.path); - break; - - case MIGRATION_DEST_CONNECT_HOST: - case MIGRATION_DEST_CONNECT_SOCKET: - /* handled above and transformed into MIGRATION_DEST_FD */ - break; - - case MIGRATION_DEST_FD: - if (spec->fwdType != MIGRATION_FWD_DIRECT) { - fd = spec->dest.fd.local; - spec->dest.fd.local = -1; - } - rc = qemuMonitorMigrateToFd(priv->mon, migrate_flags, - spec->dest.fd.qemu); - VIR_FORCE_CLOSE(spec->dest.fd.qemu); - break; - } + rc = qemuMigrationSrcStart(driver, vm, spec, migrate_flags, &fd); qemuDomainObjExitMonitor(vm); if (rc < 0) -- 2.35.1
On Tue, May 10, 2022 at 17:21:17 +0200, Jiri Denemark wrote:
qemuMigrationSrcRun does a lot of thing before and after telling QEMU to start the migration. Let's make the core reusable by moving it to a new qemuMigrationSrcStart function.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 118 +++++++++++++++++++++++--------------- 1 file changed, 71 insertions(+), 47 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index dd18a4ad63..e1c67c51ec 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4413,6 +4413,76 @@ qemuMigrationSrcRunPrepareBlockDirtyBitmaps(virDomainObj *vm, }
+/* The caller is supposed to enter monitor before calling this. */ +static int +qemuMigrationSrcStart(virQEMUDriver *driver,
driver should not be needed.
+ virDomainObj *vm, + qemuMigrationSpec *spec, + unsigned int migrateFlags, + int *tunnelFd) +{ + qemuDomainObjPrivate *priv = vm->privateData;
you can get it in priv->driver.
+ g_autofree char *timestamp = NULL; + int rc;
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 8 ++++++-- src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index e1c67c51ec..30c8dbd954 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4433,8 +4433,12 @@ qemuMigrationSrcStart(virQEMUDriver *driver, } /* log start of migration */ - if ((timestamp = virTimeStringNow()) != NULL) - qemuDomainLogAppendMessage(driver, vm, "%s: initiating migration\n", timestamp); + if ((timestamp = virTimeStringNow()) != NULL) { + if (migrateFlags & QEMU_MONITOR_MIGRATE_RESUME) + qemuDomainLogAppendMessage(driver, vm, "%s: resuming migration\n", timestamp); + else + qemuDomainLogAppendMessage(driver, vm, "%s: initiating migration\n", timestamp); + } switch (spec->destType) { case MIGRATION_DEST_HOST: diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 5949e21a39..38b65eb950 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -907,6 +907,7 @@ typedef enum { QEMU_MONITOR_MIGRATE_BACKGROUND = 1 << 0, QEMU_MONITOR_MIGRATE_NON_SHARED_DISK = 1 << 1, /* migration with non-shared storage with full disk copy */ QEMU_MONITOR_MIGRATE_NON_SHARED_INC = 1 << 2, /* migration with non-shared storage with incremental copy */ + QEMU_MONITOR_MIGRATE_RESUME = 1 << 3, /* resume failed post-copy migration */ QEMU_MONITOR_MIGRATION_FLAGS_LAST } QEMU_MONITOR_MIGRATE; diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index df3fdfe4fb..ea2e001483 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3419,10 +3419,12 @@ int qemuMonitorJSONMigrate(qemuMonitor *mon, bool detach = !!(flags & QEMU_MONITOR_MIGRATE_BACKGROUND); bool blk = !!(flags & QEMU_MONITOR_MIGRATE_NON_SHARED_DISK); bool inc = !!(flags & QEMU_MONITOR_MIGRATE_NON_SHARED_INC); + bool resume = !!(flags & QEMU_MONITOR_MIGRATE_RESUME); g_autoptr(virJSONValue) cmd = qemuMonitorJSONMakeCommand("migrate", "b:detach", detach, "b:blk", blk, "b:inc", inc, + "b:resume", resume, "s:uri", uri, NULL); g_autoptr(virJSONValue) reply = NULL; -- 2.35.1
On Tue, May 10, 2022 at 17:21:18 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 8 ++++++-- src/qemu/qemu_monitor.h | 1 + src/qemu/qemu_monitor_json.c | 2 ++ 3 files changed, 9 insertions(+), 2 deletions(-)
[...]
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index df3fdfe4fb..ea2e001483 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3419,10 +3419,12 @@ int qemuMonitorJSONMigrate(qemuMonitor *mon, bool detach = !!(flags & QEMU_MONITOR_MIGRATE_BACKGROUND); bool blk = !!(flags & QEMU_MONITOR_MIGRATE_NON_SHARED_DISK); bool inc = !!(flags & QEMU_MONITOR_MIGRATE_NON_SHARED_INC); + bool resume = !!(flags & QEMU_MONITOR_MIGRATE_RESUME); g_autoptr(virJSONValue) cmd = qemuMonitorJSONMakeCommand("migrate", "b:detach", detach, "b:blk", blk, "b:inc", inc, + "b:resume", resume,
Okay, resume is supported in qemu-3.1 already so we don't need "B:...". Reviewed-by: Peter Krempa <pkrempa@redhat.com>
It just calls migrate QMP command with resume=true without having to worry about migration capabilities or parameters, storage migration, etc. since everything has already been done in the normal Perform phase. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 110 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 106 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 30c8dbd954..29be797d78 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4857,6 +4857,51 @@ qemuMigrationSrcRun(virQEMUDriver *driver, goto error; } + +static int +qemuMigrationSrcResume(virQEMUDriver *driver, + virDomainObj *vm, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + qemuMigrationSpec *spec) +{ + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(qemuMigrationCookie) mig = NULL; + unsigned int migrateFlags = QEMU_MONITOR_MIGRATE_BACKGROUND | + QEMU_MONITOR_MIGRATE_RESUME; + int rc; + + VIR_DEBUG("vm=%p", vm); + + mig = qemuMigrationCookieParse(driver, vm->def, priv->origname, priv, + cookiein, cookieinlen, + QEMU_MIGRATION_COOKIE_CAPS); + if (!mig) + return -1; + + if (qemuDomainObjEnterMonitorAsync(driver, vm, + VIR_ASYNC_JOB_MIGRATION_OUT) < 0) + return -1; + + rc = qemuMigrationSrcStart(driver, vm, spec, migrateFlags, NULL); + + qemuDomainObjExitMonitor(vm); + if (rc < 0) + return -1; + + if (qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_SOURCE, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_STATS) < 0) { + VIR_WARN("Unable to encode migration cookie"); + } + + return 0; +} + + /* Perform migration using QEMU's native migrate support, * not encrypted obviously */ @@ -4946,10 +4991,16 @@ qemuMigrationSrcPerformNative(virQEMUDriver *driver, spec.fwdType = MIGRATION_FWD_DIRECT; - ret = qemuMigrationSrcRun(driver, vm, persist_xml, cookiein, cookieinlen, cookieout, - cookieoutlen, flags, resource, &spec, dconn, - graphicsuri, nmigrate_disks, migrate_disks, - migParams, nbdURI); + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + ret = qemuMigrationSrcResume(driver, vm, cookiein, cookieinlen, + cookieout, cookieoutlen, &spec); + } else { + ret = qemuMigrationSrcRun(driver, vm, persist_xml, cookiein, cookieinlen, + cookieout, cookieoutlen, flags, resource, + &spec, dconn, graphicsuri, + nmigrate_disks, migrate_disks, + migParams, nbdURI); + } if (spec.destType == MIGRATION_DEST_FD) VIR_FORCE_CLOSE(spec.dest.fd.qemu); @@ -5812,6 +5863,51 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, return ret; } + +static int +qemuMigrationSrcPerformResume(virQEMUDriver *driver, + virConnectPtr conn, + virDomainObj *vm, + const char *uri, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + unsigned long flags) +{ + int ret; + + VIR_DEBUG("vm=%p, uri=%s", vm, uri); + + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_OUT, flags, + QEMU_MIGRATION_PHASE_BEGIN_RESUME)) + return -1; + + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PERFORM_RESUME) < 0) + return -1; + + virCloseCallbacksUnset(driver->closeCallbacks, vm, + qemuMigrationSrcCleanup); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); + + ret = qemuMigrationSrcPerformNative(driver, vm, NULL, uri, + cookiein, cookieinlen, + cookieout, cookieoutlen, flags, + 0, NULL, NULL, 0, NULL, NULL, NULL); + + if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, + qemuMigrationSrcCleanup) < 0) + ret = -1; + + if (ret < 0) + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); + + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + return ret; +} + + /* * This implements perform phase of v3 migration protocol. */ @@ -5837,6 +5933,12 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, qemuDomainJobPrivate *jobPriv = priv->job.privateData; int ret = -1; + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + return qemuMigrationSrcPerformResume(driver, conn, vm, uri, + cookiein, cookieinlen, + cookieout, cookieoutlen, flags); + } + /* If we didn't start the job in the begin phase, start it now. */ if (!(flags & VIR_MIGRATE_CHANGE_PROTECTION)) { if (qemuMigrationJobStart(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, -- 2.35.1
On Tue, May 10, 2022 at 17:21:19 +0200, Jiri Denemark wrote:
It just calls migrate QMP command with resume=true without having to worry about migration capabilities or parameters, storage migration, etc. since everything has already been done in the normal Perform phase.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 110 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 106 insertions(+), 4 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 30c8dbd954..29be797d78 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -4857,6 +4857,51 @@ qemuMigrationSrcRun(virQEMUDriver *driver, goto error; }
+ +static int +qemuMigrationSrcResume(virQEMUDriver *driver,
'driver' is redundant.
+ virDomainObj *vm, + const char *cookiein,
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 49 +++++++++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 17 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 29be797d78..8ed1c2c2b6 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3812,15 +3812,19 @@ qemuMigrationSrcConfirmPhase(virQEMUDriver *driver, virCheckFlags(QEMU_MIGRATION_FLAGS, -1); - /* Keep the original migration phase in case post-copy failed as the job - * will stay active even though migration API finishes with an error. - */ - if (virDomainObjIsFailedPostcopy(vm)) + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + phase = QEMU_MIGRATION_PHASE_CONFIRM_RESUME; + } else if (virDomainObjIsFailedPostcopy(vm)) { + /* Keep the original migration phase in case post-copy failed as the + * job will stay active even though migration API finishes with an + * error. + */ phase = priv->job.phase; - else if (retcode == 0) + } else if (retcode == 0) { phase = QEMU_MIGRATION_PHASE_CONFIRM3; - else + } else { phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; + } if (qemuMigrationJobStartPhase(vm, phase) < 0) return -1; @@ -3895,24 +3899,35 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, qemuDomainObjPrivate *priv = vm->privateData; int ret = -1; - if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_OUT)) - goto cleanup; + VIR_DEBUG("vm=%p, flags=0x%x, cancelled=%d", vm, flags, cancelled); - /* Keep the original migration phase in case post-copy failed as the job - * will stay active even though migration API finishes with an error. - */ - if (virDomainObjIsFailedPostcopy(vm)) - phase = priv->job.phase; - else if (cancelled) - phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; - else - phase = QEMU_MIGRATION_PHASE_CONFIRM3; + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_OUT, flags, + QEMU_MIGRATION_PHASE_PERFORM_RESUME)) + goto cleanup; + phase = QEMU_MIGRATION_PHASE_CONFIRM_RESUME; + } else { + if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_OUT)) + goto cleanup; + + /* Keep the original migration phase in case post-copy failed as the + * job will stay active even though migration API finishes with an + * error. + */ + if (virDomainObjIsFailedPostcopy(vm)) + phase = priv->job.phase; + else if (cancelled) + phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; + else + phase = QEMU_MIGRATION_PHASE_CONFIRM3; + } if (qemuMigrationJobStartPhase(vm, phase) < 0) goto cleanup; virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); ret = qemuMigrationSrcConfirmPhase(driver, vm, cookiein, cookieinlen, -- 2.35.1
On Tue, May 10, 2022 at 17:21:20 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 49 +++++++++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 17 deletions(-)
[...]
if (qemuMigrationJobStartPhase(vm, phase) < 0) goto cleanup;
virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob);
This looks out of place in context of this commit. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Thu, May 12, 2022 at 16:27:24 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:20 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 49 +++++++++++++++++++++++++-------------- 1 file changed, 32 insertions(+), 17 deletions(-)
[...]
if (qemuMigrationJobStartPhase(vm, phase) < 0) goto cleanup;
virCloseCallbacksUnset(driver->closeCallbacks, vm, qemuMigrationSrcCleanup); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob);
This looks out of place in context of this commit.
Sigh. I was trying hard to remember the reason I put this change into this patch because there must have been one... but it was just a mistake and the change belongs to [14/80] qemu: Keep migration job active after failed post-copy. Jirka
Moves most of the code from qemuMigrationDstPrepareAny to a new qemuMigrationDstPrepareFresh so that qemuMigrationDstPrepareAny can be shared with post-copy resume. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 165 ++++++++++++++++++++++++-------------- 1 file changed, 103 insertions(+), 62 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 8ed1c2c2b6..f1e3774034 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3119,27 +3119,26 @@ qemuMigrationDstPrepareAnyBlockDirtyBitmaps(virDomainObj *vm, static int -qemuMigrationDstPrepareAny(virQEMUDriver *driver, - virConnectPtr dconn, - const char *cookiein, - int cookieinlen, - char **cookieout, - int *cookieoutlen, - virDomainDef **def, - const char *origname, - virStreamPtr st, - const char *protocol, - unsigned short port, - bool autoPort, - const char *listenAddress, - size_t nmigrate_disks, - const char **migrate_disks, - int nbdPort, - const char *nbdURI, - qemuMigrationParams *migParams, - unsigned long flags) +qemuMigrationDstPrepareFresh(virQEMUDriver *driver, + virConnectPtr dconn, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + virDomainDef **def, + const char *origname, + virStreamPtr st, + const char *protocol, + unsigned short port, + bool autoPort, + const char *listenAddress, + size_t nmigrate_disks, + const char **migrate_disks, + int nbdPort, + const char *nbdURI, + qemuMigrationParams *migParams, + unsigned long flags) { - g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver); virDomainObj *vm = NULL; virObjectEvent *event = NULL; virErrorPtr origErr; @@ -3159,55 +3158,18 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, int rv; g_autofree char *tlsAlias = NULL; + VIR_DEBUG("name=%s, origname=%s, protocol=%s, port=%hu, " + "listenAddress=%s, nbdPort=%d, nbdURI=%s, flags=0x%lx", + (*def)->name, NULLSTR(origname), protocol, port, + listenAddress, nbdPort, NULLSTR(nbdURI), flags); + if (flags & VIR_MIGRATE_OFFLINE) { - if (flags & (VIR_MIGRATE_NON_SHARED_DISK | - VIR_MIGRATE_NON_SHARED_INC)) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("offline migration cannot handle " - "non-shared storage")); - goto cleanup; - } - if (!(flags & VIR_MIGRATE_PERSIST_DEST)) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("offline migration must be specified with " - "the persistent flag set")); - goto cleanup; - } - if (tunnel) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("tunnelled offline migration does not " - "make sense")); - goto cleanup; - } cookieFlags = 0; } else { cookieFlags = QEMU_MIGRATION_COOKIE_GRAPHICS | QEMU_MIGRATION_COOKIE_CAPS; } - if (flags & VIR_MIGRATE_POSTCOPY && - (!(flags & VIR_MIGRATE_LIVE) || - flags & VIR_MIGRATE_PAUSED)) { - virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", - _("post-copy migration is not supported with non-live " - "or paused migration")); - goto cleanup; - } - - if (flags & VIR_MIGRATE_POSTCOPY && flags & VIR_MIGRATE_TUNNELLED) { - virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", - _("post-copy is not supported with tunnelled migration")); - goto cleanup; - } - - if (cfg->migrateTLSForce && - !(flags & VIR_MIGRATE_TUNNELLED) && - !(flags & VIR_MIGRATE_TLS)) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("this libvirtd instance allows migration only with VIR_MIGRATE_TLS flag")); - goto cleanup; - } - if (!qemuMigrationSrcIsAllowedHostdev(*def)) goto cleanup; @@ -3499,6 +3461,85 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, } +static int +qemuMigrationDstPrepareAny(virQEMUDriver *driver, + virConnectPtr dconn, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + virDomainDef **def, + const char *origname, + virStreamPtr st, + const char *protocol, + unsigned short port, + bool autoPort, + const char *listenAddress, + size_t nmigrate_disks, + const char **migrate_disks, + int nbdPort, + const char *nbdURI, + qemuMigrationParams *migParams, + unsigned long flags) +{ + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver); + + if (flags & VIR_MIGRATE_OFFLINE) { + if (flags & (VIR_MIGRATE_NON_SHARED_DISK | + VIR_MIGRATE_NON_SHARED_INC)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration cannot handle " + "non-shared storage")); + return -1; + } + if (!(flags & VIR_MIGRATE_PERSIST_DEST)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration must be specified with " + "the persistent flag set")); + return -1; + } + if (st) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("tunnelled offline migration does not " + "make sense")); + return -1; + } + } + + if (flags & VIR_MIGRATE_POSTCOPY && + (!(flags & VIR_MIGRATE_LIVE) || + flags & VIR_MIGRATE_PAUSED)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("post-copy migration is not supported with non-live " + "or paused migration")); + return -1; + } + + if (flags & VIR_MIGRATE_POSTCOPY && flags & VIR_MIGRATE_TUNNELLED) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("post-copy is not supported with tunnelled migration")); + return -1; + } + + if (cfg->migrateTLSForce && + !(flags & VIR_MIGRATE_TUNNELLED) && + !(flags & VIR_MIGRATE_TLS)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("this libvirtd instance allows migration only with VIR_MIGRATE_TLS flag")); + return -1; + } + + return qemuMigrationDstPrepareFresh(driver, dconn, + cookiein, cookieinlen, + cookieout, cookieoutlen, + def, origname, st, protocol, + port, autoPort, listenAddress, + nmigrate_disks, migrate_disks, + nbdPort, nbdURI, + migParams, flags); +} + + /* * This version starts an empty VM listening on a localhost TCP port, and * sets up the corresponding virStream to handle the incoming data. -- 2.35.1
On Tue, May 10, 2022 at 17:21:21 +0200, Jiri Denemark wrote:
Moves most of the code from qemuMigrationDstPrepareAny to a new qemuMigrationDstPrepareFresh so that qemuMigrationDstPrepareAny can be shared with post-copy resume.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 165 ++++++++++++++++++++++++-------------- 1 file changed, 103 insertions(+), 62 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 8ed1c2c2b6..f1e3774034 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c
[...]
@@ -3499,6 +3461,85 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, }
+static int +qemuMigrationDstPrepareAny(virQEMUDriver *driver, + virConnectPtr dconn, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + virDomainDef **def, + const char *origname, + virStreamPtr st, + const char *protocol, + unsigned short port, + bool autoPort, + const char *listenAddress, + size_t nmigrate_disks, + const char **migrate_disks, + int nbdPort, + const char *nbdURI, + qemuMigrationParams *migParams, + unsigned long flags) +{ + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(driver); + + if (flags & VIR_MIGRATE_OFFLINE) { + if (flags & (VIR_MIGRATE_NON_SHARED_DISK | + VIR_MIGRATE_NON_SHARED_INC)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration cannot handle " + "non-shared storage")); + return -1; + } + if (!(flags & VIR_MIGRATE_PERSIST_DEST)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("offline migration must be specified with " + "the persistent flag set")); + return -1; + } + if (st) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("tunnelled offline migration does not " + "make sense")); + return -1; + } + } + + if (flags & VIR_MIGRATE_POSTCOPY && + (!(flags & VIR_MIGRATE_LIVE) || + flags & VIR_MIGRATE_PAUSED)) { + virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", + _("post-copy migration is not supported with non-live " + "or paused migration")); + return -1; + }
Please remove linebreaks in the error messages in moved code. Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Offline migration jumps over a big part of qemuMigrationDstPrepareFresh. Let's move that part into a new qemuMigrationDstPrepareActive function to make the code easier to follow. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 374 +++++++++++++++++++++----------------- 1 file changed, 206 insertions(+), 168 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index f1e3774034..dc608fb8a4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3118,6 +3118,200 @@ qemuMigrationDstPrepareAnyBlockDirtyBitmaps(virDomainObj *vm, } +static int +qemuMigrationDstPrepareActive(virQEMUDriver *driver, + virDomainObj *vm, + virConnectPtr dconn, + qemuMigrationCookie *mig, + virStreamPtr st, + const char *protocol, + unsigned short port, + const char *listenAddress, + size_t nmigrate_disks, + const char **migrate_disks, + int nbdPort, + const char *nbdURI, + qemuMigrationParams *migParams, + unsigned long flags) +{ + qemuDomainObjPrivate *priv = vm->privateData; + qemuDomainJobPrivate *jobPriv = priv->job.privateData; + qemuProcessIncomingDef *incoming = NULL; + g_autofree char *tlsAlias = NULL; + virObjectEvent *event = NULL; + virErrorPtr origErr = NULL; + int dataFD[2] = { -1, -1 }; + bool stopProcess = false; + unsigned int startFlags; + bool relabel = false; + bool tunnel = !!st; + int ret = -1; + int rv; + + if (STREQ_NULLABLE(protocol, "rdma") && + !virMemoryLimitIsSet(vm->def->mem.hard_limit)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("cannot start RDMA migration with no memory hard " + "limit set")); + goto error; + } + + if (qemuMigrationDstPrecreateStorage(vm, mig->nbd, + nmigrate_disks, migrate_disks, + !!(flags & VIR_MIGRATE_NON_SHARED_INC)) < 0) + goto error; + + if (tunnel && + virPipe(dataFD) < 0) + goto error; + + startFlags = VIR_QEMU_PROCESS_START_AUTODESTROY; + + if (qemuProcessInit(driver, vm, mig->cpu, VIR_ASYNC_JOB_MIGRATION_IN, + true, startFlags) < 0) + goto error; + stopProcess = true; + + if (!(incoming = qemuMigrationDstPrepare(vm, tunnel, protocol, + listenAddress, port, + dataFD[0]))) + goto error; + + if (qemuProcessPrepareDomain(driver, vm, startFlags) < 0) + goto error; + + if (qemuProcessPrepareHost(driver, vm, startFlags) < 0) + goto error; + + rv = qemuProcessLaunch(dconn, driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + incoming, NULL, + VIR_NETDEV_VPORT_PROFILE_OP_MIGRATE_IN_START, + startFlags); + if (rv < 0) { + if (rv == -2) + relabel = true; + goto error; + } + relabel = true; + + if (tunnel) { + if (virFDStreamOpen(st, dataFD[1]) < 0) { + virReportSystemError(errno, "%s", + _("cannot pass pipe for tunnelled migration")); + goto error; + } + dataFD[1] = -1; /* 'st' owns the FD now & will close it */ + } + + if (STREQ_NULLABLE(protocol, "rdma") && + vm->def->mem.hard_limit > 0 && + virProcessSetMaxMemLock(vm->pid, vm->def->mem.hard_limit << 10) < 0) { + goto error; + } + + if (qemuMigrationDstPrepareAnyBlockDirtyBitmaps(vm, mig, migParams, flags) < 0) + goto error; + + if (qemuMigrationParamsCheck(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + migParams, mig->caps->automatic) < 0) + goto error; + + /* Migrations using TLS need to add the "tls-creds-x509" object and + * set the migration TLS parameters */ + if (flags & VIR_MIGRATE_TLS) { + if (qemuMigrationParamsEnableTLS(driver, vm, true, + VIR_ASYNC_JOB_MIGRATION_IN, + &tlsAlias, NULL, + migParams) < 0) + goto error; + } else { + if (qemuMigrationParamsDisableTLS(vm, migParams) < 0) + goto error; + } + + if (qemuMigrationParamsApply(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + migParams) < 0) + goto error; + + if (mig->nbd && + flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC) && + virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_SERVER)) { + const char *nbdTLSAlias = NULL; + + if (flags & VIR_MIGRATE_TLS) { + if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_TLS)) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s", + _("QEMU NBD server does not support TLS transport")); + goto error; + } + + nbdTLSAlias = tlsAlias; + } + + if (qemuMigrationDstStartNBDServer(driver, vm, incoming->address, + nmigrate_disks, migrate_disks, + nbdPort, nbdURI, + nbdTLSAlias) < 0) { + goto error; + } + } + + if (mig->lockState) { + VIR_DEBUG("Received lockstate %s", mig->lockState); + VIR_FREE(priv->lockState); + priv->lockState = g_steal_pointer(&mig->lockState); + } else { + VIR_DEBUG("Received no lockstate"); + } + + if (qemuMigrationDstRun(driver, vm, incoming->uri, + VIR_ASYNC_JOB_MIGRATION_IN) < 0) + goto error; + + if (qemuProcessFinishStartup(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + false, VIR_DOMAIN_PAUSED_MIGRATION) < 0) + goto error; + + if (!(flags & VIR_MIGRATE_OFFLINE)) { + virDomainAuditStart(vm, "migrated", true); + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_STARTED, + VIR_DOMAIN_EVENT_STARTED_MIGRATED); + } + + ret = 0; + + cleanup: + qemuProcessIncomingDefFree(incoming); + VIR_FORCE_CLOSE(dataFD[0]); + VIR_FORCE_CLOSE(dataFD[1]); + virObjectEventStateQueue(driver->domainEventState, event); + virErrorRestore(&origErr); + return ret; + + error: + virErrorPreserveLast(&origErr); + qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + jobPriv->migParams, priv->job.apiFlags); + + if (stopProcess) { + unsigned int stopFlags = VIR_QEMU_PROCESS_STOP_MIGRATED; + if (!relabel) + stopFlags |= VIR_QEMU_PROCESS_STOP_NO_RELABEL; + virDomainAuditStart(vm, "migrated", false); + qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, + VIR_ASYNC_JOB_MIGRATION_IN, stopFlags); + /* release if port is auto selected which is not the case if + * it is given in parameters + */ + if (nbdPort == 0) + virPortAllocatorRelease(priv->nbdPort); + priv->nbdPort = 0; + } + goto cleanup; +} + + static int qemuMigrationDstPrepareFresh(virQEMUDriver *driver, virConnectPtr dconn, @@ -3140,32 +3334,20 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, unsigned long flags) { virDomainObj *vm = NULL; - virObjectEvent *event = NULL; virErrorPtr origErr; int ret = -1; - int dataFD[2] = { -1, -1 }; qemuDomainObjPrivate *priv = NULL; qemuMigrationCookie *mig = NULL; - qemuDomainJobPrivate *jobPriv = NULL; - bool tunnel = !!st; g_autofree char *xmlout = NULL; - unsigned int cookieFlags; - unsigned int startFlags; - qemuProcessIncomingDef *incoming = NULL; + unsigned int cookieFlags = 0; bool taint_hook = false; - bool stopProcess = false; - bool relabel = false; - int rv; - g_autofree char *tlsAlias = NULL; VIR_DEBUG("name=%s, origname=%s, protocol=%s, port=%hu, " "listenAddress=%s, nbdPort=%d, nbdURI=%s, flags=0x%lx", (*def)->name, NULLSTR(origname), protocol, port, listenAddress, nbdPort, NULLSTR(nbdURI), flags); - if (flags & VIR_MIGRATE_OFFLINE) { - cookieFlags = 0; - } else { + if (!(flags & VIR_MIGRATE_OFFLINE)) { cookieFlags = QEMU_MIGRATION_COOKIE_GRAPHICS | QEMU_MIGRATION_COOKIE_CAPS; } @@ -3238,7 +3420,6 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, goto cleanup; priv = vm->privateData; - jobPriv = priv->job.privateData; priv->origname = g_strdup(origname); if (taint_hook) { @@ -3246,19 +3427,6 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, priv->hookRun = true; } - if (STREQ_NULLABLE(protocol, "rdma") && - !virMemoryLimitIsSet(vm->def->mem.hard_limit)) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("cannot start RDMA migration with no memory hard " - "limit set")); - goto cleanup; - } - - if (qemuMigrationDstPrecreateStorage(vm, mig->nbd, - nmigrate_disks, migrate_disks, - !!(flags & VIR_MIGRATE_NON_SHARED_INC)) < 0) - goto cleanup; - if (qemuMigrationJobStart(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, flags) < 0) goto cleanup; @@ -3269,122 +3437,21 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, /* Domain starts inactive, even if the domain XML had an id field. */ vm->def->id = -1; - if (flags & VIR_MIGRATE_OFFLINE) - goto done; - - if (tunnel && - virPipe(dataFD) < 0) - goto stopjob; - - startFlags = VIR_QEMU_PROCESS_START_AUTODESTROY; - - if (qemuProcessInit(driver, vm, mig->cpu, VIR_ASYNC_JOB_MIGRATION_IN, - true, startFlags) < 0) - goto stopjob; - stopProcess = true; - - if (!(incoming = qemuMigrationDstPrepare(vm, tunnel, protocol, - listenAddress, port, - dataFD[0]))) - goto stopjob; - - if (qemuProcessPrepareDomain(driver, vm, startFlags) < 0) - goto stopjob; - - if (qemuProcessPrepareHost(driver, vm, startFlags) < 0) - goto stopjob; - - rv = qemuProcessLaunch(dconn, driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, - incoming, NULL, - VIR_NETDEV_VPORT_PROFILE_OP_MIGRATE_IN_START, - startFlags); - if (rv < 0) { - if (rv == -2) - relabel = true; - goto stopjob; - } - relabel = true; - - if (tunnel) { - if (virFDStreamOpen(st, dataFD[1]) < 0) { - virReportSystemError(errno, "%s", - _("cannot pass pipe for tunnelled migration")); - goto stopjob; - } - dataFD[1] = -1; /* 'st' owns the FD now & will close it */ - } - - if (STREQ_NULLABLE(protocol, "rdma") && - vm->def->mem.hard_limit > 0 && - virProcessSetMaxMemLock(vm->pid, vm->def->mem.hard_limit << 10) < 0) { - goto stopjob; - } - - if (qemuMigrationDstPrepareAnyBlockDirtyBitmaps(vm, mig, migParams, flags) < 0) - goto stopjob; - - if (qemuMigrationParamsCheck(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, - migParams, mig->caps->automatic) < 0) - goto stopjob; - - /* Migrations using TLS need to add the "tls-creds-x509" object and - * set the migration TLS parameters */ - if (flags & VIR_MIGRATE_TLS) { - if (qemuMigrationParamsEnableTLS(driver, vm, true, - VIR_ASYNC_JOB_MIGRATION_IN, - &tlsAlias, NULL, - migParams) < 0) - goto stopjob; - } else { - if (qemuMigrationParamsDisableTLS(vm, migParams) < 0) - goto stopjob; - } - - if (qemuMigrationParamsApply(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, - migParams) < 0) - goto stopjob; - - if (mig->nbd && - flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC) && - virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_SERVER)) { - const char *nbdTLSAlias = NULL; - - if (flags & VIR_MIGRATE_TLS) { - if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_TLS)) { - virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s", - _("QEMU NBD server does not support TLS transport")); - goto stopjob; - } - - nbdTLSAlias = tlsAlias; - } - - if (qemuMigrationDstStartNBDServer(driver, vm, incoming->address, - nmigrate_disks, migrate_disks, - nbdPort, nbdURI, - nbdTLSAlias) < 0) { + if (!(flags & VIR_MIGRATE_OFFLINE)) { + if (qemuMigrationDstPrepareActive(driver, vm, dconn, mig, st, + protocol, port, listenAddress, + nmigrate_disks, migrate_disks, + nbdPort, nbdURI, + migParams, flags) < 0) { goto stopjob; } - cookieFlags |= QEMU_MIGRATION_COOKIE_NBD; - } - if (mig->lockState) { - VIR_DEBUG("Received lockstate %s", mig->lockState); - VIR_FREE(priv->lockState); - priv->lockState = g_steal_pointer(&mig->lockState); - } else { - VIR_DEBUG("Received no lockstate"); + if (mig->nbd && + flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC) && + virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_SERVER)) + cookieFlags |= QEMU_MIGRATION_COOKIE_NBD; } - if (qemuMigrationDstRun(driver, vm, incoming->uri, - VIR_ASYNC_JOB_MIGRATION_IN) < 0) - goto stopjob; - - if (qemuProcessFinishStartup(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, - false, VIR_DOMAIN_PAUSED_MIGRATION) < 0) - goto stopjob; - - done: if (qemuMigrationCookieFormat(mig, driver, vm, QEMU_MIGRATION_DESTINATION, cookieout, cookieoutlen, cookieFlags) < 0) { @@ -3397,13 +3464,6 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, qemuDomainCleanupAdd(vm, qemuMigrationDstPrepareCleanup); - if (!(flags & VIR_MIGRATE_OFFLINE)) { - virDomainAuditStart(vm, "migrated", true); - event = virDomainEventLifecycleNewFromObj(vm, - VIR_DOMAIN_EVENT_STARTED, - VIR_DOMAIN_EVENT_STARTED_MIGRATED); - } - /* We keep the job active across API calls until the finish() call. * This prevents any other APIs being invoked while incoming * migration is taking place. @@ -3421,41 +3481,19 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, cleanup: virErrorPreserveLast(&origErr); - qemuProcessIncomingDefFree(incoming); - VIR_FORCE_CLOSE(dataFD[0]); - VIR_FORCE_CLOSE(dataFD[1]); if (ret < 0 && priv) { /* priv is set right after vm is added to the list of domains * and there is no 'goto cleanup;' in the middle of those */ VIR_FREE(priv->origname); - /* release if port is auto selected which is not the case if - * it is given in parameters - */ - if (nbdPort == 0) - virPortAllocatorRelease(priv->nbdPort); - priv->nbdPort = 0; virDomainObjRemoveTransientDef(vm); qemuDomainRemoveInactive(driver, vm); } virDomainObjEndAPI(&vm); - virObjectEventStateQueue(driver->domainEventState, event); qemuMigrationCookieFree(mig); virErrorRestore(&origErr); return ret; stopjob: - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, - jobPriv->migParams, priv->job.apiFlags); - - if (stopProcess) { - unsigned int stopFlags = VIR_QEMU_PROCESS_STOP_MIGRATED; - if (!relabel) - stopFlags |= VIR_QEMU_PROCESS_STOP_NO_RELABEL; - virDomainAuditStart(vm, "migrated", false); - qemuProcessStop(driver, vm, VIR_DOMAIN_SHUTOFF_FAILED, - VIR_ASYNC_JOB_MIGRATION_IN, stopFlags); - } - qemuMigrationJobFinish(vm); goto cleanup; } -- 2.35.1
On Tue, May 10, 2022 at 17:21:22 +0200, Jiri Denemark wrote:
Offline migration jumps over a big part of qemuMigrationDstPrepareFresh. Let's move that part into a new qemuMigrationDstPrepareActive function to make the code easier to follow.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 374 +++++++++++++++++++++----------------- 1 file changed, 206 insertions(+), 168 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index f1e3774034..dc608fb8a4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3118,6 +3118,200 @@ qemuMigrationDstPrepareAnyBlockDirtyBitmaps(virDomainObj *vm, }
+static int +qemuMigrationDstPrepareActive(virQEMUDriver *driver, + virDomainObj *vm, + virConnectPtr dconn, + qemuMigrationCookie *mig, + virStreamPtr st, + const char *protocol, + unsigned short port, + const char *listenAddress, + size_t nmigrate_disks, + const char **migrate_disks, + int nbdPort, + const char *nbdURI, + qemuMigrationParams *migParams, + unsigned long flags) +{ + qemuDomainObjPrivate *priv = vm->privateData; + qemuDomainJobPrivate *jobPriv = priv->job.privateData; + qemuProcessIncomingDef *incoming = NULL; + g_autofree char *tlsAlias = NULL; + virObjectEvent *event = NULL; + virErrorPtr origErr = NULL; + int dataFD[2] = { -1, -1 }; + bool stopProcess = false; + unsigned int startFlags; + bool relabel = false; + bool tunnel = !!st; + int ret = -1; + int rv; + + if (STREQ_NULLABLE(protocol, "rdma") && + !virMemoryLimitIsSet(vm->def->mem.hard_limit)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("cannot start RDMA migration with no memory hard " + "limit set"));
linebreaks
+ goto error; + } + + if (qemuMigrationDstPrecreateStorage(vm, mig->nbd, + nmigrate_disks, migrate_disks, + !!(flags & VIR_MIGRATE_NON_SHARED_INC)) < 0) + goto error; + + if (tunnel && + virPipe(dataFD) < 0) + goto error; + + startFlags = VIR_QEMU_PROCESS_START_AUTODESTROY; + + if (qemuProcessInit(driver, vm, mig->cpu, VIR_ASYNC_JOB_MIGRATION_IN, + true, startFlags) < 0) + goto error; + stopProcess = true; + + if (!(incoming = qemuMigrationDstPrepare(vm, tunnel, protocol, + listenAddress, port, + dataFD[0]))) + goto error; + + if (qemuProcessPrepareDomain(driver, vm, startFlags) < 0) + goto error; + + if (qemuProcessPrepareHost(driver, vm, startFlags) < 0) + goto error; + + rv = qemuProcessLaunch(dconn, driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + incoming, NULL, + VIR_NETDEV_VPORT_PROFILE_OP_MIGRATE_IN_START, + startFlags); + if (rv < 0) { + if (rv == -2) + relabel = true; + goto error; + } + relabel = true; + + if (tunnel) { + if (virFDStreamOpen(st, dataFD[1]) < 0) { + virReportSystemError(errno, "%s", + _("cannot pass pipe for tunnelled migration")); + goto error; + } + dataFD[1] = -1; /* 'st' owns the FD now & will close it */ + } + + if (STREQ_NULLABLE(protocol, "rdma") && + vm->def->mem.hard_limit > 0 && + virProcessSetMaxMemLock(vm->pid, vm->def->mem.hard_limit << 10) < 0) { + goto error; + } + + if (qemuMigrationDstPrepareAnyBlockDirtyBitmaps(vm, mig, migParams, flags) < 0) + goto error; + + if (qemuMigrationParamsCheck(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + migParams, mig->caps->automatic) < 0) + goto error; + + /* Migrations using TLS need to add the "tls-creds-x509" object and + * set the migration TLS parameters */ + if (flags & VIR_MIGRATE_TLS) { + if (qemuMigrationParamsEnableTLS(driver, vm, true, + VIR_ASYNC_JOB_MIGRATION_IN, + &tlsAlias, NULL, + migParams) < 0) + goto error; + } else { + if (qemuMigrationParamsDisableTLS(vm, migParams) < 0) + goto error; + } + + if (qemuMigrationParamsApply(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + migParams) < 0) + goto error; + + if (mig->nbd && + flags & (VIR_MIGRATE_NON_SHARED_DISK | VIR_MIGRATE_NON_SHARED_INC) && + virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_SERVER)) { + const char *nbdTLSAlias = NULL; + + if (flags & VIR_MIGRATE_TLS) { + if (!virQEMUCapsGet(priv->qemuCaps, QEMU_CAPS_NBD_TLS)) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s", + _("QEMU NBD server does not support TLS transport")); + goto error; + } + + nbdTLSAlias = tlsAlias; + } + + if (qemuMigrationDstStartNBDServer(driver, vm, incoming->address, + nmigrate_disks, migrate_disks, + nbdPort, nbdURI, + nbdTLSAlias) < 0) { + goto error; + } + } + + if (mig->lockState) { + VIR_DEBUG("Received lockstate %s", mig->lockState); + VIR_FREE(priv->lockState); + priv->lockState = g_steal_pointer(&mig->lockState); + } else { + VIR_DEBUG("Received no lockstate"); + } + + if (qemuMigrationDstRun(driver, vm, incoming->uri, + VIR_ASYNC_JOB_MIGRATION_IN) < 0) + goto error; + + if (qemuProcessFinishStartup(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + false, VIR_DOMAIN_PAUSED_MIGRATION) < 0) + goto error; + + if (!(flags & VIR_MIGRATE_OFFLINE)) { + virDomainAuditStart(vm, "migrated", true); + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_STARTED, + VIR_DOMAIN_EVENT_STARTED_MIGRATED);
Alignment.
+ } + + ret = 0; +
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Tue, May 10, 2022 at 05:21:22PM +0200, Jiri Denemark wrote:
Offline migration jumps over a big part of qemuMigrationDstPrepareFresh. Let's move that part into a new qemuMigrationDstPrepareActive function to make the code easier to follow.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 374 +++++++++++++++++++++----------------- 1 file changed, 206 insertions(+), 168 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index f1e3774034..dc608fb8a4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3118,6 +3118,200 @@ qemuMigrationDstPrepareAnyBlockDirtyBitmaps(virDomainObj *vm, }
+static int +qemuMigrationDstPrepareActive(virQEMUDriver *driver, + virDomainObj *vm, + virConnectPtr dconn, + qemuMigrationCookie *mig, + virStreamPtr st, + const char *protocol, + unsigned short port, + const char *listenAddress, + size_t nmigrate_disks, + const char **migrate_disks, + int nbdPort, + const char *nbdURI, + qemuMigrationParams *migParams, + unsigned long flags) +{
[...]
+ if (qemuProcessFinishStartup(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + false, VIR_DOMAIN_PAUSED_MIGRATION) < 0) + goto error; + + if (!(flags & VIR_MIGRATE_OFFLINE)) {
No need for this check because the function call is guarded by the same check.
+ virDomainAuditStart(vm, "migrated", true); + event = virDomainEventLifecycleNewFromObj(vm, + VIR_DOMAIN_EVENT_STARTED, + VIR_DOMAIN_EVENT_STARTED_MIGRATED); + } + + ret = 0; +
[...]
@@ -3140,32 +3334,20 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, unsigned long flags) {
[...]
+ if (!(flags & VIR_MIGRATE_OFFLINE)) { + if (qemuMigrationDstPrepareActive(driver, vm, dconn, mig, st, + protocol, port, listenAddress, + nmigrate_disks, migrate_disks, + nbdPort, nbdURI, + migParams, flags) < 0) {
Since most of the cleanup code was moved out of this function, we don't need to preserve the original error here. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index dc608fb8a4..07b8c99343 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3334,10 +3334,9 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, unsigned long flags) { virDomainObj *vm = NULL; - virErrorPtr origErr; int ret = -1; qemuDomainObjPrivate *priv = NULL; - qemuMigrationCookie *mig = NULL; + g_autoptr(qemuMigrationCookie) mig = NULL; g_autofree char *xmlout = NULL; unsigned int cookieFlags = 0; bool taint_hook = false; @@ -3480,7 +3479,6 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, ret = 0; cleanup: - virErrorPreserveLast(&origErr); if (ret < 0 && priv) { /* priv is set right after vm is added to the list of domains * and there is no 'goto cleanup;' in the middle of those */ @@ -3489,8 +3487,6 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, qemuDomainRemoveInactive(driver, vm); } virDomainObjEndAPI(&vm); - qemuMigrationCookieFree(mig); - virErrorRestore(&origErr); return ret; stopjob: -- 2.35.1
On Tue, May 10, 2022 at 17:21:23 +0200, Jiri Denemark wrote:
Since most of the cleanup code was moved out of this function, we don't need to preserve the original error here.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
[...]
@@ -3480,7 +3479,6 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, ret = 0;
cleanup: - virErrorPreserveLast(&origErr); if (ret < 0 && priv) { /* priv is set right after vm is added to the list of domains * and there is no 'goto cleanup;' in the middle of those */ @@ -3489,8 +3487,6 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, qemuDomainRemoveInactive(driver, vm);
I'm not sure how much we care, but qemuDomainRemoveInactive has theoretical possibility to overwrite the error in at least the snapshot/checkpoint removal code and also via qemuExtDevicesCleanupHost.
} virDomainObjEndAPI(&vm); - qemuMigrationCookieFree(mig); - virErrorRestore(&origErr); return ret;
If you add justification to the commit message outlining why it's okay not to preserve the error or drop that part: Reviewed-by: Peter Krempa <pkrempa@redhat.com>
This command tells QEMU to start listening for an incoming post-copy recovery connection. Just like migrate-incoming is used for starting fresh migration on the destination host. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_monitor.c | 12 ++++++++++++ src/qemu/qemu_monitor.h | 4 ++++ src/qemu/qemu_monitor_json.c | 19 +++++++++++++++++++ src/qemu/qemu_monitor_json.h | 4 ++++ 4 files changed, 39 insertions(+) diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 589fef4385..5424fdc4fd 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -4568,3 +4568,15 @@ qemuMonitorChangeMemoryRequestedSize(qemuMonitor *mon, return qemuMonitorJSONChangeMemoryRequestedSize(mon, alias, requestedsize); } + + +int +qemuMonitorMigrateRecover(qemuMonitor *mon, + const char *uri) +{ + VIR_DEBUG("uri=%s", uri); + + QEMU_CHECK_MONITOR(mon); + + return qemuMonitorJSONMigrateRecover(mon, uri); +} diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 38b65eb950..812f8f4dc2 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -1592,3 +1592,7 @@ int qemuMonitorChangeMemoryRequestedSize(qemuMonitor *mon, const char *alias, unsigned long long requestedsize); + +int +qemuMonitorMigrateRecover(qemuMonitor *mon, + const char *uri); diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index ea2e001483..44361ae003 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -8988,3 +8988,22 @@ qemuMonitorJSONChangeMemoryRequestedSize(qemuMonitor *mon, return qemuMonitorJSONSetObjectProperty(mon, path, "requested-size", &prop); } + + +int +qemuMonitorJSONMigrateRecover(qemuMonitor *mon, + const char *uri) +{ + g_autoptr(virJSONValue) cmd = NULL; + g_autoptr(virJSONValue) reply = NULL; + + if (!(cmd = qemuMonitorJSONMakeCommand("migrate-recover", + "s:uri", uri, + NULL))) + return -1; + + if (qemuMonitorJSONCommand(mon, cmd, &reply) < 0) + return -1; + + return qemuMonitorJSONCheckError(cmd, reply); +} diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h index afd100f653..a60c0bfac4 100644 --- a/src/qemu/qemu_monitor_json.h +++ b/src/qemu/qemu_monitor_json.h @@ -874,3 +874,7 @@ int qemuMonitorJSONChangeMemoryRequestedSize(qemuMonitor *mon, const char *alias, unsigned long long requestedsize); + +int +qemuMonitorJSONMigrateRecover(qemuMonitor *mon, + const char *uri); -- 2.35.1
On Tue, May 10, 2022 at 17:21:24 +0200, Jiri Denemark wrote:
This command tells QEMU to start listening for an incoming post-copy recovery connection. Just like migrate-incoming is used for starting fresh migration on the destination host.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_monitor.c | 12 ++++++++++++ src/qemu/qemu_monitor.h | 4 ++++ src/qemu/qemu_monitor_json.c | 19 +++++++++++++++++++ src/qemu/qemu_monitor_json.h | 4 ++++ 4 files changed, 39 insertions(+)
With a qemumonitorjsontest case added: Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The function is now called qemuMigrationAnyConnectionClosed to make it clear it is supposed to handle broken connection during migration. It will soon be used on both sides of migration so the "Src" part was changed to "Any" to avoid renaming the function twice in a row. The original *Cleanup name could easily be confused with cleanup callbacks called when a domain is destroyed. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 07b8c99343..d44d5c6ec4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2336,9 +2336,9 @@ qemuMigrationDstRun(virQEMUDriver *driver, * qemuDomainMigratePerform3 and qemuDomainMigrateConfirm3. */ static void -qemuMigrationSrcCleanup(virDomainObj *vm, - virConnectPtr conn, - void *opaque) +qemuMigrationAnyConnectionClosed(virDomainObj *vm, + virConnectPtr conn, + void *opaque) { virQEMUDriver *driver = opaque; qemuDomainObjPrivate *priv = vm->privateData; @@ -2836,13 +2836,13 @@ qemuMigrationSrcBeginResumePhase(virConnectPtr conn, return NULL; virCloseCallbacksUnset(driver->closeCallbacks, vm, - qemuMigrationSrcCleanup); + qemuMigrationAnyConnectionClosed); qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); xml = qemuMigrationSrcBeginResume(driver, vm, xmlin, cookieout, cookieoutlen, flags); if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, - qemuMigrationSrcCleanup) < 0) + qemuMigrationAnyConnectionClosed) < 0) g_clear_pointer(&xml, g_free); if (!xml) @@ -2921,7 +2921,7 @@ qemuMigrationSrcBegin(virConnectPtr conn, * place. */ if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, - qemuMigrationSrcCleanup) < 0) { + qemuMigrationAnyConnectionClosed) < 0) { VIR_FREE(xml); goto endjob; } @@ -4001,7 +4001,7 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, goto cleanup; virCloseCallbacksUnset(driver->closeCallbacks, vm, - qemuMigrationSrcCleanup); + qemuMigrationAnyConnectionClosed); qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); ret = qemuMigrationSrcConfirmPhase(driver, vm, @@ -5977,7 +5977,7 @@ qemuMigrationSrcPerformResume(virQEMUDriver *driver, return -1; virCloseCallbacksUnset(driver->closeCallbacks, vm, - qemuMigrationSrcCleanup); + qemuMigrationAnyConnectionClosed); qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); ret = qemuMigrationSrcPerformNative(driver, vm, NULL, uri, @@ -5986,7 +5986,7 @@ qemuMigrationSrcPerformResume(virQEMUDriver *driver, 0, NULL, NULL, 0, NULL, NULL, NULL); if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, - qemuMigrationSrcCleanup) < 0) + qemuMigrationAnyConnectionClosed) < 0) ret = -1; if (ret < 0) @@ -6042,7 +6042,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto cleanup; virCloseCallbacksUnset(driver->closeCallbacks, vm, - qemuMigrationSrcCleanup); + qemuMigrationAnyConnectionClosed); if (qemuMigrationSrcPerformNative(driver, vm, persist_xml, uri, cookiein, cookieinlen, cookieout, cookieoutlen, @@ -6051,7 +6051,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, goto cleanup; if (virCloseCallbacksSet(driver->closeCallbacks, vm, conn, - qemuMigrationSrcCleanup) < 0) + qemuMigrationAnyConnectionClosed) < 0) goto cleanup; ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_PERFORM3_DONE)); -- 2.35.1
On Tue, May 10, 2022 at 17:21:25 +0200, Jiri Denemark wrote:
The function is now called qemuMigrationAnyConnectionClosed to make it clear it is supposed to handle broken connection during migration. It will soon be used on both sides of migration so the "Src" part was changed to "Any" to avoid renaming the function twice in a row.
The original *Cleanup name could easily be confused with cleanup callbacks called when a domain is destroyed.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
To prepare the code for handling incoming migration too. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 72 +++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 33 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index d44d5c6ec4..c0ab59f688 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2329,11 +2329,12 @@ qemuMigrationDstRun(virQEMUDriver *driver, } -/* This is called for outgoing non-p2p migrations when a connection to the - * client which initiated the migration was closed but we were waiting for it - * to follow up with the next phase, that is, in between - * qemuDomainMigrateBegin3 and qemuDomainMigratePerform3 or - * qemuDomainMigratePerform3 and qemuDomainMigrateConfirm3. +/* This is called for outgoing non-p2p and incoming migrations when a + * connection to the client which controls the migration was closed but we + * were waiting for it to follow up with the next phase, that is, in between + * qemuDomainMigrateBegin3 and qemuDomainMigratePerform3, + * qemuDomainMigratePerform3 and qemuDomainMigrateConfirm3, or + * qemuDomainMigratePrepare3 and qemuDomainMigrateFinish3. */ static void qemuMigrationAnyConnectionClosed(virDomainObj *vm, @@ -2343,6 +2344,7 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, virQEMUDriver *driver = opaque; qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobPrivate *jobPriv = priv->job.privateData; + bool postcopy = false; VIR_DEBUG("vm=%s, conn=%p, asyncJob=%s, phase=%s", vm->def->name, conn, @@ -2350,64 +2352,68 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, qemuDomainAsyncJobPhaseToString(priv->job.asyncJob, priv->job.phase)); - if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_OUT)) + if (!qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_IN) && + !qemuMigrationJobIsActive(vm, VIR_ASYNC_JOB_MIGRATION_OUT)) return; - VIR_DEBUG("The connection which started outgoing migration of domain %s" - " was closed; canceling the migration", + VIR_WARN("The connection which controls migration of domain %s was closed", vm->def->name); switch ((qemuMigrationJobPhase) priv->job.phase) { case QEMU_MIGRATION_PHASE_BEGIN3: - /* just forget we were about to migrate */ - qemuMigrationJobFinish(vm); + VIR_DEBUG("Aborting outgoing migration after Begin phase"); break; case QEMU_MIGRATION_PHASE_PERFORM3_DONE: - VIR_WARN("Migration of domain %s finished but we don't know if the" - " domain was successfully started on destination or not", - vm->def->name); - if (virDomainObjIsPostcopy(vm, VIR_DOMAIN_JOB_OPERATION_MIGRATION_OUT)) { - ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuMigrationSrcPostcopyFailed(vm); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + VIR_DEBUG("Migration protocol interrupted in post-copy mode"); + postcopy = true; } else { - qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, - jobPriv->migParams, priv->job.apiFlags); - /* clear the job and let higher levels decide what to do */ - qemuMigrationJobFinish(vm); + VIR_WARN("Migration of domain %s finished but we don't know if the " + "domain was successfully started on destination or not", + vm->def->name); } break; case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_BEGIN_RESUME: case QEMU_MIGRATION_PHASE_PERFORM_RESUME: - ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuMigrationSrcPostcopyFailed(vm); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + VIR_DEBUG("Connection closed while resuming failed post-copy migration"); + postcopy = true; break; + case QEMU_MIGRATION_PHASE_PREPARE: + case QEMU_MIGRATION_PHASE_FINISH2: + case QEMU_MIGRATION_PHASE_FINISH3: + case QEMU_MIGRATION_PHASE_PREPARE_RESUME: + case QEMU_MIGRATION_PHASE_FINISH_RESUME: + /* incoming migration; the domain will be autodestroyed */ + return; + case QEMU_MIGRATION_PHASE_PERFORM3: /* cannot be seen without an active migration API; unreachable */ case QEMU_MIGRATION_PHASE_CONFIRM3: case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: case QEMU_MIGRATION_PHASE_CONFIRM_RESUME: /* all done; unreachable */ - case QEMU_MIGRATION_PHASE_PREPARE: - case QEMU_MIGRATION_PHASE_FINISH2: - case QEMU_MIGRATION_PHASE_FINISH3: - case QEMU_MIGRATION_PHASE_PREPARE_RESUME: - case QEMU_MIGRATION_PHASE_FINISH_RESUME: - /* incoming migration; unreachable */ case QEMU_MIGRATION_PHASE_PERFORM2: /* single phase outgoing migration; unreachable */ case QEMU_MIGRATION_PHASE_NONE: case QEMU_MIGRATION_PHASE_LAST: /* unreachable */ - ; + return; + } + + if (postcopy) { + if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) + qemuMigrationSrcPostcopyFailed(vm); + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + } else { + qemuMigrationParamsReset(driver, vm, priv->job.asyncJob, + jobPriv->migParams, priv->job.apiFlags); + qemuMigrationJobFinish(vm); } } -- 2.35.1
On Tue, May 10, 2022 at 17:21:26 +0200, Jiri Denemark wrote:
To prepare the code for handling incoming migration too.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 72 +++++++++++++++++++++------------------ 1 file changed, 39 insertions(+), 33 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c0ab59f688..c1a60c90ef 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2378,15 +2378,12 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_BEGIN_RESUME: case QEMU_MIGRATION_PHASE_PERFORM_RESUME: + case QEMU_MIGRATION_PHASE_PREPARE_RESUME: VIR_DEBUG("Connection closed while resuming failed post-copy migration"); postcopy = true; break; case QEMU_MIGRATION_PHASE_PREPARE: - case QEMU_MIGRATION_PHASE_FINISH2: - case QEMU_MIGRATION_PHASE_FINISH3: - case QEMU_MIGRATION_PHASE_PREPARE_RESUME: - case QEMU_MIGRATION_PHASE_FINISH_RESUME: /* incoming migration; the domain will be autodestroyed */ return; @@ -2395,6 +2392,9 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, case QEMU_MIGRATION_PHASE_CONFIRM3: case QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED: case QEMU_MIGRATION_PHASE_CONFIRM_RESUME: + case QEMU_MIGRATION_PHASE_FINISH2: + case QEMU_MIGRATION_PHASE_FINISH3: + case QEMU_MIGRATION_PHASE_FINISH_RESUME: /* all done; unreachable */ case QEMU_MIGRATION_PHASE_PERFORM2: /* single phase outgoing migration; unreachable */ @@ -2407,6 +2407,8 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, if (postcopy) { if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) qemuMigrationSrcPostcopyFailed(vm); + else + qemuMigrationDstPostcopyFailed(vm); ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); -- 2.35.1
On Tue, May 10, 2022 at 17:21:27 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c0ab59f688..c1a60c90ef 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2378,15 +2378,12 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_BEGIN_RESUME: case QEMU_MIGRATION_PHASE_PERFORM_RESUME: + case QEMU_MIGRATION_PHASE_PREPARE_RESUME: VIR_DEBUG("Connection closed while resuming failed post-copy migration"); postcopy = true; break;
case QEMU_MIGRATION_PHASE_PREPARE: - case QEMU_MIGRATION_PHASE_FINISH2: - case QEMU_MIGRATION_PHASE_FINISH3: - case QEMU_MIGRATION_PHASE_PREPARE_RESUME: - case QEMU_MIGRATION_PHASE_FINISH_RESUME:
What was the point of moving these to a separate section and then moving them back, where they were before? Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Thu, May 12, 2022 at 17:09:27 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:27 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c0ab59f688..c1a60c90ef 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2378,15 +2378,12 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, case QEMU_MIGRATION_PHASE_POSTCOPY_FAILED: case QEMU_MIGRATION_PHASE_BEGIN_RESUME: case QEMU_MIGRATION_PHASE_PERFORM_RESUME: + case QEMU_MIGRATION_PHASE_PREPARE_RESUME: VIR_DEBUG("Connection closed while resuming failed post-copy migration"); postcopy = true; break;
case QEMU_MIGRATION_PHASE_PREPARE: - case QEMU_MIGRATION_PHASE_FINISH2: - case QEMU_MIGRATION_PHASE_FINISH3: - case QEMU_MIGRATION_PHASE_PREPARE_RESUME: - case QEMU_MIGRATION_PHASE_FINISH_RESUME:
What was the point of moving these to a separate section and then moving them back, where they were before?
No idea. Possibly wrong split between patches. I'll squash the changes into the previous patch. Jirka
Non-postcopy case talks to QEMU monitor and thus needs to create a nested job. Since qemuMigrationAnyConnectionClosed is called in case there's no thread processing a migration API, we need to make the current thread a temporary owner of the migration job to avoid "This thread doesn't seem to be the async job owner: 0". This is done by starting a migration phase. While no monitor interaction happens in postcopy case and just setting the phase (to indicate a broken postcopy migration) would be enough, being consistent and setting the owner does not hurt anything. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index c1a60c90ef..2e9235e1d5 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2345,6 +2345,7 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobPrivate *jobPriv = priv->job.privateData; bool postcopy = false; + int phase; VIR_DEBUG("vm=%s, conn=%p, asyncJob=%s, phase=%s", vm->def->name, conn, @@ -2404,12 +2405,17 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, return; } + if (postcopy) + phase = QEMU_MIGRATION_PHASE_POSTCOPY_FAILED; + else + phase = QEMU_MIGRATION_PHASE_CONFIRM3_CANCELLED; + ignore_value(qemuMigrationJobStartPhase(vm, phase)); + if (postcopy) { if (priv->job.asyncJob == VIR_ASYNC_JOB_MIGRATION_OUT) qemuMigrationSrcPostcopyFailed(vm); else qemuMigrationDstPostcopyFailed(vm); - ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); } else { -- 2.35.1
On Tue, May 10, 2022 at 17:21:28 +0200, Jiri Denemark wrote:
Non-postcopy case talks to QEMU monitor and thus needs to create a nested job. Since qemuMigrationAnyConnectionClosed is called in case there's no thread processing a migration API, we need to make the current thread a temporary owner of the migration job to avoid "This thread doesn't seem to be the async job owner: 0". This is done by starting a migration phase.
While no monitor interaction happens in postcopy case and just setting the phase (to indicate a broken postcopy migration) would be enough, being consistent and setting the owner does not hurt anything.
Especially since you clear it right after that :)
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The QEMU process is already running, all we need to do is to call migrate-recover QMP command. Except for some checks and cookie handling, of course. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 99 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 2e9235e1d5..a8481f7515 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3509,6 +3509,98 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, } +static int +qemuMigrationDstPrepareResume(virQEMUDriver *driver, + virConnectPtr conn, + const char *cookiein, + int cookieinlen, + char **cookieout, + int *cookieoutlen, + virDomainDef *def, + const char *origname, + const char *protocol, + unsigned short port, + const char *listenAddress, + unsigned long flags) +{ + g_autoptr(qemuMigrationCookie) mig = NULL; + qemuProcessIncomingDef *incoming = NULL; + qemuDomainObjPrivate *priv; + virDomainJobStatus status; + virDomainObj *vm; + int ret = -1; + + VIR_DEBUG("name=%s, origname=%s, protocol=%s, port=%hu, " + "listenAddress=%s, flags=0x%lx", + def->name, NULLSTR(origname), protocol, port, + NULLSTR(listenAddress), flags); + + vm = virDomainObjListFindByName(driver->domains, def->name); + if (!vm) { + virReportError(VIR_ERR_NO_DOMAIN, + _("no domain with matching name '%s'"), def->name); + qemuMigrationDstErrorReport(driver, def->name); + return -1; + } + priv = vm->privateData; + + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_IN, flags, + QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)) + goto cleanup; + + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_PREPARE_RESUME) < 0) + goto cleanup; + + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); + + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + &status) < 0) + goto cleanup; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + goto cleanup; + } + + if (!(mig = qemuMigrationCookieParse(driver, def, origname, NULL, + cookiein, cookieinlen, + QEMU_MIGRATION_COOKIE_CAPS))) + goto cleanup; + + if (!(incoming = qemuMigrationDstPrepare(vm, false, protocol, + listenAddress, port, -1))) + goto cleanup; + + if (qemuDomainObjEnterMonitorAsync(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN) < 0) + goto cleanup; + + ret = qemuMonitorMigrateRecover(priv->mon, incoming->uri); + qemuDomainObjExitMonitor(vm); + + if (ret < 0) + goto cleanup; + + if (qemuMigrationCookieFormat(mig, driver, vm, + QEMU_MIGRATION_DESTINATION, + cookieout, cookieoutlen, + QEMU_MIGRATION_COOKIE_CAPS) < 0) + VIR_WARN("Unable to encode migration cookie"); + + virCloseCallbacksSet(driver->closeCallbacks, vm, conn, + qemuMigrationAnyConnectionClosed); + + cleanup: + qemuProcessIncomingDefFree(incoming); + if (ret < 0) + ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); + qemuMigrationJobContinue(vm); + virDomainObjEndAPI(&vm); + return ret; +} + + static int qemuMigrationDstPrepareAny(virQEMUDriver *driver, virConnectPtr dconn, @@ -3577,6 +3669,13 @@ qemuMigrationDstPrepareAny(virQEMUDriver *driver, return -1; } + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + return qemuMigrationDstPrepareResume(driver, dconn, cookiein, cookieinlen, + cookieout, cookieoutlen, + *def, origname, protocol, port, + listenAddress, flags); + } + return qemuMigrationDstPrepareFresh(driver, dconn, cookiein, cookieinlen, cookieout, cookieoutlen, -- 2.35.1
On Tue, May 10, 2022 at 17:21:29 +0200, Jiri Denemark wrote:
The QEMU process is already running, all we need to do is to call migrate-recover QMP command. Except for some checks and cookie handling, of course.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 99 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Tue, May 10, 2022 at 17:21:29 +0200, Jiri Denemark wrote:
The QEMU process is already running, all we need to do is to call migrate-recover QMP command. Except for some checks and cookie handling, of course.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 99 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 2e9235e1d5..a8481f7515 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3509,6 +3509,98 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, }
[...]
+ + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + &status) < 0) + goto cleanup; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + goto cleanup; + }
Is there any reason not to allow adopting a running unattended migration with a recovery API?
On Thu, May 12, 2022 at 17:16:32 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:29 +0200, Jiri Denemark wrote:
The QEMU process is already running, all we need to do is to call migrate-recover QMP command. Except for some checks and cookie handling, of course.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 99 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 2e9235e1d5..a8481f7515 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3509,6 +3509,98 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, }
[...]
+ + if (qemuMigrationAnyRefreshStatus(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, + &status) < 0) + goto cleanup; + + if (status != VIR_DOMAIN_JOB_STATUS_POSTCOPY_PAUSED) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("QEMU reports migration is still running")); + goto cleanup; + }
Is there any reason not to allow adopting a running unattended migration with a recovery API?
It's easier this way :-) The migration will happily finish anyway so it does not really matter. You can't just have a synchronous API call and finished migration is only announced via events. Anyway, it can definitely be done in a follow up series if desired, although it will make the code even more complicated as we would have to do all the steps and just avoid running any QMP commands and jump right into waiting for migration to finish on both sides. Jirka
Everything was already done in the normal Finish phase and vCPUs are running. We just need to wait for all remaining data to be transferred. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 46 ++++++++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 6 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a8481f7515..430dfb1abb 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -6600,6 +6600,22 @@ qemuMigrationDstFinishFresh(virQEMUDriver *driver, } +static int +qemuMigrationDstFinishResume(virQEMUDriver *driver, + virDomainObj *vm) +{ + VIR_DEBUG("vm=%p", vm); + + if (qemuMigrationDstWaitForCompletion(driver, vm, + VIR_ASYNC_JOB_MIGRATION_IN, + false) < 0) { + return -1; + } + + return 0; +} + + static virDomainPtr qemuMigrationDstFinishActive(virQEMUDriver *driver, virConnectPtr dconn, @@ -6647,8 +6663,14 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, goto error; } - rc = qemuMigrationDstFinishFresh(driver, vm, mig, flags, v3proto, - timeReceived, &doKill, &inPostCopy); + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + rc = qemuMigrationDstFinishResume(driver, vm); + inPostCopy = true; + } else { + rc = qemuMigrationDstFinishFresh(driver, vm, mig, flags, v3proto, + timeReceived, &doKill, &inPostCopy); + } + if (rc < 0 || !(dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id))) goto error; @@ -6719,6 +6741,8 @@ qemuMigrationDstFinish(virQEMUDriver *driver, qemuDomainObjPrivate *priv = vm->privateData; unsigned short port; unsigned long long timeReceived = 0; + int phase = v3proto ? QEMU_MIGRATION_PHASE_FINISH3 + : QEMU_MIGRATION_PHASE_FINISH2; VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " "cookieout=%p, cookieoutlen=%p, flags=0x%lx, retcode=%d", @@ -6733,14 +6757,24 @@ qemuMigrationDstFinish(virQEMUDriver *driver, goto cleanup; } + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_IN, flags, + QEMU_MIGRATION_PHASE_PREPARE_RESUME)) + goto cleanup; + phase = QEMU_MIGRATION_PHASE_FINISH_RESUME; + } ignore_value(virTimeMillisNow(&timeReceived)); - if (qemuMigrationJobStartPhase(vm, - v3proto ? QEMU_MIGRATION_PHASE_FINISH3 - : QEMU_MIGRATION_PHASE_FINISH2) < 0) + if (qemuMigrationJobStartPhase(vm, phase) < 0) goto cleanup; - qemuDomainCleanupRemove(vm, qemuMigrationDstPrepareCleanup); + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + virCloseCallbacksUnset(driver->closeCallbacks, vm, + qemuMigrationAnyConnectionClosed); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); + } else { + qemuDomainCleanupRemove(vm, qemuMigrationDstPrepareCleanup); + } g_clear_pointer(&priv->job.completed, virDomainJobDataFree); cookie_flags = QEMU_MIGRATION_COOKIE_NETWORK | -- 2.35.1
On Tue, May 10, 2022 at 17:21:30 +0200, Jiri Denemark wrote:
Everything was already done in the normal Finish phase and vCPUs are running. We just need to wait for all remaining data to be transferred.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 46 ++++++++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 6 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a8481f7515..430dfb1abb 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -6600,6 +6600,22 @@ qemuMigrationDstFinishFresh(virQEMUDriver *driver, }
+static int +qemuMigrationDstFinishResume(virQEMUDriver *driver, + virDomainObj *vm) +{ + VIR_DEBUG("vm=%p", vm); + + if (qemuMigrationDstWaitForCompletion(driver, vm, + VIR_ASYNC_JOB_MIGRATION_IN, + false) < 0) { + return -1; + }
As I mentioned in another reply, IMO it would be useful to allow adoption of a unattended running migration precisely for this case, so that mgmt apps don't have to encode more logic to wait for events if migration was actually running fine.
+ + return 0; +} + + static virDomainPtr qemuMigrationDstFinishActive(virQEMUDriver *driver, virConnectPtr dconn, @@ -6647,8 +6663,14 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, goto error; }
- rc = qemuMigrationDstFinishFresh(driver, vm, mig, flags, v3proto, - timeReceived, &doKill, &inPostCopy); + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + rc = qemuMigrationDstFinishResume(driver, vm); + inPostCopy = true; + } else { + rc = qemuMigrationDstFinishFresh(driver, vm, mig, flags, v3proto, + timeReceived, &doKill, &inPostCopy); + } + if (rc < 0 || !(dom = virGetDomain(dconn, vm->def->name, vm->def->uuid, vm->def->id))) goto error; @@ -6719,6 +6741,8 @@ qemuMigrationDstFinish(virQEMUDriver *driver, qemuDomainObjPrivate *priv = vm->privateData; unsigned short port; unsigned long long timeReceived = 0; + int phase = v3proto ? QEMU_MIGRATION_PHASE_FINISH3 + : QEMU_MIGRATION_PHASE_FINISH2;
Avoid use of ?
VIR_DEBUG("driver=%p, dconn=%p, vm=%p, cookiein=%s, cookieinlen=%d, " "cookieout=%p, cookieoutlen=%p, flags=0x%lx, retcode=%d",
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Thu, May 12, 2022 at 17:20:34 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:30 +0200, Jiri Denemark wrote:
Everything was already done in the normal Finish phase and vCPUs are running. We just need to wait for all remaining data to be transferred.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 46 ++++++++++++++++++++++++++++++++++----- 1 file changed, 40 insertions(+), 6 deletions(-)
diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index a8481f7515..430dfb1abb 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -6600,6 +6600,22 @@ qemuMigrationDstFinishFresh(virQEMUDriver *driver, }
+static int +qemuMigrationDstFinishResume(virQEMUDriver *driver, + virDomainObj *vm) +{ + VIR_DEBUG("vm=%p", vm); + + if (qemuMigrationDstWaitForCompletion(driver, vm, + VIR_ASYNC_JOB_MIGRATION_IN, + false) < 0) { + return -1; + }
As I mentioned in another reply, IMO it would be useful to allow adoption of a unattended running migration precisely for this case, so that mgmt apps don't have to encode more logic to wait for events if migration was actually running fine.
I think event processing in mgmt apps is inevitable anyway as the migration may finish before an app even tries to recover the migration in case only libvirt side of migration was broken. And it may still be broken to make recovery impossible while migration is progressing just fine (esp. if migration streams use separate network). Even plain post-copy which never fails requires event processing since the management has to explicitly switch migration to post-copy mode. Also events are often the only way to get statistics about the completed migration. That said, I think it would be nice to have this functionality anyway so I'll try to look at it as a follow up series. Jirka
Normally the structure is created once the source reports completed migration, but with post-copy migration we can get here even after libvirt daemon was restarted. It doesn't make sense to preserve the structure in our status XML as we're going to rewrite almost all of it while refreshing the stats anyway. So we just create the structure here if it doesn't exist to make sure we can properly report statistics of a completed migration. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 30 +++++++++++++++++------------- 1 file changed, 17 insertions(+), 13 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 430dfb1abb..1fc978bad5 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -3947,22 +3947,26 @@ qemuMigrationSrcComplete(virQEMUDriver *driver, virObjectEvent *event; int reason; - if (jobData) { - /* We need to refresh migration statistics after a completed post-copy - * migration since jobData contains obsolete data from the time we - * switched to post-copy mode. - */ - if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && - reason == VIR_DOMAIN_PAUSED_POSTCOPY) { - VIR_DEBUG("Refreshing migration statistics"); - if (qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, - jobData, NULL) < 0) - VIR_WARN("Could not refresh migration statistics"); - } + if (!jobData) { + priv->job.completed = virDomainJobDataCopy(priv->job.current); + jobData = priv->job.completed; + jobData->status = VIR_DOMAIN_JOB_STATUS_COMPLETED; + } - qemuDomainJobDataUpdateTime(jobData); + /* We need to refresh migration statistics after a completed post-copy + * migration since jobData contains obsolete data from the time we + * switched to post-copy mode. + */ + if (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && + reason == VIR_DOMAIN_PAUSED_POSTCOPY) { + VIR_DEBUG("Refreshing migration statistics"); + if (qemuMigrationAnyFetchStats(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + jobData, NULL) < 0) + VIR_WARN("Could not refresh migration statistics"); } + qemuDomainJobDataUpdateTime(jobData); + /* If guest uses SPICE and supports seamless migration we have to hold * up domain shutdown until SPICE server transfers its data */ qemuMigrationSrcWaitForSpice(vm); -- 2.35.1
On Tue, May 10, 2022 at 17:21:31 +0200, Jiri Denemark wrote:
Normally the structure is created once the source reports completed migration, but with post-copy migration we can get here even after libvirt daemon was restarted. It doesn't make sense to preserve the structure in our status XML as we're going to rewrite almost all of it while refreshing the stats anyway. So we just create the structure here if it doesn't exist to make sure we can properly report statistics of a completed migration.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 30 +++++++++++++++++-------------
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The callback will properly cleanup non-p2p migration job in case the migrating domain dies between Begin and Perform while the client which controls the migration is not cooperating (normally the API for the next migration phase would handle this). The same situation can happen even after Prepare and Perform phases, but they both already register a suitable callback, so no fix is needed there. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 1 + 1 file changed, 1 insertion(+) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 1fc978bad5..38a67f7e20 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2939,6 +2939,7 @@ qemuMigrationSrcBegin(virConnectPtr conn, VIR_FREE(xml); goto endjob; } + qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); qemuMigrationJobContinue(vm); } else { goto endjob; -- 2.35.1
On Tue, May 10, 2022 at 17:21:32 +0200, Jiri Denemark wrote:
The callback will properly cleanup non-p2p migration job in case the migrating domain dies between Begin and Perform while the client which controls the migration is not cooperating (normally the API for the next migration phase would handle this).
The same situation can happen even after Prepare and Perform phases, but they both already register a suitable callback, so no fix is needed there.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 1 + 1 file changed, 1 insertion(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Every single call to qemuMigrationJobContinue needs to register a cleanup callback in case the migrating domain dies between phases or when migration is paused due to a failure in postcopy mode. Let's integrate registering the callback in qemuMigrationJobContinue to make sure the current thread does not release a migration job without setting a cleanup callback. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 35 +++++++++++++---------------------- 1 file changed, 13 insertions(+), 22 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index 38a67f7e20..d7e8b9d1e4 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -183,8 +183,10 @@ qemuMigrationJobStartPhase(virDomainObj *vm, static void ATTRIBUTE_NONNULL(1) -qemuMigrationJobContinue(virDomainObj *vm) +qemuMigrationJobContinue(virDomainObj *vm, + qemuDomainCleanupCallback cleanup) { + qemuDomainCleanupAdd(vm, cleanup); qemuDomainObjReleaseAsyncJob(vm); } @@ -2416,8 +2418,7 @@ qemuMigrationAnyConnectionClosed(virDomainObj *vm, qemuMigrationSrcPostcopyFailed(vm); else qemuMigrationDstPostcopyFailed(vm); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); } else { qemuMigrationParamsReset(driver, vm, priv->job.asyncJob, jobPriv->migParams, priv->job.apiFlags); @@ -2862,8 +2863,7 @@ qemuMigrationSrcBeginResumePhase(virConnectPtr conn, if (!xml) ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); return g_steal_pointer(&xml); } @@ -2939,8 +2939,7 @@ qemuMigrationSrcBegin(virConnectPtr conn, VIR_FREE(xml); goto endjob; } - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); } else { goto endjob; } @@ -3476,13 +3475,11 @@ qemuMigrationDstPrepareFresh(virQEMUDriver *driver, VIR_WARN("Unable to encode migration cookie"); } - qemuDomainCleanupAdd(vm, qemuMigrationDstPrepareCleanup); - /* We keep the job active across API calls until the finish() call. * This prevents any other APIs being invoked while incoming * migration is taking place. */ - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuMigrationDstPrepareCleanup); if (autoPort) priv->migrationPort = port; @@ -3595,8 +3592,7 @@ qemuMigrationDstPrepareResume(virQEMUDriver *driver, qemuProcessIncomingDefFree(incoming); if (ret < 0) ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); virDomainObjEndAPI(&vm); return ret; } @@ -4128,8 +4124,7 @@ qemuMigrationSrcConfirm(virQEMUDriver *driver, if (virDomainObjIsFailedPostcopy(vm)) { ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); } else { qemuMigrationJobFinish(vm); } @@ -6041,8 +6036,7 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, if (virDomainObjIsFailedPostcopy(vm)) { ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); } else { /* v2 proto has no confirm phase so we need to reset migration parameters * here @@ -6110,8 +6104,7 @@ qemuMigrationSrcPerformResume(virQEMUDriver *driver, if (ret < 0) ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); return ret; } @@ -6184,8 +6177,7 @@ qemuMigrationSrcPerformPhase(virQEMUDriver *driver, } else { if (ret < 0) ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); } if (!virDomainObjIsActive(vm)) @@ -6714,7 +6706,6 @@ qemuMigrationDstFinishActive(virQEMUDriver *driver, if (virDomainObjIsFailedPostcopy(vm)) { ignore_value(qemuMigrationJobSetPhase(vm, QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)); qemuProcessAutoDestroyRemove(driver, vm); - qemuDomainCleanupAdd(vm, qemuProcessCleanupMigrationJob); *finishJob = false; } else { qemuMigrationParamsReset(driver, vm, VIR_ASYNC_JOB_MIGRATION_IN, @@ -6805,7 +6796,7 @@ qemuMigrationDstFinish(virQEMUDriver *driver, flags, retcode, v3proto, timeReceived, &orig_err, &finishJob); if (!finishJob) { - qemuMigrationJobContinue(vm); + qemuMigrationJobContinue(vm, qemuProcessCleanupMigrationJob); goto cleanup; } } -- 2.35.1
On Tue, May 10, 2022 at 17:21:33 +0200, Jiri Denemark wrote:
Every single call to qemuMigrationJobContinue needs to register a cleanup callback in case the migrating domain dies between phases or when migration is paused due to a failure in postcopy mode.
Let's integrate registering the callback in qemuMigrationJobContinue to make sure the current thread does not release a migration job without setting a cleanup callback.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 35 +++++++++++++---------------------- 1 file changed, 13 insertions(+), 22 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 53 ++++++++++++++++++++++++++------------- 1 file changed, 36 insertions(+), 17 deletions(-) diff --git a/src/qemu/qemu_migration.c b/src/qemu/qemu_migration.c index d7e8b9d1e4..24aa831c45 100644 --- a/src/qemu/qemu_migration.c +++ b/src/qemu/qemu_migration.c @@ -2759,7 +2759,7 @@ qemuMigrationAnyCanResume(virDomainObj *vm, */ if (job == VIR_ASYNC_JOB_MIGRATION_OUT && expectedPhase < QEMU_MIGRATION_PHASE_PERFORM_RESUME && - !(flags & VIR_MIGRATE_CHANGE_PROTECTION)) { + !(flags & (VIR_MIGRATE_CHANGE_PROTECTION | VIR_MIGRATE_PEER2PEER))) { virReportError(VIR_ERR_ARGUMENT_UNSUPPORTED, "%s", _("resuming failed post-copy migration requires " "change protection")); @@ -5473,9 +5473,14 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, * bit here, because we are already running inside the context of * a single job. */ - dom_xml = qemuMigrationSrcBeginPhase(driver, vm, xmlin, dname, - &cookieout, &cookieoutlen, - nmigrate_disks, migrate_disks, flags); + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + dom_xml = qemuMigrationSrcBeginResume(driver, vm, xmlin, + &cookieout, &cookieoutlen, flags); + } else { + dom_xml = qemuMigrationSrcBeginPhase(driver, vm, xmlin, dname, + &cookieout, &cookieoutlen, + nmigrate_disks, migrate_disks, flags); + } if (!dom_xml) goto cleanup; @@ -5531,7 +5536,8 @@ qemuMigrationSrcPerformPeer2Peer3(virQEMUDriver *driver, goto cleanup; } - if (virDomainObjGetState(vm, NULL) == VIR_DOMAIN_PAUSED) + if (!(flags & VIR_MIGRATE_POSTCOPY_RESUME) && + virDomainObjGetState(vm, NULL) == VIR_DOMAIN_PAUSED) flags |= VIR_MIGRATE_PAUSED; destflags = flags & ~(VIR_MIGRATE_ABORT_ON_ERROR | @@ -5980,22 +5986,35 @@ qemuMigrationSrcPerformJob(virQEMUDriver *driver, qemuDomainObjPrivate *priv = vm->privateData; qemuDomainJobPrivate *jobPriv = priv->job.privateData; - if (qemuMigrationJobStart(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, - flags) < 0) - goto cleanup; + if (flags & VIR_MIGRATE_POSTCOPY_RESUME) { + if (!qemuMigrationAnyCanResume(vm, VIR_ASYNC_JOB_MIGRATION_OUT, flags, + QEMU_MIGRATION_PHASE_POSTCOPY_FAILED)) + goto cleanup; - if (!(flags & VIR_MIGRATE_OFFLINE) && virDomainObjCheckActive(vm) < 0) - goto endjob; + if (qemuMigrationJobStartPhase(vm, QEMU_MIGRATION_PHASE_BEGIN_RESUME) < 0) + goto cleanup; - if (!qemuMigrationSrcIsAllowed(driver, vm, true, flags)) - goto endjob; + virCloseCallbacksUnset(driver->closeCallbacks, vm, + qemuMigrationAnyConnectionClosed); + qemuDomainCleanupRemove(vm, qemuProcessCleanupMigrationJob); + } else { + if (qemuMigrationJobStart(driver, vm, VIR_ASYNC_JOB_MIGRATION_OUT, + flags) < 0) + goto cleanup; - if (!(flags & (VIR_MIGRATE_UNSAFE | VIR_MIGRATE_OFFLINE)) && - !qemuMigrationSrcIsSafe(vm->def, priv->qemuCaps, - nmigrate_disks, migrate_disks, flags)) - goto endjob; + if (!(flags & VIR_MIGRATE_OFFLINE) && virDomainObjCheckActive(vm) < 0) + goto endjob; - qemuMigrationSrcStoreDomainState(vm); + if (!qemuMigrationSrcIsAllowed(driver, vm, true, flags)) + goto endjob; + + if (!(flags & (VIR_MIGRATE_UNSAFE | VIR_MIGRATE_OFFLINE)) && + !qemuMigrationSrcIsSafe(vm->def, priv->qemuCaps, + nmigrate_disks, migrate_disks, flags)) + goto endjob; + + qemuMigrationSrcStoreDomainState(vm); + } if ((flags & (VIR_MIGRATE_TUNNELLED | VIR_MIGRATE_PEER2PEER))) { ret = qemuMigrationSrcPerformPeer2Peer(driver, conn, vm, xmlin, persist_xml, -- 2.35.1
On Tue, May 10, 2022 at 17:21:34 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 53 ++++++++++++++++++++++++++------------- 1 file changed, 36 insertions(+), 17 deletions(-)
On Thu, May 12, 2022 at 17:46:34 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:34 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.c | 53 ++++++++++++++++++++++++++------------- 1 file changed, 36 insertions(+), 17 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Since all parts of post-copy recovery have been implemented now, it's time to enable the corresponding flag. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.h | 1 + 1 file changed, 1 insertion(+) diff --git a/src/qemu/qemu_migration.h b/src/qemu/qemu_migration.h index 7eb0d4fe02..fbc0549b34 100644 --- a/src/qemu/qemu_migration.h +++ b/src/qemu/qemu_migration.h @@ -60,6 +60,7 @@ VIR_MIGRATE_TLS | \ VIR_MIGRATE_PARALLEL | \ VIR_MIGRATE_NON_SHARED_SYNCHRONOUS_WRITES | \ + VIR_MIGRATE_POSTCOPY_RESUME | \ 0) /* All supported migration parameters and their types. */ -- 2.35.1
On Tue, May 10, 2022 at 17:21:35 +0200, Jiri Denemark wrote:
Since all parts of post-copy recovery have been implemented now, it's time to enable the corresponding flag.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_migration.h | 1 + 1 file changed, 1 insertion(+)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
The original virDomainAbortJob did not support flags. Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 3 +++ src/driver-hypervisor.h | 5 ++++ src/libvirt-domain.c | 45 ++++++++++++++++++++++++++++++++ src/libvirt_public.syms | 1 + src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 14 +++++++++- src/remote_protocol-structs | 5 ++++ 7 files changed, 73 insertions(+), 1 deletion(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index aee2b49114..af4406cb61 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -4099,6 +4099,9 @@ int virDomainGetJobStats(virDomainPtr domain, unsigned int flags); int virDomainAbortJob(virDomainPtr dom); +int virDomainAbortJobFlags(virDomainPtr dom, + unsigned int flags); + /** * virDomainJobOperation: * diff --git a/src/driver-hypervisor.h b/src/driver-hypervisor.h index 69516e8fea..016d5cec7c 100644 --- a/src/driver-hypervisor.h +++ b/src/driver-hypervisor.h @@ -746,6 +746,10 @@ typedef int typedef int (*virDrvDomainAbortJob)(virDomainPtr domain); +typedef int +(*virDrvDomainAbortJobFlags)(virDomainPtr domain, + unsigned int flags); + typedef int (*virDrvDomainMigrateGetMaxDowntime)(virDomainPtr domain, unsigned long long *downtime, @@ -1590,6 +1594,7 @@ struct _virHypervisorDriver { virDrvDomainGetJobInfo domainGetJobInfo; virDrvDomainGetJobStats domainGetJobStats; virDrvDomainAbortJob domainAbortJob; + virDrvDomainAbortJobFlags domainAbortJobFlags; virDrvDomainMigrateGetMaxDowntime domainMigrateGetMaxDowntime; virDrvDomainMigrateSetMaxDowntime domainMigrateSetMaxDowntime; virDrvDomainMigrateGetCompressionCache domainMigrateGetCompressionCache; diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 07fb8119d9..1e1b378361 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -9459,6 +9459,51 @@ virDomainAbortJob(virDomainPtr domain) } +/** + * virDomainAbortJobFlags: + * @domain: a domain object + * @flags: extra flags; not used yet, callers should always pass 0 + * + * Requests that the current background job be aborted at the + * soonest opportunity. In case the job is a migration in a post-copy mode, + * this function will report an error (see virDomainMigrateStartPostCopy for + * more details). + * + * Returns 0 in case of success and -1 in case of failure. + * + * Since: 8.4.0 + */ +int +virDomainAbortJobFlags(virDomainPtr domain, + unsigned int flags) +{ + virConnectPtr conn; + + VIR_DOMAIN_DEBUG(domain); + + virResetLastError(); + + virCheckDomainReturn(domain, -1); + conn = domain->conn; + + virCheckReadOnlyGoto(conn->flags, error); + + if (conn->driver->domainAbortJobFlags) { + int ret; + ret = conn->driver->domainAbortJobFlags(domain, flags); + if (ret < 0) + goto error; + return ret; + } + + virReportUnsupportedError(); + + error: + virDispatchError(conn); + return -1; +} + + /** * virDomainMigrateSetMaxDowntime: * @domain: a domain object diff --git a/src/libvirt_public.syms b/src/libvirt_public.syms index 9f58b52924..c68bbf3680 100644 --- a/src/libvirt_public.syms +++ b/src/libvirt_public.syms @@ -918,6 +918,7 @@ LIBVIRT_8.0.0 { LIBVIRT_8.4.0 { global: + virDomainAbortJobFlags; virDomainSaveParams; virDomainRestoreParams; } LIBVIRT_8.0.0; diff --git a/src/remote/remote_driver.c b/src/remote/remote_driver.c index 423f5f9fb9..5ad517a387 100644 --- a/src/remote/remote_driver.c +++ b/src/remote/remote_driver.c @@ -8534,6 +8534,7 @@ static virHypervisorDriver hypervisor_driver = { .domainGetJobInfo = remoteDomainGetJobInfo, /* 0.7.7 */ .domainGetJobStats = remoteDomainGetJobStats, /* 1.0.3 */ .domainAbortJob = remoteDomainAbortJob, /* 0.7.7 */ + .domainAbortJobFlags = remoteDomainAbortJobFlags, /* 8.2.0 */ .domainMigrateGetMaxDowntime = remoteDomainMigrateGetMaxDowntime, /* 3.7.0 */ .domainMigrateSetMaxDowntime = remoteDomainMigrateSetMaxDowntime, /* 0.8.0 */ .domainMigrateGetCompressionCache = remoteDomainMigrateGetCompressionCache, /* 1.0.3 */ diff --git a/src/remote/remote_protocol.x b/src/remote/remote_protocol.x index 085631c11b..79ffc63f03 100644 --- a/src/remote/remote_protocol.x +++ b/src/remote/remote_protocol.x @@ -2492,6 +2492,12 @@ struct remote_domain_abort_job_args { }; +struct remote_domain_abort_job_flags_args { + remote_nonnull_domain dom; + unsigned int flags; +}; + + struct remote_domain_migrate_get_max_downtime_args { remote_nonnull_domain dom; unsigned int flags; @@ -6947,5 +6953,11 @@ enum remote_procedure { * @acl: domain:start * @acl: domain:write */ - REMOTE_PROC_DOMAIN_RESTORE_PARAMS = 441 + REMOTE_PROC_DOMAIN_RESTORE_PARAMS = 441, + + /** + * @generate: both + * @acl: domain:write + */ + REMOTE_PROC_DOMAIN_ABORT_JOB_FLAGS = 442 }; diff --git a/src/remote_protocol-structs b/src/remote_protocol-structs index 4ffdce5679..ca5222439d 100644 --- a/src/remote_protocol-structs +++ b/src/remote_protocol-structs @@ -1869,6 +1869,10 @@ struct remote_domain_get_job_stats_ret { struct remote_domain_abort_job_args { remote_nonnull_domain dom; }; +struct remote_domain_abort_job_flags_args { + remote_nonnull_domain dom; + u_int flags; +}; struct remote_domain_migrate_get_max_downtime_args { remote_nonnull_domain dom; u_int flags; @@ -3706,4 +3710,5 @@ enum remote_procedure { REMOTE_PROC_DOMAIN_SET_LAUNCH_SECURITY_STATE = 439, REMOTE_PROC_DOMAIN_SAVE_PARAMS = 440, REMOTE_PROC_DOMAIN_RESTORE_PARAMS = 441, + REMOTE_PROC_DOMAIN_ABORT_JOB_FLAGS = 442, }; -- 2.35.1
On Tue, May 10, 2022 at 17:21:36 +0200, Jiri Denemark wrote:
The original virDomainAbortJob did not support flags.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 3 +++ src/driver-hypervisor.h | 5 ++++ src/libvirt-domain.c | 45 ++++++++++++++++++++++++++++++++ src/libvirt_public.syms | 1 + src/remote/remote_driver.c | 1 + src/remote/remote_protocol.x | 14 +++++++++- src/remote_protocol-structs | 5 ++++ 7 files changed, 73 insertions(+), 1 deletion(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index e268d3a478..efa6cbe14d 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12770,7 +12770,9 @@ qemuDomainAbortJobMigration(virDomainObj *vm) } -static int qemuDomainAbortJob(virDomainPtr dom) +static int +qemuDomainAbortJobFlags(virDomainPtr dom, + unsigned int flags) { virQEMUDriver *driver = dom->conn->privateData; virDomainObj *vm; @@ -12778,10 +12780,12 @@ static int qemuDomainAbortJob(virDomainPtr dom) qemuDomainObjPrivate *priv; int reason; + virCheckFlags(0, -1); + if (!(vm = qemuDomainObjFromDomain(dom))) goto cleanup; - if (virDomainAbortJobEnsureACL(dom->conn, vm->def) < 0) + if (virDomainAbortJobFlagsEnsureACL(dom->conn, vm->def) < 0) goto cleanup; if (qemuDomainObjBeginJob(driver, vm, VIR_JOB_ABORT) < 0) @@ -12860,6 +12864,13 @@ static int qemuDomainAbortJob(virDomainPtr dom) } +static int +qemuDomainAbortJob(virDomainPtr dom) +{ + return qemuDomainAbortJobFlags(dom, 0); +} + + static int qemuDomainMigrateSetMaxDowntime(virDomainPtr dom, unsigned long long downtime, @@ -21018,6 +21029,7 @@ static virHypervisorDriver qemuHypervisorDriver = { .domainGetJobInfo = qemuDomainGetJobInfo, /* 0.7.7 */ .domainGetJobStats = qemuDomainGetJobStats, /* 1.0.3 */ .domainAbortJob = qemuDomainAbortJob, /* 0.7.7 */ + .domainAbortJobFlags = qemuDomainAbortJobFlags, /* 8.4.0 */ .domainMigrateGetMaxDowntime = qemuDomainMigrateGetMaxDowntime, /* 3.7.0 */ .domainMigrateSetMaxDowntime = qemuDomainMigrateSetMaxDowntime, /* 0.8.0 */ .domainMigrateGetCompressionCache = qemuDomainMigrateGetCompressionCache, /* 1.0.3 */ -- 2.35.1
On Tue, May 10, 2022 at 17:21:37 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 16 ++++++++++++++++ src/libvirt-domain.c | 10 ++++++---- 2 files changed, 22 insertions(+), 4 deletions(-) diff --git a/include/libvirt/libvirt-domain.h b/include/libvirt/libvirt-domain.h index af4406cb61..bcd377b74f 100644 --- a/include/libvirt/libvirt-domain.h +++ b/include/libvirt/libvirt-domain.h @@ -4099,6 +4099,22 @@ int virDomainGetJobStats(virDomainPtr domain, unsigned int flags); int virDomainAbortJob(virDomainPtr dom); +/** + * virDomainAbortJobFlagsValues: + * + * Flags OR'ed together to provide specific behavior when aborting a domain job. + * + * Since: 8.4.0 + */ +typedef enum { + /* Interrupt post-copy migration. Since migration in a post-copy phase + * cannot be aborted without losing the domain (none of the hosts involved + * in migration has a complete state of the domain), the migration will be + * suspended and it can later be resumed using virDomainMigrate* APIs with + * VIR_MIGRATE_POSTCOPY_RESUME flag. (Since: 8.4.0) */ + VIR_DOMAIN_ABORT_JOB_POSTCOPY = 1 << 0, +} virDomainAbortJobFlagsValues; + int virDomainAbortJobFlags(virDomainPtr dom, unsigned int flags); diff --git a/src/libvirt-domain.c b/src/libvirt-domain.c index 1e1b378361..5f0f60781a 100644 --- a/src/libvirt-domain.c +++ b/src/libvirt-domain.c @@ -9462,12 +9462,12 @@ virDomainAbortJob(virDomainPtr domain) /** * virDomainAbortJobFlags: * @domain: a domain object - * @flags: extra flags; not used yet, callers should always pass 0 + * @flags: bitwise-OR of virDomainAbortJobFlagsValues * * Requests that the current background job be aborted at the * soonest opportunity. In case the job is a migration in a post-copy mode, - * this function will report an error (see virDomainMigrateStartPostCopy for - * more details). + * this function will report an error unless VIR_DOMAIN_ABORT_JOB_POSTCOPY + * flag is used (see virDomainMigrateStartPostCopy for more details). * * Returns 0 in case of success and -1 in case of failure. * @@ -9807,7 +9807,9 @@ virDomainMigrateGetMaxSpeed(virDomainPtr domain, * running, individual tasks in the guest may be blocked waiting for a page that * has not been migrated yet. It's up to the upper layer to decide what to do * in such case. Because of this, libvirt will refuse to cancel post-copy - * migration via virDomainAbortJob. + * migration via virDomainAbortJobFlags unless it is called with + * VIR_DOMAIN_ABORT_JOB_POSTCOPY, in which case the post-copy migration will be + * paused. * * Failed post-copy migration can be recovered once the cause for the failure * (e.g., a network issue) is resolved by repeating the migration with an -- 2.35.1
On Tue, May 10, 2022 at 17:21:38 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- include/libvirt/libvirt-domain.h | 16 ++++++++++++++++ src/libvirt-domain.c | 10 ++++++----
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 43 +++++++++++++++++++++++++++++------- src/qemu/qemu_monitor.c | 9 ++++++++ src/qemu/qemu_monitor.h | 3 +++ src/qemu/qemu_monitor_json.c | 22 ++++++++++++++++++ src/qemu/qemu_monitor_json.h | 3 +++ 5 files changed, 72 insertions(+), 8 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index efa6cbe14d..344bbe2196 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -12770,6 +12770,36 @@ qemuDomainAbortJobMigration(virDomainObj *vm) } +static int +qemuDomainAbortJobPostcopy(virDomainObj *vm, + unsigned int flags) +{ + qemuDomainObjPrivate *priv = vm->privateData; + int rc; + + if (!(flags & VIR_DOMAIN_ABORT_JOB_POSTCOPY)) { + virReportError(VIR_ERR_OPERATION_INVALID, "%s", + _("cannot abort migration in post-copy mode")); + return -1; + } + + VIR_DEBUG("Suspending post-copy migration at client request"); + + qemuDomainObjAbortAsyncJob(vm); + qemuDomainObjEnterMonitor(priv->driver, vm); + rc = qemuMonitorMigratePause(priv->mon); + qemuDomainObjExitMonitor(vm); + + if (rc == -2) { + virReportError(VIR_ERR_OPERATION_UNSUPPORTED, "%s", + _("cannot suspend post-copy migration")); + return -1; + } + + return rc; +} + + static int qemuDomainAbortJobFlags(virDomainPtr dom, unsigned int flags) @@ -12780,7 +12810,7 @@ qemuDomainAbortJobFlags(virDomainPtr dom, qemuDomainObjPrivate *priv; int reason; - virCheckFlags(0, -1); + virCheckFlags(VIR_DOMAIN_ABORT_JOB_POSTCOPY, -1); if (!(vm = qemuDomainObjFromDomain(dom))) goto cleanup; @@ -12817,13 +12847,10 @@ qemuDomainAbortJobFlags(virDomainPtr dom, case VIR_ASYNC_JOB_MIGRATION_OUT: if ((priv->job.current->status == VIR_DOMAIN_JOB_STATUS_POSTCOPY || (virDomainObjGetState(vm, &reason) == VIR_DOMAIN_PAUSED && - reason == VIR_DOMAIN_PAUSED_POSTCOPY))) { - virReportError(VIR_ERR_OPERATION_INVALID, "%s", - _("cannot abort migration in post-copy mode")); - goto endjob; - } - - ret = qemuDomainAbortJobMigration(vm); + reason == VIR_DOMAIN_PAUSED_POSTCOPY))) + ret = qemuDomainAbortJobPostcopy(vm, flags); + else + ret = qemuDomainAbortJobMigration(vm); break; case VIR_ASYNC_JOB_SAVE: diff --git a/src/qemu/qemu_monitor.c b/src/qemu/qemu_monitor.c index 5424fdc4fd..af37ce7d19 100644 --- a/src/qemu/qemu_monitor.c +++ b/src/qemu/qemu_monitor.c @@ -2429,6 +2429,15 @@ qemuMonitorMigrateCancel(qemuMonitor *mon) } +int +qemuMonitorMigratePause(qemuMonitor *mon) +{ + QEMU_CHECK_MONITOR(mon); + + return qemuMonitorJSONMigratePause(mon); +} + + int qemuMonitorQueryDump(qemuMonitor *mon, qemuMonitorDumpStats *stats) diff --git a/src/qemu/qemu_monitor.h b/src/qemu/qemu_monitor.h index 812f8f4dc2..e36ce8b4eb 100644 --- a/src/qemu/qemu_monitor.h +++ b/src/qemu/qemu_monitor.h @@ -927,6 +927,9 @@ int qemuMonitorMigrateToSocket(qemuMonitor *mon, int qemuMonitorMigrateCancel(qemuMonitor *mon); +int +qemuMonitorMigratePause(qemuMonitor *mon); + int qemuMonitorGetDumpGuestMemoryCapability(qemuMonitor *mon, const char *capability); diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 44361ae003..66e0b870b4 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3458,6 +3458,28 @@ int qemuMonitorJSONMigrateCancel(qemuMonitor *mon) } +int +qemuMonitorJSONMigratePause(qemuMonitor *mon) +{ + g_autoptr(virJSONValue) cmd = NULL; + g_autoptr(virJSONValue) reply = NULL; + + if (!(cmd = qemuMonitorJSONMakeCommand("migrate-pause", NULL))) + return -1; + + if (qemuMonitorJSONCommand(mon, cmd, &reply) < 0) + return -1; + + if (qemuMonitorJSONHasError(reply, "DeviceNotFound")) + return -2; + + if (qemuMonitorJSONCheckError(cmd, reply) < 0) + return -1; + + return 0; +} + + /* qemuMonitorJSONQueryDump: * @mon: Monitor pointer * @stats: Monitor dump stats diff --git a/src/qemu/qemu_monitor_json.h b/src/qemu/qemu_monitor_json.h index a60c0bfac4..20c3f63f93 100644 --- a/src/qemu/qemu_monitor_json.h +++ b/src/qemu/qemu_monitor_json.h @@ -208,6 +208,9 @@ qemuMonitorJSONGetSpiceMigrationStatus(qemuMonitor *mon, int qemuMonitorJSONMigrateCancel(qemuMonitor *mon); +int +qemuMonitorJSONMigratePause(qemuMonitor *mon); + int qemuMonitorJSONQueryDump(qemuMonitor *mon, qemuMonitorDumpStats *stats); -- 2.35.1
On Tue, May 10, 2022 at 17:21:39 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 43 +++++++++++++++++++++++++++++------- src/qemu/qemu_monitor.c | 9 ++++++++ src/qemu/qemu_monitor.h | 3 +++ src/qemu/qemu_monitor_json.c | 22 ++++++++++++++++++ src/qemu/qemu_monitor_json.h | 3 +++ 5 files changed, 72 insertions(+), 8 deletions(-)
[...]
diff --git a/src/qemu/qemu_monitor_json.c b/src/qemu/qemu_monitor_json.c index 44361ae003..66e0b870b4 100644 --- a/src/qemu/qemu_monitor_json.c +++ b/src/qemu/qemu_monitor_json.c @@ -3458,6 +3458,28 @@ int qemuMonitorJSONMigrateCancel(qemuMonitor *mon) }
+int +qemuMonitorJSONMigratePause(qemuMonitor *mon) +{ + g_autoptr(virJSONValue) cmd = NULL; + g_autoptr(virJSONValue) reply = NULL; + + if (!(cmd = qemuMonitorJSONMakeCommand("migrate-pause", NULL))) + return -1; + + if (qemuMonitorJSONCommand(mon, cmd, &reply) < 0) + return -1; + + if (qemuMonitorJSONHasError(reply, "DeviceNotFound")) + return -2;
Please add an explanation when this can happen and document the return values in the function comment.
+ + if (qemuMonitorJSONCheckError(cmd, reply) < 0) + return -1; + + return 0; +}
Also add the trivial qemumonitorjsontest case so that this undergoes QMP schema validation, even when it's a trivial command. With both of the above addressed: Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Thu, May 12, 2022 at 14:52:11 +0200, Peter Krempa wrote:
On Tue, May 10, 2022 at 17:21:39 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- src/qemu/qemu_driver.c | 43 +++++++++++++++++++++++++++++------- src/qemu/qemu_monitor.c | 9 ++++++++ src/qemu/qemu_monitor.h | 3 +++ src/qemu/qemu_monitor_json.c | 22 ++++++++++++++++++ src/qemu/qemu_monitor_json.h | 3 +++ 5 files changed, 72 insertions(+), 8 deletions(-)
[...]
[...]
Also add the trivial qemumonitorjsontest case so that this undergoes QMP schema validation, even when it's a trivial command.
With both of the above addressed:
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
One other thing to consider would be to reject the API if the postcopy flag is used but the migration is not post-copy or it's a backup job.
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- docs/manpages/virsh.rst | 8 +++++++- tools/virsh-domain.c | 16 +++++++++++++++- 2 files changed, 22 insertions(+), 2 deletions(-) diff --git a/docs/manpages/virsh.rst b/docs/manpages/virsh.rst index e36d64c164..1bb9044e0a 100644 --- a/docs/manpages/virsh.rst +++ b/docs/manpages/virsh.rst @@ -2053,10 +2053,16 @@ domjobabort :: - domjobabort domain + domjobabort domain [--postcopy] Abort the currently running domain job. +When the job to be aborted is a migration which entered post-copy mode, it +cannot be aborted as none of the hosts involved in migration has a complete +state of the domain. Optional *--postcopy* can be used to interrupt such +migration although doing so may effectively suspend the domain until the +migration is resumed (see also *--postcopy-resume* option of ``migrate``). + domjobinfo ---------- diff --git a/tools/virsh-domain.c b/tools/virsh-domain.c index b0d5b15dff..63a5ee7a34 100644 --- a/tools/virsh-domain.c +++ b/tools/virsh-domain.c @@ -6477,6 +6477,10 @@ static const vshCmdInfo info_domjobabort[] = { static const vshCmdOptDef opts_domjobabort[] = { VIRSH_COMMON_OPT_DOMAIN_FULL(VIR_CONNECT_LIST_DOMAINS_ACTIVE), + {.name = "postcopy", + .type = VSH_OT_BOOL, + .help = N_("interrupt post-copy migration") + }, {.name = NULL} }; @@ -6484,11 +6488,21 @@ static bool cmdDomjobabort(vshControl *ctl, const vshCmd *cmd) { g_autoptr(virshDomain) dom = NULL; + unsigned int flags = 0; + int rc; if (!(dom = virshCommandOptDomain(ctl, cmd, NULL))) return false; - if (virDomainAbortJob(dom) < 0) + if (vshCommandOptBool(cmd, "postcopy")) + flags |= VIR_DOMAIN_ABORT_JOB_POSTCOPY; + + if (flags == 0) + rc = virDomainAbortJob(dom); + else + rc = virDomainAbortJobFlags(dom, flags); + + if (rc < 0) return false; return true; -- 2.35.1
On Tue, May 10, 2022 at 17:21:40 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- docs/manpages/virsh.rst | 8 +++++++- tools/virsh-domain.c | 16 +++++++++++++++- 2 files changed, 22 insertions(+), 2 deletions(-)
Reviewed-by: Peter Krempa <pkrempa@redhat.com>
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- NEWS.rst | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/NEWS.rst b/NEWS.rst index 7903449f9b..bafc70ba90 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -17,6 +17,11 @@ v8.4.0 (unreleased) * **New features** + * qemu: Add support for post-copy recovery + + A new VIR_MIGRATE_POSTCOPY_RESUME flag (virsh migrate --postcopy-resume) + was introduced for fecovering from a failed post-copy migration. + * **Improvements** * **Bug fixes** -- 2.35.1
On Tue, May 10, 2022 at 17:21:41 +0200, Jiri Denemark wrote:
Signed-off-by: Jiri Denemark <jdenemar@redhat.com> --- NEWS.rst | 5 +++++ 1 file changed, 5 insertions(+)
diff --git a/NEWS.rst b/NEWS.rst index 7903449f9b..bafc70ba90 100644 --- a/NEWS.rst +++ b/NEWS.rst @@ -17,6 +17,11 @@ v8.4.0 (unreleased)
* **New features**
+ * qemu: Add support for post-copy recovery + + A new VIR_MIGRATE_POSTCOPY_RESUME flag (virsh migrate --postcopy-resume)
``VIR_MIGRATE_POSTCOPY_RESUME`` and ``virsh migrate --postcopy-resume``
+ was introduced for fecovering from a failed post-copy migration.
s/fecovering/recovering/ Reviewed-by: Peter Krempa <pkrempa@redhat.com>
On Tue, May 10, 2022 at 05:20:21PM +0200, Jiri Denemark wrote:
This series implements a new VIR_MIGRATE_POSTCOPY_RESUME flag (virsh migrate --resume) for recovering from a failed post-copy migration.
You can also fetch the series from my gitlab fork:
git fetch https://gitlab.com/jirkade/libvirt.git post-copy-recovery
Jiri Denemark (80): qemu: Add debug messages to job recovery code qemumonitorjsontest: Test more migration capabilities qemu: Return state from qemuMonitorGetMigrationCapabilities qemu: Enable migration events only when disabled Introduce VIR_DOMAIN_RUNNING_POSTCOPY_FAILED qemu: Keep domain running on dst on failed post-copy migration qemu: Explicitly emit events on post-copy failure qemu: Make qemuDomainCleanupAdd return void conf: Introduce virDomainObjIsFailedPostcopy helper conf: Introduce virDomainObjIsPostcopy helper qemu: Introduce qemuProcessCleanupMigrationJob qemu: Rename qemuDomainObjRestoreJob as qemuDomainObjPreserveJob qemu: Add qemuDomainObjRestoreAsyncJob qemu: Keep migration job active after failed post-copy qemu: Abort failed post-copy when we haven't called Finish yet qemu: Restore failed migration job on reconnect qemu: Restore async job start timestamp on reconnect qemu: Drop forward declarations in migration code qemu: Don't wait for migration job when migration is running qemu: Use switch in qemuDomainGetJobInfoMigrationStats qemu: Fetch paused migration stats qemu: Handle 'postcopy-paused' migration state qemu: Add support for postcopy-recover QEMU migration state qemu: Create domain object at the end of qemuMigrationDstFinish qemu: Move success-only code out of endjob in qemuMigrationDstFinish qemu: Separate success and failure path in qemuMigrationDstFinish qemu: Rename "endjob" label in qemuMigrationDstFinish qemu: Generate migration cookie in Finish phase earlier qemu: Make final part of migration Finish phase reusable qemu: Drop obsolete comment in qemuMigrationDstFinish qemu: Preserve error in qemuMigrationDstFinish qemu: Introduce qemuMigrationDstFinishFresh qemu: Introduce qemuMigrationDstFinishOffline qemu: Separate cookie parsing for qemuMigrationDstFinishOffline qemu: Introduce qemuMigrationDstFinishActive qemu: Handle migration job in qemuMigrationDstFinish qemu: Make final part of migration Confirm phase reusable qemu: Make sure migrationPort is released even in callbacks qemu: Pass qemuDomainJobObj to qemuMigrationDstComplete qemu: Finish completed unattended migration qemu: Ignore missing memory statistics in query-migrate qemu: Improve post-copy migration handling on reconnect qemu: Check flags incompatible with offline migration earlier qemu: Introduce qemuMigrationSrcBeginXML helper qemu: Add new migration phases for post-copy recovery qemu: Separate protocol checks from qemuMigrationJobSetPhase qemu: Make qemuMigrationCheckPhase failure fatal qemu: Refactor qemuDomainObjSetJobPhase qemu: Do not set job owner in qemuMigrationJobSetPhase qemu: Use QEMU_MIGRATION_PHASE_POSTCOPY_FAILED Introduce VIR_MIGRATE_POSTCOPY_RESUME flag virsh: Add --postcopy-resume option for migrate command qemu: Don't set VIR_MIGRATE_PAUSED for post-copy resume qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Begin phase qmeu: Refactor qemuMigrationSrcPerformPhase qemu: Separate starting migration from qemuMigrationSrcRun qemu: Add support for 'resume' parameter of migrate QMP command qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Perform phase qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Confirm phase qemu: Introduce qemuMigrationDstPrepareFresh qemu: Refactor qemuMigrationDstPrepareFresh qemu: Simplify cleanup in qemuMigrationDstPrepareFresh qemu: Add support for migrate-recover QMP command qemu: Rename qemuMigrationSrcCleanup qemu: Refactor qemuMigrationAnyConnectionClosed qemu: Handle incoming migration in qemuMigrationAnyConnectionClosed qemu: Start a migration phase in qemuMigrationAnyConnectionClosed qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Prepare phase qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for Finish phase qemu: Create completed jobData in qemuMigrationSrcComplete qemu: Register qemuProcessCleanupMigrationJob after Begin phase qemu: Call qemuDomainCleanupAdd from qemuMigrationJobContinue qemu: Implement VIR_MIGRATE_POSTCOPY_RESUME for peer-to-peer migration qemu: Enable support for VIR_MIGRATE_POSTCOPY_RESUME Add virDomainAbortJobFlags public API qemu: Implement virDomainAbortJobFlags Add VIR_DOMAIN_ABORT_JOB_POSTCOPY flag for virDomainAbortJobFlags qemu: Implement VIR_DOMAIN_ABORT_JOB_POSTCOPY flag virsh: Add --postcopy option for domjobabort command NEWS: Add support for post-copy recovery
For the patches with simple fixes or once the questions asked by Peter are resolved. But there will be v2 to handle pausing/keeping VM running on dst for failed post-copy migration. Reviewed-by: Pavel Hrdina <phrdina@redhat.com>
participants (5)
-
Daniel P. Berrangé -
Fangge Jin -
Jiri Denemark -
Pavel Hrdina -
Peter Krempa