On Fri, Apr 26, 2024 at 10:14:05AM -0300, Fabiano Rosas wrote: Can we drop this var too? Perhaps with block_cleanup_parameters()? -- Peter Xu

On Mon, Apr 29, 2024 at 02:18:57PM -0300, Fabiano Rosas wrote: Right, noticed that it's removed actually in the next patch. But iiuc it can already been removed in this patch. If we want to remove it in the next, logically we should set must_remove_block_options=false here, though.. So maybe easier to just drop it here. -- Peter Xu

On Mon, Apr 29, 2024 at 03:35:02PM -0300, Fabiano Rosas wrote: [1] I'm reading this patch the same as "blk==false" always above [1]. In that case, only if we keep must_remove_block_options=false could we maintain the behavior? Otherwise at the end of migration (or cancellations) QEMU can wrongly clear MIGRATION_CAPABILITY_BLOCK bit if the user set it manually? (But hey, we're discussing whoever applies this patch only without the rest.. definitely not a huge deal :) -- Peter Xu

The 'compress' migration capability enables the old compression code which has shown issues over the years and is thought to be less stable and tested than the more recent multifd-based compression. The old compression code has been deprecated in 8.2 and now is time to remove it. Deprecation commit 864128df46 ("migration: Deprecate old compression method"). Signed-off-by: Fabiano Rosas <farosas(a)suse.de&gt; --- docs/about/deprecated.rst | 11 - docs/about/removed-features.rst | 55 ++++ hw/core/machine.c | 1 - migration/meson.build | 1 - migration/migration-hmp-cmds.c | 47 --- migration/migration.c | 14 - migration/migration.h | 7 - migration/options.c | 164 ---------- migration/ram-compress.c | 564 -------------------------------- migration/ram.c | 151 +-------- qapi/migration.json | 112 ------- tests/qtest/migration-test.c | 139 -------- 12 files changed, 63 insertions(+), 1203 deletions(-) delete mode 100644 migration/ram-compress.c diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst index 3b9b5ba6ef..0fb5c82640 100644 --- a/docs/about/deprecated.rst +++ b/docs/about/deprecated.rst @@ -464,14 +464,3 @@ both, older and future versions of QEMU. The ``blacklist`` config file option has been renamed to ``block-rpcs`` (to be in sync with the renaming of the corresponding command line option). - -Migration ---------- - -old compression method (since 8.2) -'''''''''''''''''''''''''''''''''' - -Compression method fails too much. Too many races. We are going to -remove it if nobody fixes it. For starters, migration-test -compression tests are disabled because they fail randomly. If you need -compression, use multifd compression methods. diff --git a/docs/about/removed-features.rst b/docs/about/removed-features.rst index 9e7d8ee4ff..fba0cfb0b0 100644 --- a/docs/about/removed-features.rst +++ b/docs/about/removed-features.rst @@ -505,6 +505,11 @@ configurations (e.g. -smp 8,sockets=0) is removed since 9.0, users have to ensure that all the topology members described with -smp are greater than zero. +``-global migration.decompress-error-check`` (removed in 9.1) +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Removed along with the ``compression`` migration capability. + User-mode emulator command line arguments ----------------------------------------- @@ -641,6 +646,31 @@ Block migration has been removed. For a replacement, see "QMP invocation for live storage migration with ``blockdev-mirror`` + NBD" in docs/interop/live-block-operations.rst. +``migrate-set-parameter`` ``compress-level`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-zlib-level`` or ``multifd-zstd-level`` instead. + +``migrate-set-parameter`` ``compress-threads`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-channels`` instead. + +``migrate-set-parameter`` ``compress-wait-thread`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Removed with no replacement. + +``migrate-set-parameter`` ``decompress-threads`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-channels`` instead. + +``migrate-set-capability`` ``compress`` option (removed in 9.1) +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-compression`` instead. + Human Monitor Protocol (HMP) commands ------------------------------------- @@ -722,6 +752,31 @@ Block migration has been removed. For a replacement, see "QMP invocation for live storage migration with ``blockdev-mirror`` + NBD" in docs/interop/live-block-operations.rst. +``migrate_set_parameter`` ``compress-level`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-zlib-level`` or ``multifd-zstd-level`` instead. + +``migrate_set_parameter`` ``compress-threads`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-channels`` instead. + +``migrate_set_parameter`` ``compress-wait-thread`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Removed with no replacement. + +``migrate_set_parameter`` ``decompress-threads`` option (removed in 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-channels`` instead. + +``migrate_set_capability`` ``compress`` option (removed in 9.1) +''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''' + +Use ``multifd-compression`` instead. + Host Architectures ------------------ diff --git a/hw/core/machine.c b/hw/core/machine.c index 0dec48e802..60009c639c 100644 --- a/hw/core/machine.c +++ b/hw/core/machine.c @@ -190,7 +190,6 @@ GlobalProperty hw_compat_3_0[] = {}; const size_t hw_compat_3_0_len = G_N_ELEMENTS(hw_compat_3_0); GlobalProperty hw_compat_2_12[] = { - { "migration", "decompress-error-check", "off" }, { "hda-audio", "use-timer", "false" }, { "cirrus-vga", "global-vmstate", "true" }, { "VGA", "global-vmstate", "true" }, diff --git a/migration/meson.build b/migration/meson.build index d7e118f584..8815f80837 100644 --- a/migration/meson.build +++ b/migration/meson.build @@ -23,7 +23,6 @@ system_ss.add(files( 'multifd.c', 'multifd-zlib.c', 'multifd-zero-page.c', - 'ram-compress.c', 'options.c', 'postcopy-ram.c', 'savevm.c', diff --git a/migration/migration-hmp-cmds.c b/migration/migration-hmp-cmds.c index 8040f1a46f..2499414235 100644 --- a/migration/migration-hmp-cmds.c +++ b/migration/migration-hmp-cmds.c @@ -46,8 +46,6 @@ static void migration_global_dump(Monitor *mon) ms->send_configuration ? "on" : "off"); monitor_printf(mon, "send-section-footer: %s\n", ms->send_section_footer ? "on" : "off"); - monitor_printf(mon, "decompress-error-check: %s\n", - ms->decompress_error_check ? "on" : "off"); monitor_printf(mon, "clear-bitmap-shift: %u\n", ms->clear_bitmap_shift); } @@ -162,19 +160,6 @@ void hmp_info_migrate(Monitor *mon, const QDict *qdict) info->xbzrle_cache->overflow); } - if (info->compression) { - monitor_printf(mon, "compression pages: %" PRIu64 " pages\n", - info->compression->pages); - monitor_printf(mon, "compression busy: %" PRIu64 "\n", - info->compression->busy); - monitor_printf(mon, "compression busy rate: %0.2f\n", - info->compression->busy_rate); - monitor_printf(mon, "compressed size: %" PRIu64 " kbytes\n", - info->compression->compressed_size >> 10); - monitor_printf(mon, "compression rate: %0.2f\n", - info->compression->compression_rate); - } - if (info->has_cpu_throttle_percentage) { monitor_printf(mon, "cpu throttle percentage: %" PRIu64 "\n", info->cpu_throttle_percentage); @@ -263,22 +248,6 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict) monitor_printf(mon, "%s: %" PRIu64 " ms\n", MigrationParameter_str(MIGRATION_PARAMETER_ANNOUNCE_STEP), params->announce_step); - assert(params->has_compress_level); - monitor_printf(mon, "%s: %u\n", - MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_LEVEL), - params->compress_level); - assert(params->has_compress_threads); - monitor_printf(mon, "%s: %u\n", - MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_THREADS), - params->compress_threads); - assert(params->has_compress_wait_thread); - monitor_printf(mon, "%s: %s\n", - MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_WAIT_THREAD), - params->compress_wait_thread ? "on" : "off"); - assert(params->has_decompress_threads); - monitor_printf(mon, "%s: %u\n", - MigrationParameter_str(MIGRATION_PARAMETER_DECOMPRESS_THREADS), - params->decompress_threads); assert(params->has_throttle_trigger_threshold); monitor_printf(mon, "%s: %u\n", MigrationParameter_str(MIGRATION_PARAMETER_THROTTLE_TRIGGER_THRESHOLD), @@ -520,22 +489,6 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict) } switch (val) { - case MIGRATION_PARAMETER_COMPRESS_LEVEL: - p->has_compress_level = true; - visit_type_uint8(v, param, &p->compress_level, &err); - break; - case MIGRATION_PARAMETER_COMPRESS_THREADS: - p->has_compress_threads = true; - visit_type_uint8(v, param, &p->compress_threads, &err); - break; - case MIGRATION_PARAMETER_COMPRESS_WAIT_THREAD: - p->has_compress_wait_thread = true; - visit_type_bool(v, param, &p->compress_wait_thread, &err); - break; - case MIGRATION_PARAMETER_DECOMPRESS_THREADS: - p->has_decompress_threads = true; - visit_type_uint8(v, param, &p->decompress_threads, &err); - break; case MIGRATION_PARAMETER_THROTTLE_TRIGGER_THRESHOLD: p->has_throttle_trigger_threshold = true; visit_type_uint8(v, param, &p->throttle_trigger_threshold, &err); diff --git a/migration/migration.c b/migration/migration.c index 26017c9a2c..5dad66b810 100644 --- a/migration/migration.c +++ b/migration/migration.c @@ -27,7 +27,6 @@ #include "sysemu/cpu-throttle.h" #include "rdma.h" #include "ram.h" -#include "ram-compress.h" #include "migration/global_state.h" #include "migration/misc.h" #include "migration.h" @@ -352,7 +351,6 @@ void migration_incoming_state_destroy(void) struct MigrationIncomingState *mis = migration_incoming_get_current(); multifd_recv_cleanup(); - compress_threads_load_cleanup(); if (mis->to_src_file) { /* Tell source that we are done */ @@ -645,10 +643,6 @@ static void qemu_start_incoming_migration(const char *uri, bool has_channels, } #ifdef CONFIG_RDMA } else if (addr->transport == MIGRATION_ADDRESS_TYPE_RDMA) { - if (migrate_compress()) { - error_setg(errp, "RDMA and compression can't be used together"); - return; - } if (migrate_xbzrle()) { error_setg(errp, "RDMA and XBZRLE can't be used together"); return; @@ -739,11 +733,6 @@ process_incoming_migration_co(void *opaque) assert(mis->from_src_file); - if (compress_threads_load_setup(mis->from_src_file)) { - error_report("Failed to setup decompress threads"); - goto fail; - } - mis->largest_page_size = qemu_ram_pagesize_largest(); postcopy_state_set(POSTCOPY_INCOMING_NONE); migrate_set_state(&mis->state, MIGRATION_STATUS_SETUP, @@ -800,7 +789,6 @@ fail: qemu_fclose(mis->from_src_file); multifd_recv_cleanup(); - compress_threads_load_cleanup(); exit(EXIT_FAILURE); } @@ -1174,8 +1162,6 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s) info->xbzrle_cache->overflow = xbzrle_counters.overflow; } - populate_compress(info); - if (cpu_throttle_active()) { info->has_cpu_throttle_percentage = true; info->cpu_throttle_percentage = cpu_throttle_get_percentage(); diff --git a/migration/migration.h b/migration/migration.h index 8045e39c26..4432e8b0bb 100644 --- a/migration/migration.h +++ b/migration/migration.h @@ -393,13 +393,6 @@ struct MigrationState { /* Needed by postcopy-pause state */ QemuSemaphore postcopy_pause_sem; - /* - * Whether we abort the migration if decompression errors are - * detected at the destination. It is left at false for qemu - * older than 3.0, since only newer qemu sends streams that - * do not trigger spurious decompression errors. - */ - bool decompress_error_check; /* * This variable only affects behavior when postcopy preempt mode is * enabled. diff --git a/migration/options.c b/migration/options.c index cc2599ae4a..5ab5b6d85d 100644 --- a/migration/options.c +++ b/migration/options.c @@ -40,13 +40,6 @@ * for sending the last part */ #define DEFAULT_MIGRATE_SET_DOWNTIME 300 -/* Default compression thread count */ -#define DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT 8 -/* Default decompression thread count, usually decompression is at - * least 4 times as fast as compression.*/ -#define DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT 2 -/*0: means nocompress, 1: best speed, ... 9: best compress ratio */ -#define DEFAULT_MIGRATE_COMPRESS_LEVEL 1 /* Define default autoconverge cpu throttle migration parameters */ #define DEFAULT_MIGRATE_THROTTLE_TRIGGER_THRESHOLD 50 #define DEFAULT_MIGRATE_CPU_THROTTLE_INITIAL 20 @@ -92,8 +85,6 @@ Property migration_properties[] = { send_configuration, true), DEFINE_PROP_BOOL("send-section-footer", MigrationState, send_section_footer, true), - DEFINE_PROP_BOOL("decompress-error-check", MigrationState, - decompress_error_check, true), DEFINE_PROP_BOOL("multifd-flush-after-each-section", MigrationState, multifd_flush_after_each_section, false), DEFINE_PROP_UINT8("x-clear-bitmap-shift", MigrationState, @@ -102,17 +93,6 @@ Property migration_properties[] = { preempt_pre_7_2, false), /* Migration parameters */ - DEFINE_PROP_UINT8("x-compress-level", MigrationState, - parameters.compress_level, - DEFAULT_MIGRATE_COMPRESS_LEVEL), - DEFINE_PROP_UINT8("x-compress-threads", MigrationState, - parameters.compress_threads, - DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT), - DEFINE_PROP_BOOL("x-compress-wait-thread", MigrationState, - parameters.compress_wait_thread, true), - DEFINE_PROP_UINT8("x-decompress-threads", MigrationState, - parameters.decompress_threads, - DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT), DEFINE_PROP_UINT8("x-throttle-trigger-threshold", MigrationState, parameters.throttle_trigger_threshold, DEFAULT_MIGRATE_THROTTLE_TRIGGER_THRESHOLD), @@ -188,7 +168,6 @@ Property migration_properties[] = { DEFINE_PROP_MIG_CAP("x-rdma-pin-all", MIGRATION_CAPABILITY_RDMA_PIN_ALL), DEFINE_PROP_MIG_CAP("x-auto-converge", MIGRATION_CAPABILITY_AUTO_CONVERGE), DEFINE_PROP_MIG_CAP("x-zero-blocks", MIGRATION_CAPABILITY_ZERO_BLOCKS), - DEFINE_PROP_MIG_CAP("x-compress", MIGRATION_CAPABILITY_COMPRESS), DEFINE_PROP_MIG_CAP("x-events", MIGRATION_CAPABILITY_EVENTS), DEFINE_PROP_MIG_CAP("x-postcopy-ram", MIGRATION_CAPABILITY_POSTCOPY_RAM), DEFINE_PROP_MIG_CAP("x-postcopy-preempt", @@ -231,13 +210,6 @@ bool migrate_colo(void) return s->capabilities[MIGRATION_CAPABILITY_X_COLO]; } -bool migrate_compress(void) -{ - MigrationState *s = migrate_get_current(); - - return s->capabilities[MIGRATION_CAPABILITY_COMPRESS]; -} - bool migrate_dirty_bitmaps(void) { MigrationState *s = migrate_get_current(); @@ -451,7 +423,6 @@ INITIALIZE_MIGRATE_CAPS_SET(check_caps_background_snapshot, MIGRATION_CAPABILITY_AUTO_CONVERGE, MIGRATION_CAPABILITY_RELEASE_RAM, MIGRATION_CAPABILITY_RDMA_PIN_ALL, - MIGRATION_CAPABILITY_COMPRESS, MIGRATION_CAPABILITY_XBZRLE, MIGRATION_CAPABILITY_X_COLO, MIGRATION_CAPABILITY_VALIDATE_UUID, @@ -476,11 +447,6 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) ERRP_GUARD(); MigrationIncomingState *mis = migration_incoming_get_current(); - if (new_caps[MIGRATION_CAPABILITY_COMPRESS]) { - warn_report("old compression method is deprecated;" - " use multifd compression methods instead"); - } - #ifndef CONFIG_REPLICATION if (new_caps[MIGRATION_CAPABILITY_X_COLO]) { error_setg(errp, "QEMU compiled without replication module" @@ -549,7 +515,6 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) #ifdef CONFIG_LINUX if (new_caps[MIGRATION_CAPABILITY_ZERO_COPY_SEND] && (!new_caps[MIGRATION_CAPABILITY_MULTIFD] || - new_caps[MIGRATION_CAPABILITY_COMPRESS] || new_caps[MIGRATION_CAPABILITY_XBZRLE] || migrate_multifd_compression() || migrate_tls())) { @@ -571,17 +536,6 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) return false; } - /* - * Preempt mode requires urgent pages to be sent in separate - * channel, OTOH compression logic will disorder all pages into - * different compression channels, which is not compatible with the - * preempt assumptions on channel assignments. - */ - if (new_caps[MIGRATION_CAPABILITY_COMPRESS]) { - error_setg(errp, "Postcopy preempt not compatible with compress"); - return false; - } - if (migrate_incoming_started()) { error_setg(errp, "Postcopy preempt must be set before incoming starts"); @@ -590,10 +544,6 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) } if (new_caps[MIGRATION_CAPABILITY_MULTIFD]) { - if (new_caps[MIGRATION_CAPABILITY_COMPRESS]) { - error_setg(errp, "Multifd is not compatible with compress"); - return false; - } if (migrate_incoming_started()) { error_setg(errp, "Multifd must be set before incoming starts"); return false; @@ -628,13 +578,6 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) } } - if (new_caps[MIGRATION_CAPABILITY_COMPRESS]) { - if (new_caps[MIGRATION_CAPABILITY_XBZRLE]) { - error_setg(errp, "Compression is not compatible with xbzrle"); - return false; - } - } - if (new_caps[MIGRATION_CAPABILITY_MAPPED_RAM]) { if (new_caps[MIGRATION_CAPABILITY_XBZRLE]) { error_setg(errp, @@ -642,12 +585,6 @@ bool migrate_caps_check(bool *old_caps, bool *new_caps, Error **errp) return false; } - if (new_caps[MIGRATION_CAPABILITY_COMPRESS]) { - error_setg(errp, - "Mapped-ram migration is incompatible with compression"); - return false; - } - if (new_caps[MIGRATION_CAPABILITY_POSTCOPY_RAM]) { error_setg(errp, "Mapped-ram migration is incompatible with postcopy"); @@ -744,27 +681,6 @@ uint32_t migrate_checkpoint_delay(void) return s->parameters.x_checkpoint_delay; } -int migrate_compress_level(void) -{ - MigrationState *s = migrate_get_current(); - - return s->parameters.compress_level; -} - -int migrate_compress_threads(void) -{ - MigrationState *s = migrate_get_current(); - - return s->parameters.compress_threads; -} - -int migrate_compress_wait_thread(void) -{ - MigrationState *s = migrate_get_current(); - - return s->parameters.compress_wait_thread; -} - uint8_t migrate_cpu_throttle_increment(void) { MigrationState *s = migrate_get_current(); @@ -786,13 +702,6 @@ bool migrate_cpu_throttle_tailslow(void) return s->parameters.cpu_throttle_tailslow; } -int migrate_decompress_threads(void) -{ - MigrationState *s = migrate_get_current(); - - return s->parameters.decompress_threads; -} - uint64_t migrate_downtime_limit(void) { MigrationState *s = migrate_get_current(); @@ -938,14 +847,6 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp) /* TODO use QAPI_CLONE() instead of duplicating it inline */ params = g_malloc0(sizeof(*params)); - params->has_compress_level = true; - params->compress_level = s->parameters.compress_level; - params->has_compress_threads = true; - params->compress_threads = s->parameters.compress_threads; - params->has_compress_wait_thread = true; - params->compress_wait_thread = s->parameters.compress_wait_thread; - params->has_decompress_threads = true; - params->decompress_threads = s->parameters.decompress_threads; params->has_throttle_trigger_threshold = true; params->throttle_trigger_threshold = s->parameters.throttle_trigger_threshold; params->has_cpu_throttle_initial = true; @@ -1014,10 +915,6 @@ void migrate_params_init(MigrationParameters *params) params->tls_creds = g_strdup(""); /* Set has_* up only for parameter checks */ - params->has_compress_level = true; - params->has_compress_threads = true; - params->has_compress_wait_thread = true; - params->has_decompress_threads = true; params->has_throttle_trigger_threshold = true; params->has_cpu_throttle_initial = true; params->has_cpu_throttle_increment = true; @@ -1050,27 +947,6 @@ bool migrate_params_check(MigrationParameters *params, Error **errp) { ERRP_GUARD(); - if (params->has_compress_level && - (params->compress_level > 9)) { - error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_level", - "a value between 0 and 9"); - return false; - } - - if (params->has_compress_threads && (params->compress_threads < 1)) { - error_setg(errp, QERR_INVALID_PARAMETER_VALUE, - "compress_threads", - "a value between 1 and 255"); - return false; - } - - if (params->has_decompress_threads && (params->decompress_threads < 1)) { - error_setg(errp, QERR_INVALID_PARAMETER_VALUE, - "decompress_threads", - "a value between 1 and 255"); - return false; - } - if (params->has_throttle_trigger_threshold && (params->throttle_trigger_threshold < 1 || params->throttle_trigger_threshold > 100)) { @@ -1244,22 +1120,6 @@ static void migrate_params_test_apply(MigrateSetParameters *params, /* TODO use QAPI_CLONE() instead of duplicating it inline */ - if (params->has_compress_level) { - dest->compress_level = params->compress_level; - } - - if (params->has_compress_threads) { - dest->compress_threads = params->compress_threads; - } - - if (params->has_compress_wait_thread) { - dest->compress_wait_thread = params->compress_wait_thread; - } - - if (params->has_decompress_threads) { - dest->decompress_threads = params->decompress_threads; - } - if (params->has_throttle_trigger_threshold) { dest->throttle_trigger_threshold = params->throttle_trigger_threshold; } @@ -1364,30 +1224,6 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp) /* TODO use QAPI_CLONE() instead of duplicating it inline */ - if (params->has_compress_level) { - warn_report("old compression is deprecated;" - " use multifd compression methods instead"); - s->parameters.compress_level = params->compress_level; - } - - if (params->has_compress_threads) { - warn_report("old compression is deprecated;" - " use multifd compression methods instead"); - s->parameters.compress_threads = params->compress_threads; - } - - if (params->has_compress_wait_thread) { - warn_report("old compression is deprecated;" - " use multifd compression methods instead"); - s->parameters.compress_wait_thread = params->compress_wait_thread; - } - - if (params->has_decompress_threads) { - warn_report("old compression is deprecated;" - " use multifd compression methods instead"); - s->parameters.decompress_threads = params->decompress_threads; - } - if (params->has_throttle_trigger_threshold) { s->parameters.throttle_trigger_threshold = params->throttle_trigger_threshold; } diff --git a/migration/ram-compress.c b/migration/ram-compress.c deleted file mode 100644 index fa4388f6a6..0000000000 --- a/migration/ram-compress.c +++ /dev/null @@ -1,564 +0,0 @@ -/* - * QEMU System Emulator - * - * Copyright (c) 2003-2008 Fabrice Bellard - * Copyright (c) 2011-2015 Red Hat Inc - * - * Authors: - * Juan Quintela <quintela(a)redhat.com&gt; - * - * Permission is hereby granted, free of charge, to any person obtaining a copy - * of this software and associated documentation files (the "Software"), to deal - * in the Software without restriction, including without limitation the rights - * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell - * copies of the Software, and to permit persons to whom the Software is - * furnished to do so, subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in - * all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL - * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER - * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, - * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN - * THE SOFTWARE. - */ - -#include "qemu/osdep.h" -#include "qemu/cutils.h" - -#include "ram-compress.h" - -#include "qemu/error-report.h" -#include "qemu/stats64.h" -#include "migration.h" -#include "options.h" -#include "io/channel-null.h" -#include "exec/target_page.h" -#include "exec/ramblock.h" -#include "ram.h" -#include "migration-stats.h" - -static struct { - int64_t pages; - int64_t busy; - double busy_rate; - int64_t compressed_size; - double compression_rate; - /* compression statistics since the beginning of the period */ - /* amount of count that no free thread to compress data */ - uint64_t compress_thread_busy_prev; - /* amount bytes after compression */ - uint64_t compressed_size_prev; - /* amount of compressed pages */ - uint64_t compress_pages_prev; -} compression_counters; - -static CompressParam *comp_param; -static QemuThread *compress_threads; -/* comp_done_cond is used to wake up the migration thread when - * one of the compression threads has finished the compression. - * comp_done_lock is used to co-work with comp_done_cond. - */ -static QemuMutex comp_done_lock; -static QemuCond comp_done_cond; - -struct DecompressParam { - bool done; - bool quit; - QemuMutex mutex; - QemuCond cond; - void *des; - uint8_t *compbuf; - int len; - z_stream stream; -}; -typedef struct DecompressParam DecompressParam; - -static QEMUFile *decomp_file; -static DecompressParam *decomp_param; -static QemuThread *decompress_threads; -static QemuMutex decomp_done_lock; -static QemuCond decomp_done_cond; - -static CompressResult do_compress_ram_page(QEMUFile *f, z_stream *stream, - RAMBlock *block, ram_addr_t offset, - uint8_t *source_buf); - -static void *do_data_compress(void *opaque) -{ - CompressParam *param = opaque; - RAMBlock *block; - ram_addr_t offset; - CompressResult result; - - qemu_mutex_lock(&param->mutex); - while (!param->quit) { - if (param->trigger) { - block = param->block; - offset = param->offset; - param->trigger = false; - qemu_mutex_unlock(&param->mutex); - - result = do_compress_ram_page(param->file, &param->stream, - block, offset, param->originbuf); - - qemu_mutex_lock(&comp_done_lock); - param->done = true; - param->result = result; - qemu_cond_signal(&comp_done_cond); - qemu_mutex_unlock(&comp_done_lock); - - qemu_mutex_lock(&param->mutex); - } else { - qemu_cond_wait(&param->cond, &param->mutex); - } - } - qemu_mutex_unlock(&param->mutex); - - return NULL; -} - -void compress_threads_save_cleanup(void) -{ - int i, thread_count; - - if (!migrate_compress() || !comp_param) { - return; - } - - thread_count = migrate_compress_threads(); - for (i = 0; i < thread_count; i++) { - /* - * we use it as a indicator which shows if the thread is - * properly init'd or not - */ - if (!comp_param[i].file) { - break; - } - - qemu_mutex_lock(&comp_param[i].mutex); - comp_param[i].quit = true; - qemu_cond_signal(&comp_param[i].cond); - qemu_mutex_unlock(&comp_param[i].mutex); - - qemu_thread_join(compress_threads + i); - qemu_mutex_destroy(&comp_param[i].mutex); - qemu_cond_destroy(&comp_param[i].cond); - deflateEnd(&comp_param[i].stream); - g_free(comp_param[i].originbuf); - qemu_fclose(comp_param[i].file); - comp_param[i].file = NULL; - } - qemu_mutex_destroy(&comp_done_lock); - qemu_cond_destroy(&comp_done_cond); - g_free(compress_threads); - g_free(comp_param); - compress_threads = NULL; - comp_param = NULL; -} - -int compress_threads_save_setup(void) -{ - int i, thread_count; - - if (!migrate_compress()) { - return 0; - } - thread_count = migrate_compress_threads(); - compress_threads = g_new0(QemuThread, thread_count); - comp_param = g_new0(CompressParam, thread_count); - qemu_cond_init(&comp_done_cond); - qemu_mutex_init(&comp_done_lock); - for (i = 0; i < thread_count; i++) { - comp_param[i].originbuf = g_try_malloc(qemu_target_page_size()); - if (!comp_param[i].originbuf) { - goto exit; - } - - if (deflateInit(&comp_param[i].stream, - migrate_compress_level()) != Z_OK) { - g_free(comp_param[i].originbuf); - goto exit; - } - - /* comp_param[i].file is just used as a dummy buffer to save data, - * set its ops to empty. - */ - comp_param[i].file = qemu_file_new_output( - QIO_CHANNEL(qio_channel_null_new())); - comp_param[i].done = true; - comp_param[i].quit = false; - qemu_mutex_init(&comp_param[i].mutex); - qemu_cond_init(&comp_param[i].cond); - qemu_thread_create(compress_threads + i, "compress", - do_data_compress, comp_param + i, - QEMU_THREAD_JOINABLE); - } - return 0; - -exit: - compress_threads_save_cleanup(); - return -1; -} - -static CompressResult do_compress_ram_page(QEMUFile *f, z_stream *stream, - RAMBlock *block, ram_addr_t offset, - uint8_t *source_buf) -{ - uint8_t *p = block->host + offset; - size_t page_size = qemu_target_page_size(); - int ret; - - assert(qemu_file_buffer_empty(f)); - - if (buffer_is_zero(p, page_size)) { - return RES_ZEROPAGE; - } - - /* - * copy it to a internal buffer to avoid it being modified by VM - * so that we can catch up the error during compression and - * decompression - */ - memcpy(source_buf, p, page_size); - ret = qemu_put_compression_data(f, stream, source_buf, page_size); - if (ret < 0) { - qemu_file_set_error(migrate_get_current()->to_dst_file, ret); - error_report("compressed data failed!"); - qemu_fflush(f); - return RES_NONE; - } - return RES_COMPRESS; -} - -static inline void compress_reset_result(CompressParam *param) -{ - param->result = RES_NONE; - param->block = NULL; - param->offset = 0; -} - -void compress_flush_data(void) -{ - int thread_count = migrate_compress_threads(); - - if (!migrate_compress()) { - return; - } - - qemu_mutex_lock(&comp_done_lock); - for (int i = 0; i < thread_count; i++) { - while (!comp_param[i].done) { - qemu_cond_wait(&comp_done_cond, &comp_done_lock); - } - } - qemu_mutex_unlock(&comp_done_lock); - - for (int i = 0; i < thread_count; i++) { - qemu_mutex_lock(&comp_param[i].mutex); - if (!comp_param[i].quit) { - CompressParam *param = &comp_param[i]; - compress_send_queued_data(param); - assert(qemu_file_buffer_empty(param->file)); - compress_reset_result(param); - } - qemu_mutex_unlock(&comp_param[i].mutex); - } -} - -static inline void set_compress_params(CompressParam *param, RAMBlock *block, - ram_addr_t offset) -{ - param->block = block; - param->offset = offset; - param->trigger = true; -} - -/* - * Return true when it compress a page - */ -bool compress_page_with_multi_thread(RAMBlock *block, ram_addr_t offset, - int (send_queued_data(CompressParam *))) -{ - int thread_count; - bool wait = migrate_compress_wait_thread(); - - thread_count = migrate_compress_threads(); - qemu_mutex_lock(&comp_done_lock); - - while (true) { - for (int i = 0; i < thread_count; i++) { - if (comp_param[i].done) { - CompressParam *param = &comp_param[i]; - qemu_mutex_lock(&param->mutex); - param->done = false; - send_queued_data(param); - assert(qemu_file_buffer_empty(param->file)); - compress_reset_result(param); - set_compress_params(param, block, offset); - - qemu_cond_signal(&param->cond); - qemu_mutex_unlock(&param->mutex); - qemu_mutex_unlock(&comp_done_lock); - return true; - } - } - if (!wait) { - qemu_mutex_unlock(&comp_done_lock); - compression_counters.busy++; - return false; - } - /* - * wait for a free thread if the user specifies - * 'compress-wait-thread', otherwise we will post the page out - * in the main thread as normal page. - */ - qemu_cond_wait(&comp_done_cond, &comp_done_lock); - } -} - -/* return the size after decompression, or negative value on error */ -static int -qemu_uncompress_data(z_stream *stream, uint8_t *dest, size_t dest_len, - const uint8_t *source, size_t source_len) -{ - int err; - - err = inflateReset(stream); - if (err != Z_OK) { - return -1; - } - - stream->avail_in = source_len; - stream->next_in = (uint8_t *)source; - stream->avail_out = dest_len; - stream->next_out = dest; - - err = inflate(stream, Z_NO_FLUSH); - if (err != Z_STREAM_END) { - return -1; - } - - return stream->total_out; -} - -static void *do_data_decompress(void *opaque) -{ - DecompressParam *param = opaque; - unsigned long pagesize; - uint8_t *des; - int len, ret; - - qemu_mutex_lock(&param->mutex); - while (!param->quit) { - if (param->des) { - des = param->des; - len = param->len; - param->des = 0; - qemu_mutex_unlock(&param->mutex); - - pagesize = qemu_target_page_size(); - - ret = qemu_uncompress_data(&param->stream, des, pagesize, - param->compbuf, len); - if (ret < 0 && migrate_get_current()->decompress_error_check) { - error_report("decompress data failed"); - qemu_file_set_error(decomp_file, ret); - } - - qemu_mutex_lock(&decomp_done_lock); - param->done = true; - qemu_cond_signal(&decomp_done_cond); - qemu_mutex_unlock(&decomp_done_lock); - - qemu_mutex_lock(&param->mutex); - } else { - qemu_cond_wait(&param->cond, &param->mutex); - } - } - qemu_mutex_unlock(&param->mutex); - - return NULL; -} - -int wait_for_decompress_done(void) -{ - if (!migrate_compress()) { - return 0; - } - - int thread_count = migrate_decompress_threads(); - qemu_mutex_lock(&decomp_done_lock); - for (int i = 0; i < thread_count; i++) { - while (!decomp_param[i].done) { - qemu_cond_wait(&decomp_done_cond, &decomp_done_lock); - } - } - qemu_mutex_unlock(&decomp_done_lock); - return qemu_file_get_error(decomp_file); -} - -void compress_threads_load_cleanup(void) -{ - int i, thread_count; - - if (!migrate_compress()) { - return; - } - thread_count = migrate_decompress_threads(); - for (i = 0; i < thread_count; i++) { - /* - * we use it as a indicator which shows if the thread is - * properly init'd or not - */ - if (!decomp_param[i].compbuf) { - break; - } - - qemu_mutex_lock(&decomp_param[i].mutex); - decomp_param[i].quit = true; - qemu_cond_signal(&decomp_param[i].cond); - qemu_mutex_unlock(&decomp_param[i].mutex); - } - for (i = 0; i < thread_count; i++) { - if (!decomp_param[i].compbuf) { - break; - } - - qemu_thread_join(decompress_threads + i); - qemu_mutex_destroy(&decomp_param[i].mutex); - qemu_cond_destroy(&decomp_param[i].cond); - inflateEnd(&decomp_param[i].stream); - g_free(decomp_param[i].compbuf); - decomp_param[i].compbuf = NULL; - } - g_free(decompress_threads); - g_free(decomp_param); - decompress_threads = NULL; - decomp_param = NULL; - decomp_file = NULL; -} - -int compress_threads_load_setup(QEMUFile *f) -{ - int i, thread_count; - - if (!migrate_compress()) { - return 0; - } - - /* - * set compression_counters memory to zero for a new migration - */ - memset(&compression_counters, 0, sizeof(compression_counters)); - - thread_count = migrate_decompress_threads(); - decompress_threads = g_new0(QemuThread, thread_count); - decomp_param = g_new0(DecompressParam, thread_count); - qemu_mutex_init(&decomp_done_lock); - qemu_cond_init(&decomp_done_cond); - decomp_file = f; - for (i = 0; i < thread_count; i++) { - if (inflateInit(&decomp_param[i].stream) != Z_OK) { - goto exit; - } - - size_t compbuf_size = compressBound(qemu_target_page_size()); - decomp_param[i].compbuf = g_malloc0(compbuf_size); - qemu_mutex_init(&decomp_param[i].mutex); - qemu_cond_init(&decomp_param[i].cond); - decomp_param[i].done = true; - decomp_param[i].quit = false; - qemu_thread_create(decompress_threads + i, "decompress", - do_data_decompress, decomp_param + i, - QEMU_THREAD_JOINABLE); - } - return 0; -exit: - compress_threads_load_cleanup(); - return -1; -} - -void decompress_data_with_multi_threads(QEMUFile *f, void *host, int len) -{ - int thread_count = migrate_decompress_threads(); - QEMU_LOCK_GUARD(&decomp_done_lock); - while (true) { - for (int i = 0; i < thread_count; i++) { - if (decomp_param[i].done) { - decomp_param[i].done = false; - qemu_mutex_lock(&decomp_param[i].mutex); - qemu_get_buffer(f, decomp_param[i].compbuf, len); - decomp_param[i].des = host; - decomp_param[i].len = len; - qemu_cond_signal(&decomp_param[i].cond); - qemu_mutex_unlock(&decomp_param[i].mutex); - return; - } - } - qemu_cond_wait(&decomp_done_cond, &decomp_done_lock); - } -} - -void populate_compress(MigrationInfo *info) -{ - if (!migrate_compress()) { - return; - } - info->compression = g_malloc0(sizeof(*info->compression)); - info->compression->pages = compression_counters.pages; - info->compression->busy = compression_counters.busy; - info->compression->busy_rate = compression_counters.busy_rate; - info->compression->compressed_size = compression_counters.compressed_size; - info->compression->compression_rate = compression_counters.compression_rate; -} - -uint64_t compress_ram_pages(void) -{ - return compression_counters.pages; -} - -void update_compress_thread_counts(const CompressParam *param, int bytes_xmit) -{ - ram_transferred_add(bytes_xmit); - - if (param->result == RES_ZEROPAGE) { - stat64_add(&mig_stats.zero_pages, 1); - return; - } - - /* 8 means a header with RAM_SAVE_FLAG_CONTINUE. */ - compression_counters.compressed_size += bytes_xmit - 8; - compression_counters.pages++; -} - -void compress_update_rates(uint64_t page_count) -{ - if (!migrate_compress()) { - return; - } - compression_counters.busy_rate = (double)(compression_counters.busy - - compression_counters.compress_thread_busy_prev) / page_count; - compression_counters.compress_thread_busy_prev = - compression_counters.busy; - - double compressed_size = compression_counters.compressed_size - - compression_counters.compressed_size_prev; - if (compressed_size) { - double uncompressed_size = (compression_counters.pages - - compression_counters.compress_pages_prev) * - qemu_target_page_size(); - - /* Compression-Ratio = Uncompressed-size / Compressed-size */ - compression_counters.compression_rate = - uncompressed_size / compressed_size; - - compression_counters.compress_pages_prev = - compression_counters.pages; - compression_counters.compressed_size_prev = - compression_counters.compressed_size; - } -} diff --git a/migration/ram.c b/migration/ram.c index 5debb2e455..4ecc9f7f48 100644 --- a/migration/ram.c +++ b/migration/ram.c @@ -33,7 +33,6 @@ #include "qemu/madvise.h" #include "qemu/main-loop.h" #include "xbzrle.h" -#include "ram-compress.h" #include "ram.h" #include "migration.h" #include "migration-stats.h" @@ -77,9 +76,10 @@ * worked for pages that were filled with the same char. We switched * it to only search for the zero value. And to avoid confusion with * RAM_SAVE_FLAG_COMPRESS_PAGE just rename it. - */ -/* - * RAM_SAVE_FLAG_FULL was obsoleted in 2009, it can be reused now + * + * RAM_SAVE_FLAG_FULL was obsoleted in 2009. + * + * RAM_SAVE_FLAG_COMPRESS_PAGE (0x100) was removed in QEMU 9.1. */ #define RAM_SAVE_FLAG_FULL 0x01 #define RAM_SAVE_FLAG_ZERO 0x02 @@ -89,7 +89,6 @@ #define RAM_SAVE_FLAG_CONTINUE 0x20 #define RAM_SAVE_FLAG_XBZRLE 0x40 /* 0x80 is reserved in rdma.h for RAM_SAVE_FLAG_HOOK */ -#define RAM_SAVE_FLAG_COMPRESS_PAGE 0x100 #define RAM_SAVE_FLAG_MULTIFD_FLUSH 0x200 /* We can't use any flag that is bigger than 0x200 */ @@ -949,7 +948,7 @@ uint64_t ram_get_total_transferred_pages(void) { return stat64_get(&mig_stats.normal_pages) + stat64_get(&mig_stats.zero_pages) + - compress_ram_pages() + xbzrle_counters.pages; + xbzrle_counters.pages; } static void migration_update_rates(RAMState *rs, int64_t end_time) @@ -982,7 +981,6 @@ static void migration_update_rates(RAMState *rs, int64_t end_time) rs->xbzrle_pages_prev = xbzrle_counters.pages; rs->xbzrle_bytes_prev = xbzrle_counters.bytes; } - compress_update_rates(page_count); } /* @@ -1288,41 +1286,6 @@ static int ram_save_multifd_page(RAMBlock *block, ram_addr_t offset) return 1; } -int compress_send_queued_data(CompressParam *param) -{ - PageSearchStatus *pss = &ram_state->pss[RAM_CHANNEL_PRECOPY]; - MigrationState *ms = migrate_get_current(); - QEMUFile *file = ms->to_dst_file; - int len = 0; - - RAMBlock *block = param->block; - ram_addr_t offset = param->offset; - - if (param->result == RES_NONE) { - return 0; - } - - assert(block == pss->last_sent_block); - - if (param->result == RES_ZEROPAGE) { - assert(qemu_file_buffer_empty(param->file)); - len += save_page_header(pss, file, block, offset | RAM_SAVE_FLAG_ZERO); - qemu_put_byte(file, 0); - len += 1; - ram_release_page(block->idstr, offset); - } else if (param->result == RES_COMPRESS) { - assert(!qemu_file_buffer_empty(param->file)); - len += save_page_header(pss, file, block, - offset | RAM_SAVE_FLAG_COMPRESS_PAGE); - len += qemu_put_qemu_file(file, param->file); - } else { - abort(); - } - - update_compress_thread_counts(param, len); - - return len; -} #define PAGE_ALL_CLEAN 0 #define PAGE_TRY_AGAIN 1 @@ -1374,16 +1337,6 @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss) qemu_fflush(f); } } - /* - * If memory migration starts over, we will meet a dirtied page - * which may still exists in compression threads's ring, so we - * should flush the compressed data to make sure the new page - * is not overwritten by the old one in the destination. - * - * Also If xbzrle is on, stop using the data compression at this - * point. In theory, xbzrle can do better than compression. - */ - compress_flush_data(); /* Hit the end of the list */ pss->block = QLIST_FIRST_RCU(&ram_list.blocks); @@ -2034,37 +1987,6 @@ int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len, return 0; } -/* - * try to compress the page before posting it out, return true if the page - * has been properly handled by compression, otherwise needs other - * paths to handle it - */ -static bool save_compress_page(RAMState *rs, PageSearchStatus *pss, - ram_addr_t offset) -{ - if (!migrate_compress()) { - return false; - } - - /* - * When starting the process of a new block, the first page of - * the block should be sent out before other pages in the same - * block, and all the pages in last block should have been sent - * out, keeping this order is important, because the 'cont' flag - * is used to avoid resending the block name. - * - * We post the fist page as normal page as compression will take - * much CPU resource. - */ - if (pss->block != pss->last_sent_block) { - compress_flush_data(); - return false; - } - - return compress_page_with_multi_thread(pss->block, offset, - compress_send_queued_data); -} - /** * ram_save_target_page_legacy: save one target page * @@ -2082,10 +2004,6 @@ static int ram_save_target_page_legacy(RAMState *rs, PageSearchStatus *pss) return res; } - if (save_compress_page(rs, pss, offset)) { - return 1; - } - if (save_zero_page(rs, pss, offset)) { return 1; } @@ -2470,7 +2388,6 @@ static void ram_save_cleanup(void *opaque) ram_bitmaps_destroy(); xbzrle_cleanup(); - compress_threads_save_cleanup(); ram_state_cleanup(rsp); g_free(migration_ops); migration_ops = NULL; @@ -3089,15 +3006,9 @@ static int ram_save_setup(QEMUFile *f, void *opaque, Error **errp) RAMBlock *block; int ret, max_hg_page_size; - if (compress_threads_save_setup()) { - error_setg(errp, "%s: failed to start compress threads", __func__); - return -1; - } - /* migration has already setup the bitmap, reuse it. */ if (!migration_in_colo_state()) { if (ram_init_all(rsp, errp) != 0) { - compress_threads_save_cleanup(); return -1; } } @@ -3268,14 +3179,6 @@ static int ram_save_iterate(QEMUFile *f, void *opaque) rs->target_page_count += pages; - /* - * During postcopy, it is necessary to make sure one whole host - * page is sent in one chunk. - */ - if (migrate_postcopy_ram()) { - compress_flush_data(); - } - /* * we want to check in the 1st loop, just in case it was the 1st * time and we had to sync the dirty bitmap. @@ -3374,8 +3277,6 @@ static int ram_save_complete(QEMUFile *f, void *opaque) } qemu_mutex_unlock(&rs->bitmap_mutex); - compress_flush_data(); - ret = rdma_registration_stop(f, RAM_CONTROL_FINISH); if (ret < 0) { qemu_file_set_error(f, ret); @@ -3789,7 +3690,6 @@ int ram_load_postcopy(QEMUFile *f, int channel) void *place_source = NULL; RAMBlock *block = NULL; uint8_t ch; - int len; addr = qemu_get_be64(f); @@ -3806,8 +3706,7 @@ int ram_load_postcopy(QEMUFile *f, int channel) addr &= TARGET_PAGE_MASK; trace_ram_load_postcopy_loop(channel, (uint64_t)addr, flags); - if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE | - RAM_SAVE_FLAG_COMPRESS_PAGE)) { + if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE)) { block = ram_block_from_stream(mis, f, flags, channel); if (!block) { ret = -EINVAL; @@ -3902,16 +3801,6 @@ int ram_load_postcopy(QEMUFile *f, int channel) TARGET_PAGE_SIZE); } break; - case RAM_SAVE_FLAG_COMPRESS_PAGE: - tmp_page->all_zero = false; - len = qemu_get_be32(f); - if (len < 0 || len > compressBound(TARGET_PAGE_SIZE)) { - error_report("Invalid compressed data length: %d", len); - ret = -EINVAL; - break; - } - decompress_data_with_multi_threads(f, page_buffer, len); - break; case RAM_SAVE_FLAG_MULTIFD_FLUSH: multifd_recv_sync_main(); break; @@ -3929,11 +3818,6 @@ int ram_load_postcopy(QEMUFile *f, int channel) break; } - /* Got the whole host page, wait for decompress before placing. */ - if (place_needed) { - ret |= wait_for_decompress_done(); - } - /* Detect for any possible file errors */ if (!ret && qemu_file_get_error(f)) { ret = qemu_file_get_error(f); @@ -4238,11 +4122,7 @@ static int parse_ramblocks(QEMUFile *f, ram_addr_t total_ram_bytes) static int ram_load_precopy(QEMUFile *f) { MigrationIncomingState *mis = migration_incoming_get_current(); - int flags = 0, ret = 0, invalid_flags = 0, len = 0, i = 0; - - if (!migrate_compress()) { - invalid_flags |= RAM_SAVE_FLAG_COMPRESS_PAGE; - } + int flags = 0, ret = 0, invalid_flags = 0, i = 0; if (migrate_mapped_ram()) { invalid_flags |= (RAM_SAVE_FLAG_HOOK | RAM_SAVE_FLAG_MULTIFD_FLUSH | @@ -4279,16 +4159,12 @@ static int ram_load_precopy(QEMUFile *f) if (flags & invalid_flags) { error_report("Unexpected RAM flags: %d", flags & invalid_flags); - if (flags & invalid_flags & RAM_SAVE_FLAG_COMPRESS_PAGE) { - error_report("Received an unexpected compressed page"); - } - ret = -EINVAL; break; } if (flags & (RAM_SAVE_FLAG_ZERO | RAM_SAVE_FLAG_PAGE | - RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) { + RAM_SAVE_FLAG_XBZRLE)) { RAMBlock *block = ram_block_from_stream(mis, f, flags, RAM_CHANNEL_PRECOPY); @@ -4357,16 +4233,6 @@ static int ram_load_precopy(QEMUFile *f) qemu_get_buffer(f, host, TARGET_PAGE_SIZE); break; - case RAM_SAVE_FLAG_COMPRESS_PAGE: - len = qemu_get_be32(f); - if (len < 0 || len > compressBound(TARGET_PAGE_SIZE)) { - error_report("Invalid compressed data length: %d", len); - ret = -EINVAL; - break; - } - decompress_data_with_multi_threads(f, host, len); - break; - case RAM_SAVE_FLAG_XBZRLE: if (load_xbzrle(f, addr, host) < 0) { error_report("Failed to decompress XBZRLE page at " @@ -4408,7 +4274,6 @@ static int ram_load_precopy(QEMUFile *f) } } - ret |= wait_for_decompress_done(); return ret; } diff --git a/qapi/migration.json b/qapi/migration.json index 10e13d0b17..fc4b13050a 100644 --- a/qapi/migration.json +++ b/qapi/migration.json @@ -229,10 +229,6 @@ # This is only present when the postcopy-blocktime migration # capability is enabled. (Since 3.0) # -# @compression: migration compression statistics, only returned if -# compression feature is on and status is 'active' or 'completed' -# (Since 3.1) -# # @socket-address: Only used for tcp, to know what the real port is # (Since 4.0) # @@ -257,13 +253,6 @@ # average memory load of the virtual CPU indirectly. Note that # zero means guest doesn't dirty memory. (Since 8.1) # -# Features: -# -# @deprecated: Member @compression is deprecated because it is -# unreliable and untested. It is recommended to use multifd -# migration, which offers an alternative compression -# implementation that is reliable and tested. -# # Since: 0.14 ## { 'struct': 'MigrationInfo', @@ -279,7 +268,6 @@ '*blocked-reasons': ['str'], '*postcopy-blocktime': 'uint32', '*postcopy-vcpu-blocktime': ['uint32'], - '*compression': { 'type': 'CompressionStats', 'features': [ 'deprecated' ] }, '*socket-address': ['SocketAddress'], '*dirty-limit-throttle-time-per-round': 'uint64', '*dirty-limit-ring-full-time': 'uint64'} } @@ -401,14 +389,6 @@ # capability on the source VM. The feature is disabled by default. # (since 1.6) # -# @compress: Use multiple compression threads to accelerate live -# migration. This feature can help to reduce the migration -# traffic, by sending compressed pages. Please note that if -# compress and xbzrle are both on, compress only takes effect in -# the ram bulk stage, after that, it will be disabled and only -# xbzrle takes effect, this can help to minimize migration -# traffic. The feature is disabled by default. (since 2.4) -# # @events: generate events for each migration state change (since 2.4) # # @auto-converge: If enabled, QEMU will automatically throttle down @@ -491,18 +471,12 @@ # # Features: # -# @deprecated: Member @compress is deprecated because it is unreliable -# and untested. It is recommended to use multifd migration, which -# offers an alternative compression implementation that is -# reliable and tested. -# # @unstable: Members @x-colo and @x-ignore-shared are experimental. # # Since: 1.2 ## { 'enum': 'MigrationCapability', 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks', - { 'name': 'compress', 'features': [ 'deprecated' ] }, 'events', 'postcopy-ram', { 'name': 'x-colo', 'features': [ 'unstable' ] }, 'release-ram', @@ -562,7 +536,6 @@ # {"state": false, "capability": "rdma-pin-all"}, # {"state": false, "capability": "auto-converge"}, # {"state": false, "capability": "zero-blocks"}, -# {"state": false, "capability": "compress"}, # {"state": true, "capability": "events"}, # {"state": false, "capability": "postcopy-ram"}, # {"state": false, "capability": "x-colo"} @@ -710,27 +683,6 @@ # @announce-step: Increase in delay (in milliseconds) between # subsequent packets in the announcement (Since 4.0) # -# @compress-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 9, -# where 0 means no compression, 1 means the best compression -# speed, and 9 means best compression ratio which will consume -# more CPU. -# -# @compress-threads: Set compression thread count to be used in live -# migration, the compression thread count is an integer between 1 -# and 255. -# -# @compress-wait-thread: Controls behavior when all compression -# threads are currently busy. If true (default), wait for a free -# compression thread to become available; otherwise, send the page -# uncompressed. (Since 3.1) -# -# @decompress-threads: Set decompression thread count to be used in -# live migration, the decompression thread count is an integer -# between 1 and 255. Usually, decompression is at least 4 times as -# fast as compression, so set the decompress-threads to the number -# about 1/4 of compress-threads is adequate. -# # @throttle-trigger-threshold: The ratio of bytes_dirty_period and # bytes_xfer_period to trigger throttling. It is expressed as # percentage. The default value is 50. (Since 5.0) @@ -862,10 +814,6 @@ # # Features: # -# @deprecated: Members @compress-level, @compress-threads, -# @decompress-threads and @compress-wait-thread are deprecated -# because @compression is deprecated. -# # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # @@ -874,10 +822,6 @@ { 'enum': 'MigrationParameter', 'data': ['announce-initial', 'announce-max', 'announce-rounds', 'announce-step', - { 'name': 'compress-level', 'features': [ 'deprecated' ] }, - { 'name': 'compress-threads', 'features': [ 'deprecated' ] }, - { 'name': 'decompress-threads', 'features': [ 'deprecated' ] }, - { 'name': 'compress-wait-thread', 'features': [ 'deprecated' ] }, 'throttle-trigger-threshold', 'cpu-throttle-initial', 'cpu-throttle-increment', 'cpu-throttle-tailslow', @@ -909,27 +853,6 @@ # @announce-step: Increase in delay (in milliseconds) between # subsequent packets in the announcement (Since 4.0) # -# @compress-level: Set the compression level to be used in live -# migration, the compression level is an integer between 0 and 9, -# where 0 means no compression, 1 means the best compression -# speed, and 9 means best compression ratio which will consume -# more CPU. -# -# @compress-threads: Set compression thread count to be used in live -# migration, the compression thread count is an integer between 1 -# and 255. -# -# @compress-wait-thread: Controls behavior when all compression -# threads are currently busy. If true (default), wait for a free -# compression thread to become available; otherwise, send the page -# uncompressed. (Since 3.1) -# -# @decompress-threads: Set decompression thread count to be used in -# live migration, the decompression thread count is an integer -# between 1 and 255. Usually, decompression is at least 4 times as -# fast as compression, so set the decompress-threads to the number -# about 1/4 of compress-threads is adequate. -# # @throttle-trigger-threshold: The ratio of bytes_dirty_period and # bytes_xfer_period to trigger throttling. It is expressed as # percentage. The default value is 50. (Since 5.0) @@ -1061,10 +984,6 @@ # # Features: # -# @deprecated: Members @compress-level, @compress-threads, -# @decompress-threads and @compress-wait-thread are deprecated -# because @compression is deprecated. -# # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # @@ -1078,14 +997,6 @@ '*announce-max': 'size', '*announce-rounds': 'size', '*announce-step': 'size', - '*compress-level': { 'type': 'uint8', - 'features': [ 'deprecated' ] }, - '*compress-threads': { 'type': 'uint8', - 'features': [ 'deprecated' ] }, - '*compress-wait-thread': { 'type': 'bool', - 'features': [ 'deprecated' ] }, - '*decompress-threads': { 'type': 'uint8', - 'features': [ 'deprecated' ] }, '*throttle-trigger-threshold': 'uint8', '*cpu-throttle-initial': 'uint8', '*cpu-throttle-increment': 'uint8', @@ -1145,17 +1056,6 @@ # @announce-step: Increase in delay (in milliseconds) between # subsequent packets in the announcement (Since 4.0) # -# @compress-level: compression level -# -# @compress-threads: compression thread count -# -# @compress-wait-thread: Controls behavior when all compression -# threads are currently busy. If true (default), wait for a free -# compression thread to become available; otherwise, send the page -# uncompressed. (Since 3.1) -# -# @decompress-threads: decompression thread count -# # @throttle-trigger-threshold: The ratio of bytes_dirty_period and # bytes_xfer_period to trigger throttling. It is expressed as # percentage. The default value is 50. (Since 5.0) @@ -1283,10 +1183,6 @@ # # Features: # -# @deprecated: Members @compress-level, @compress-threads, -# @decompress-threads and @compress-wait-thread are deprecated -# because @compression is deprecated. -# # @unstable: Members @x-checkpoint-delay and # @x-vcpu-dirty-limit-period are experimental. # @@ -1297,14 +1193,6 @@ '*announce-max': 'size', '*announce-rounds': 'size', '*announce-step': 'size', - '*compress-level': { 'type': 'uint8', - 'features': [ 'deprecated' ] }, - '*compress-threads': { 'type': 'uint8', - 'features': [ 'deprecated' ] }, - '*compress-wait-thread': { 'type': 'bool', - 'features': [ 'deprecated' ] }, - '*decompress-threads': { 'type': 'uint8', - 'features': [ 'deprecated' ] }, '*throttle-trigger-threshold': 'uint8', '*cpu-throttle-initial': 'uint8', '*cpu-throttle-increment': 'uint8', diff --git a/tests/qtest/migration-test.c b/tests/qtest/migration-test.c index 5d6d8cd634..7a1345f80f 100644 --- a/tests/qtest/migration-test.c +++ b/tests/qtest/migration-test.c @@ -427,38 +427,6 @@ static void migrate_set_parameter_str(QTestState *who, const char *parameter, migrate_check_parameter_str(who, parameter, value); } -static long long migrate_get_parameter_bool(QTestState *who, - const char *parameter) -{ - QDict *rsp; - int result; - - rsp = qtest_qmp_assert_success_ref( - who, "{ 'execute': 'query-migrate-parameters' }"); - result = qdict_get_bool(rsp, parameter); - qobject_unref(rsp); - return !!result; -} - -static void migrate_check_parameter_bool(QTestState *who, const char *parameter, - int value) -{ - int result; - - result = migrate_get_parameter_bool(who, parameter); - g_assert_cmpint(result, ==, value); -} - -static void migrate_set_parameter_bool(QTestState *who, const char *parameter, - int value) -{ - qtest_qmp_assert_success(who, - "{ 'execute': 'migrate-set-parameters'," - "'arguments': { %s: %i } }", - parameter, value); - migrate_check_parameter_bool(who, parameter, value); -} - static void migrate_ensure_non_converge(QTestState *who) { /* Can't converge with 1ms downtime + 3 mbs bandwidth limit */ @@ -1240,36 +1208,6 @@ test_migrate_tls_x509_finish(QTestState *from, #endif /* CONFIG_TASN1 */ #endif /* CONFIG_GNUTLS */ -static void * -test_migrate_compress_start(QTestState *from, - QTestState *to) -{ - migrate_set_parameter_int(from, "compress-level", 1); - migrate_set_parameter_int(from, "compress-threads", 4); - migrate_set_parameter_bool(from, "compress-wait-thread", true); - migrate_set_parameter_int(to, "decompress-threads", 4); - - migrate_set_capability(from, "compress", true); - migrate_set_capability(to, "compress", true); - - return NULL; -} - -static void * -test_migrate_compress_nowait_start(QTestState *from, - QTestState *to) -{ - migrate_set_parameter_int(from, "compress-level", 9); - migrate_set_parameter_int(from, "compress-threads", 1); - migrate_set_parameter_bool(from, "compress-wait-thread", false); - migrate_set_parameter_int(to, "decompress-threads", 1); - - migrate_set_capability(from, "compress", true); - migrate_set_capability(to, "compress", true); - - return NULL; -} - static int migrate_postcopy_prepare(QTestState **from_ptr, QTestState **to_ptr, MigrateCommon *args) @@ -1370,15 +1308,6 @@ static void test_postcopy_suspend(void) test_postcopy_common(&args); } -static void test_postcopy_compress(void) -{ - MigrateCommon args = { - .start_hook = test_migrate_compress_start - }; - - test_postcopy_common(&args); -} - static void test_postcopy_preempt(void) { MigrateCommon args = { @@ -1561,15 +1490,6 @@ static void test_postcopy_recovery(void) test_postcopy_recovery_common(&args); } -static void test_postcopy_recovery_compress(void) -{ - MigrateCommon args = { - .start_hook = test_migrate_compress_start - }; - - test_postcopy_recovery_common(&args); -} - #ifndef _WIN32 static void test_postcopy_recovery_double_fail(void) { @@ -2027,48 +1947,6 @@ static void test_precopy_unix_xbzrle(void) test_precopy_common(&args); } -static void test_precopy_unix_compress(void) -{ - g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs); - MigrateCommon args = { - .connect_uri = uri, - .listen_uri = uri, - .start_hook = test_migrate_compress_start, - /* - * Test that no invalid thread state is left over from - * the previous iteration. - */ - .iterations = 2, - /* - * We make sure the compressor can always work well even if guest - * memory is changing. See commit 34ab9e9743 where we used to fix - * a bug when only trigger-able with guest memory changing. - */ - .live = true, - }; - - test_precopy_common(&args); -} - -static void test_precopy_unix_compress_nowait(void) -{ - g_autofree char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs); - MigrateCommon args = { - .connect_uri = uri, - .listen_uri = uri, - .start_hook = test_migrate_compress_nowait_start, - /* - * Test that no invalid thread state is left over from - * the previous iteration. - */ - .iterations = 2, - /* Same reason for the wait version of precopy compress test */ - .live = true, - }; - - test_precopy_common(&args); -} - static void test_precopy_file(void) { g_autofree char *uri = g_strdup_printf("file:%s/%s", tmpfs, @@ -3597,12 +3475,6 @@ int main(int argc, char **argv) test_postcopy_preempt); migration_test_add("/migration/postcopy/preempt/recovery/plain", test_postcopy_preempt_recovery); - if (getenv("QEMU_TEST_FLAKY_TESTS")) { - migration_test_add("/migration/postcopy/compress/plain", - test_postcopy_compress); - migration_test_add("/migration/postcopy/recovery/compress/plain", - test_postcopy_recovery_compress); - } #ifndef _WIN32 migration_test_add("/migration/postcopy/recovery/double-failures", test_postcopy_recovery_double_fail); @@ -3623,17 +3495,6 @@ int main(int argc, char **argv) test_precopy_unix_plain); migration_test_add("/migration/precopy/unix/xbzrle", test_precopy_unix_xbzrle); - /* - * Compression fails from time to time. - * Put test here but don't enable it until everything is fixed. - */ - if (getenv("QEMU_TEST_FLAKY_TESTS")) { - migration_test_add("/migration/precopy/unix/compress/wait", - test_precopy_unix_compress); - migration_test_add("/migration/precopy/unix/compress/nowait", - test_precopy_unix_compress_nowait); - } - migration_test_add("/migration/precopy/file", test_precopy_file); migration_test_add("/migration/precopy/file/offset", -- 2.35.3

The fd: URI can currently trigger two different types of migration, a TCP migration using sockets and a file migration using a plain file. This is in conflict with the recently introduced (8.2) QMP migrate API that takes structured data as JSON-like format. We cannot keep the same backend for both types of migration because with the new API the code is more tightly coupled to the type of transport. This means a TCP migration must use the 'socket' transport and a file migration must use the 'file' transport. If we keep allowing fd: when using a file, this creates an issue when the user converts the old-style (fd:) to the new style ("transport": "socket") invocation because the file descriptor in question has previously been allowed to be either a plain file or a socket. To avoid creating too much confusion, we can simply deprecate the fd: + file usage, which is thought to be rarely used currently and instead establish a 1:1 correspondence between fd: URI and socket transport, and file: URI and file transport. Signed-off-by: Fabiano Rosas <farosas(a)suse.de&gt; --- docs/about/deprecated.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/docs/about/deprecated.rst b/docs/about/deprecated.rst index 0fb5c82640..813f7996fe 100644 --- a/docs/about/deprecated.rst +++ b/docs/about/deprecated.rst @@ -464,3 +464,17 @@ both, older and future versions of QEMU. The ``blacklist`` config file option has been renamed to ``block-rpcs`` (to be in sync with the renaming of the corresponding command line option). + +Migration +--------- + +``fd:`` URI when used for file migration (since 9.1) +'''''''''''''''''''''''''''''''''''''''''''''''''''' + +The ``fd:`` URI can currently provide a file descriptor that +references either a socket or a plain file. These are two different +types of migration. In order to reduce ambiguity, the ``fd:`` URI +usage of providing a file descriptor to a plain file has been +deprecated in favor of explicitly using the ``file:`` URI with the +file descriptor being passed as an ``fdset``. Refer to the ``add-fd`` +command documentation for details on the ``fdset`` usage. -- 2.35.3

Peter Xu <peterx(a)redhat.com&gt; writes: Yes, good point. Something along those lines. We currently use fd_is_socket(): bool fd_is_socket(int fd) { int optval; socklen_t optlen = sizeof(optval); return !getsockopt(fd, SOL_SOCKET, SO_TYPE, &optval, &optlen); } I'm thinking of this in fd_start_outgoing_migation(): if (!fd_is_socket(fd)) { warn_report("fd: migration to a file is deprecated." " Use file: instead."); }