libvirt-6.5.0 breaks host passthrough migration

Hi all: With 6.4.0, live migration was working fine with Qemu 5.0. After trying out 6.5.0, migration broke with the following error: libvirt.libvirtError: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable CPU device (invtsc flag) I believe I traced the error back to this commit: commit 201bd5db639c063862b0c1b1abfab9a9a7c92591 Author: Jiri Denemark <jdenemar(a)redhat.com&gt; Date: Tue Jun 2 15:34:07 2020 +0200 qemu: Fill default value in //cpu/@migratable attribute Before QEMU introduced migratable CPU property, "-cpu host" included all features that could be enabled on the host, even those which would block migration. In other words, the default was equivalent to migratable=off. When the migratable property was introduced, the default changed to migratable=on. Let's record the default in domain XML. Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com&gt; Reviewed-by: Michal Privoznik <mprivozn(a)redhat.com&gt; Before this change, qemu was still being launched with "-cpu host", which for any somewhat modern version of qemu, defaults to migratable=on. The above comment acknowledges this, however, the implementation chooses the pessimistic and ancient (and no longer applicable!) value of migratable=off: + if (qemuCaps && + def->cpu->mode == VIR_CPU_MODE_HOST_PASSTHROUGH && + !def->cpu->migratable) { + if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_CPU_MIGRATABLE)) + def->cpu->migratable = VIR_TRISTATE_SWITCH_ON; *+ else if (ARCH_IS_X86(def->os.arch))+ def->cpu->migratable = VIR_TRISTATE_SWITCH_OFF;* + } Consequently, after live migration qemu is started with "-cpu host,migratable=off" and this enables invtsc, which then prevents migration as when invtsc is enabled it activates a migration blocker. I think this patch should be reverted. If the migratable state cannot be determined, then it should not be guessed. It should be left as "absent", just as it has been working fine prior to libvirt-6.5.0. Qemu still knows what flags are enabled, so Qemu remains the authority on what can be safely migrated, and what cannot be. I reverted this patch and re-built, and this seems to clear the migration problems. If the user chooses to explicitly specify migratable=yes or migratable=no, then this value should be preserved and passed through to the Qemu "-cpu host,XXX" option. I think it is not a requirement for "migratable=XXX" to be explicit in libvirt. However, if there is some reason I am unaware of, and it is important for libvirt to know, then I think it is important for libvirt to find out the authoritative state rather than guessing. Please start by reverting this patch, so that other people do not get broken in the same way, and I don't need to carry my revert patch. Personally, I think this is important enough to build a 6.5.1. However, since I have a local revert patch in place, I am not waiting for this. Thanks! -- Mark Mielke <mark.mielke(a)gmail.com&gt;