Hi all: With 6.4.0, live migration was working fine with Qemu 5.0. After trying out 6.5.0, migration broke with the following error: libvirt.libvirtError: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable CPU device (invtsc flag) I believe I traced the error back to this commit: commit 201bd5db639c063862b0c1b1abfab9a9a7c92591 Author: Jiri Denemark <jdenemar(a)redhat.com&gt; Date: Tue Jun 2 15:34:07 2020 +0200 qemu: Fill default value in //cpu/@migratable attribute Before QEMU introduced migratable CPU property, "-cpu host" included all features that could be enabled on the host, even those which would block migration. In other words, the default was equivalent to migratable=off. When the migratable property was introduced, the default changed to migratable=on. Let's record the default in domain XML. Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com&gt; Reviewed-by: Michal Privoznik <mprivozn(a)redhat.com&gt; Before this change, qemu was still being launched with "-cpu host", which for any somewhat modern version of qemu, defaults to migratable=on. The above comment acknowledges this, however, the implementation chooses the pessimistic and ancient (and no longer applicable!) value of migratable=off: + if (qemuCaps && + def->cpu->mode == VIR_CPU_MODE_HOST_PASSTHROUGH && + !def->cpu->migratable) { + if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_CPU_MIGRATABLE)) + def->cpu->migratable = VIR_TRISTATE_SWITCH_ON; *+ else if (ARCH_IS_X86(def->os.arch))+ def->cpu->migratable = VIR_TRISTATE_SWITCH_OFF;* + } Consequently, after live migration qemu is started with "-cpu host,migratable=off" and this enables invtsc, which then prevents migration as when invtsc is enabled it activates a migration blocker. I think this patch should be reverted. If the migratable state cannot be determined, then it should not be guessed. It should be left as "absent", just as it has been working fine prior to libvirt-6.5.0. Qemu still knows what flags are enabled, so Qemu remains the authority on what can be safely migrated, and what cannot be. I reverted this patch and re-built, and this seems to clear the migration problems. If the user chooses to explicitly specify migratable=yes or migratable=no, then this value should be preserved and passed through to the Qemu "-cpu host,XXX" option. I think it is not a requirement for "migratable=XXX" to be explicit in libvirt. However, if there is some reason I am unaware of, and it is important for libvirt to know, then I think it is important for libvirt to find out the authoritative state rather than guessing. Please start by reverting this patch, so that other people do not get broken in the same way, and I don't need to carry my revert patch. Personally, I think this is important enough to build a 6.5.1. However, since I have a local revert patch in place, I am not waiting for this. Thanks! -- Mark Mielke <mark.mielke(a)gmail.com&gt;

Hi all: With 6.4.0, live migration was working fine with Qemu 5.0. After trying out 6.5.0, migration broke with the following error: libvirt.libvirtError: internal error: unable to execute QEMU command 'migrate': State blocked by non-migratable CPU device (invtsc flag)

I believe I traced the error back to this commit: commit 201bd5db639c063862b0c1b1abfab9a9a7c92591 Author: Jiri Denemark <jdenemar(a)redhat.com&gt; Date: Tue Jun 2 15:34:07 2020 +0200 qemu: Fill default value in //cpu/@migratable attribute Before QEMU introduced migratable CPU property, "-cpu host" included all features that could be enabled on the host, even those which would block migration. In other words, the default was equivalent to migratable=off. When the migratable property was introduced, the default changed to migratable=on. Let's record the default in domain XML. Signed-off-by: Jiri Denemark <jdenemar(a)redhat.com&gt; Reviewed-by: Michal Privoznik <mprivozn(a)redhat.com&gt; Before this change, qemu was still being launched with "-cpu host", which for any somewhat modern version of qemu, defaults to migratable=on. The above comment acknowledges this, however, the implementation chooses the pessimistic and ancient (and no longer applicable!) value of migratable=off: + if (qemuCaps && + def->cpu->mode == VIR_CPU_MODE_HOST_PASSTHROUGH && + !def->cpu->migratable) { + if (virQEMUCapsGet(qemuCaps, QEMU_CAPS_CPU_MIGRATABLE)) + def->cpu->migratable = VIR_TRISTATE_SWITCH_ON; *+ else if (ARCH_IS_X86(def->os.arch))+ def->cpu->migratable = VIR_TRISTATE_SWITCH_OFF;* + }

I think it is not a requirement for "migratable=XXX" to be explicit in libvirt. However, if there is some reason I am unaware of, and it is important for libvirt to know, then I think it is important for libvirt to find out the authoritative state rather than guessing.

On Fri, Jul 10, 2020 at 7:14 AM Jiri Denemark <jdenemar(a)redhat.com&gt; wrote: > On Sun, Jul 05, 2020 at 12:45:55 -0400, Mark Mielke wrote: > > With 6.4.0, live migration was working fine with Qemu 5.0. After trying > out > > 6.5.0, migration broke with the following error: > > > > libvirt.libvirtError: internal error: unable to execute QEMU command > > 'migrate': State blocked by non-migratable CPU device (invtsc flag) > > Could you please describe the reproducer steps? For example, was the > domain you're trying to migrate already running when you upgrade libvirt > or is it freshly started by the new libvirt? > The original case was: 1) Machine X running libvirt 6.4.0 + qemu 5.0 2) Machine Y running libvirt 6.5.0 + qemu 5.0 3) Live migration from X to Y works. Guest appears fine. 4) Upgrade Machine X from libvirt 6.4.0 to 6.5.0 and reboot. 5) Live migration from Y to X fails with the message shown.

In each case, live migration was done with OpenStack Train directing libvirt + qemu. And it would be helpful to see the <cpu> element as shown by virsh > dumpxml before you try to start the domain as well as the QEMU command > line libvirt used to start the domain (in > /var/log/libvirt/qemu/$VM.log). > The <cpu> element looks like this: <cpu mode='host-passthrough' check='none'> <topology sockets='1' dies='1' cores='4' threads='2'/> </cpu> The QEMU command line is very long, and includes details I would avoid publishing publicly unless you need them. The "-cpu" portion is just: -cpu host The QEMU command line itself is generated from libvirt, which is directed by OpenStack Train.

I wasn't sure what QEMU_CAPS_CPU_MIGRATABLE represents. I initially suspected what you are saying, but since it apparently did not work the way I expected, I then presumed it does not work the way I expected. :-) Is QEMU_CAPS_CPU_MIGRATABLE only from the <cpu> element? If so, doesn't this mean that it is not explicitly listed for host-passthrough, and this means the check is not detecting whether it is enabled or not properly?

I think it can go either way. There is also convention over configuration as a competing principle. However, I also prefer explicit. Just, it needs to be correct, otherwise explicit can be very bad, as it seems in my case. :-)

On Fri, Jul 10, 2020 at 7:48 AM Mark Mielke <mark.mielke(a)gmail.com&gt; wrote: > On Fri, Jul 10, 2020 at 7:14 AM Jiri Denemark <jdenemar(a)redhat.com&gt; > wrote: > >> The implementation seems to be doing exactly what the commit message >> > says. The migratable=off default should be used only when QEMU does not >> support -cpu host,migratable=on|off, that is only when QEMU is very old. >> Every non-ancient version of libvirt should have the >> QEMU_CAPS_CPU_MIGRATABLE set and thus this code should choose >> migrateble=on default. >> > QEMU_CAPS_CPU_MIGRATABLE only from the <cpu> element? If so, doesn't this > mean that it is not explicitly listed for host-passthrough, and this means > the check is not detecting whether it is enabled or not properly? > Trying to understand what is going on more - I see "migratable" seems to be ok when launching a new machine, but the failure scenario was live migration from 6.4.0 to 6.5.0. Is this because the QEMU_CAPS_CPU_MIGRATABLE was not filled in for 6.4.0, and live migration grabs the capabilities from the source, where the absence of this capability makes it presume an older Qemu in the above code?

On Sat, Jul 11, 2020 at 13:44:19 -0400, Mark Mielke wrote: > On Sat, Jul 11, 2020 at 6:04 AM Mark Mielke <mark.mielke(a)gmail.com&gt; wrote: > > > On Fri, Jul 10, 2020 at 7:48 AM Mark Mielke <mark.mielke(a)gmail.com&gt; wrote: > > > >> On Fri, Jul 10, 2020 at 7:14 AM Jiri Denemark <jdenemar(a)redhat.com&gt; > >> wrote: > >> > >>> The implementation seems to be doing exactly what the commit message > >>> > >> says. The migratable=off default should be used only when QEMU does not > >>> support -cpu host,migratable=on|off, that is only when QEMU is very old. > >>> Every non-ancient version of libvirt should have the > >>> QEMU_CAPS_CPU_MIGRATABLE set and thus this code should choose > >>> migrateble=on default. > >>> > >> QEMU_CAPS_CPU_MIGRATABLE only from the <cpu> element? If so, doesn't this > >> mean that it is not explicitly listed for host-passthrough, and this means > >> the check is not detecting whether it is enabled or not properly? > >> > > Trying to understand what is going on more - I see "migratable" seems to > > be ok when launching a new machine, but the failure scenario was live > > migration from 6.4.0 to 6.5.0. > > > > Is this because the QEMU_CAPS_CPU_MIGRATABLE was not filled in for 6.4.0, > > and live migration grabs the capabilities from the source, where the > > absence of this capability makes it presume an older Qemu in the above code? > > > > Sorry all - I am having trouble reproducing now. The expected use cases are > now working. > > Is it possible that the "migratable" flag might have been missing on some > of the instances, although migration worked fine, and despite having used > Qemu 4.2 and Qemu 5.0? When an updated libvirtd which knows about this new capability starts, it would reprobe all QEMU capabilities (lazily, i.e., once they are needed). However, if there is a running domain, libvirt will use cached capabilities probed when the domain was started. I suspect migrating such domain could be a problem. I'll try to reproduce locally.