Re: [libvirt] How to best handle the reoccurring of rom changes breaking cross version migrations?
Ping - since there wasn't any reply so far - any best practices one could share? Let me add a TL;DR: - bump of ipxe rom versions change the size of virtio-net-pci.rom - that breaks on migration "Length mismatch" I'd guess the size of that rom has to be fixed up on the fly, but if that is really ok and how/where is the question. Also to +1 on bad things for today - I made this a cross post to libvirt in case there is one that has done that in the past. On Mon, Aug 28, 2017 at 4:36 PM, Christian Ehrhardt < christian.ehrhardt@canonical.com> wrote:
Hi, migration issues due to rom changes seem to occur over and over in past years [1], [2],[3],[4],[5]. From the past I know several workarounds (like just truncating to the bigger size) but all have various deficiencies.
But OTOH rom's will always change due to fixes in them. And recently I found one such change [6] that affects the next Ubuntu release and wonder what the ?right?, well lets say best way to fix it would be. Current issue: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 0x40000 in != 0x80000: Invalid argument Due to efi-virtio.rom growing over 256k due to an update to a newer ipxe from upstream.
I beg your pardon (for not being educated enough on this yet), but I want to avoid to go a route that is fixing it in a sub-optimal way and ask for some guidance. There might be better ways in the modern qemu of today than there were in the past. TL;DR I look for the best way (if any) to declare in the new qemu: "I know that the old size was smaller, let me fix that up on migration".
I'll try on my own as I expect the machine type structures/mechanisms (and we have Ubuntu specific types that could encapsulate the rom status from the ipxe package to be coupled with the type) have all that is needed. Yet me almost randomly trying around there surely isn't the best way to go - so if there is some existing case or a short hint at what in there might be the best way to fixup a changed rom size on a migration I'd be very happy to hear about that.
[1]: https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1536331 [2]: https://lists.gnu.org/archive/html/qemu-devel/2016-01/msg01881.html [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1293566 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1090093 [5]: https://forge.univention.org/bugzilla/show_bug.cgi?id=38877 [6]: https://bugs.launchpad.net/ubuntu/+source/ipxe/+bug/1713490
P.S: As everybody else I don't mind so much on reverse migration to older releases
-- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
-- Christian Ehrhardt Software Engineer, Ubuntu Server Canonical Ltd
On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
Ping - since there wasn't any reply so far - any best practices one could share?
Let me add a TL;DR: - bump of ipxe rom versions change the size of virtio-net-pci.rom - that breaks on migration "Length mismatch"
I'd guess the size of that rom has to be fixed up on the fly, but if that is really ok and how/where is the question.
The actual ROM contents will be transferred in the migration stream, so the fact that the target host has ROMs with different content is not important. The key thing that matters is that QEMU the target host loads the ROMs at the same location, so that when the ROM contents is overwritten with data from the incoming migration scheme, it all ends up at the same place as it was on the source. Getting this to happen requires pre-planning when building the ROMs. By the time you hit the size change during migration it is too late and your VM is basically doomed. When building you need to add padding. IIUC for RHEL we artificially increased the size of the seabios and ipxe ROMs to 256k. So when later RHEL updates ship new seabios/ipxe they still fit in the 256k region previously allowed for. <tangent> ... QEMU could really benefit from more formal docs around migration to describe how users / vendors can protect themselves from the many pitfalls like this... </tangent> Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On Thu, Nov 2, 2017 at 4:34 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
Ping - since there wasn't any reply so far - any best practices one could share?
Let me add a TL;DR: - bump of ipxe rom versions change the size of virtio-net-pci.rom - that breaks on migration "Length mismatch"
I'd guess the size of that rom has to be fixed up on the fly, but if that is really ok and how/where is the question.
The actual ROM contents will be transferred in the migration stream, so the fact that the target host has ROMs with different content is not important. The key thing that matters is that QEMU the target host loads the ROMs at the same location, so that when the ROM contents is overwritten with data from the incoming migration scheme, it all ends up at the same place as it was on the source.
Thanks Daniel for your answer, although you try to kill my remaining hopes of a better solution :-) But if the actual ROM content is migrated over anyway "all I'd have to do" is to make clear that the newer system (qemu) knows that the incoming rom has a different size than it thinks. I thought that was what the machine types and their mechanisms were for, but so far I only scratched like 10% of those - maybe they don't cover these rom sizes?
Getting this to happen requires pre-planning when building the ROMs. By the time you hit the size change during migration it is too late and your VM is basically doomed. When building you need to add padding. IIUC for RHEL we artificially increased the size of the seabios and ipxe ROMs to 256k. So when later RHEL updates ship new seabios/ipxe they still fit in the 256k region previously allowed for.
That would have been a good solution to this - if it was done in the past :-) If I end up forced to draw a line anyway, that is a good way to start over safe against further size changes.
Hello Am 03.11.2017 um 08:30 schrieb Christian Ehrhardt:
On Thu, Nov 2, 2017 at 4:34 PM, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Nov 02, 2017 at 04:14:06PM +0100, Christian Ehrhardt wrote:
Ping - since there wasn't any reply so far - any best practices one could share?
Let me add a TL;DR: - bump of ipxe rom versions change the size of virtio-net-pci.rom - that breaks on migration "Length mismatch"
I'd guess the size of that rom has to be fixed up on the fly, but if that is really ok and how/where is the question.
The actual ROM contents will be transferred in the migration stream, so the fact that the target host has ROMs with different content is not important. The key thing that matters is that QEMU the target host loads the ROMs at the same location, so that when the ROM contents is overwritten with data from the incoming migration scheme, it all ends up at the same place as it was on the source.
Thanks Daniel for your answer, although you try to kill my remaining hopes of a better solution :-) But if the actual ROM content is migrated over anyway "all I'd have to do" is to make clear that the newer system (qemu) knows that the incoming rom has a different size than it thinks. I thought that was what the machine types and their mechanisms were for, but so far I only scratched like 10% of those - maybe they don't cover these rom sizes?
The problem is that libvirt launches a qemu process without knowing those details. Later when you try to restore an old snapshot or try to do a live migration, the newly spawned qemu process must have the same layout to be able to load the old VMState. Qemu has gone through several such changes and is worsened by the fact, that distributions like Debian use their own ROM files, which might be different from the sizes shipped with Qemu. For that reason we (Univention) still ship the old ROM images next to the new ones and carry the attached patch to switch between them based on the model. Maybe that works for you, too. One more warning (we experiences): Even if the sizes are the same, live migration can still fail afterwards: We have observed several cases, where a migrated VM carried an old version of SeaBIOS, which fails when you reboot the VM. We traced that down to Qemu implementing a new feature (SMM), which was not supported by the old SeaBIOS and then got it wrong after reboot only. Hope that helps. Philipp
participants (3)
-
Christian Ehrhardt -
Daniel P. Berrange -
Philipp Hahn