Hi Martin, Very good question below..... On 8/9/24 06:10, Martin Kletzander wrote:
On Wed, Aug 07, 2024 at 10:20:49AM -0500, Michael Galaxy wrote:
What we have here so far is the minimal set of changes needed to make it work. I'd like to avoid making this set too complicated. How we handle QMP abstractions can be improved later if we want to engage the original CPR author (steven.sistare@oracle.com) at some point.
So you are solely relying on the fact that when we start QEMU again it will use the same paths and just resume working? That could be another reason to make changes to libvirt, if only to make sure the paths are the same.
Yes, we are relying on that fact, correct. We have not had any serious issues on this front, as when we're doing a live updates, we're expecting all the paths to be the same.
Well, if you are starting the domains without libvirt's prior knowledge (i.e. not from as saved snapshot or anything like that) libvirt will generate the paths based on the domain ID. But if there is nothing running during the start libvirt will start the IDs from number 1. Unless the previous VM was also the first one started it will have different ID and the whole path will be different.
That is in fact exactly what we are doing. =) I'll be demonstrating this at KVM Forum next month, but basically, to save time, we ended up starting QEMU *manually* inside the initramfs phase of the boot process. This shaved many valuable seconds off the kexec resume process. We are using libvirt's "domxml-to-native" command to provide us with most of the QEMU command line and then we *manually* bring QEMU back to life before libvirt is back. Later when systemd comes online, libvirt then starts back up again and continues taking ownership of the VM that it started before the kexec occurred. When we do it this way, there are no path naming issues. We did this for many reasons: Pulling libvirt into the initramfs was extremely heavy weight and we did not want to package the whole daemon with all the systemd dependencies inside of the initramfs.... it's just too much baggage. On the other hand, if libvirt had an offline "helper" binary capable of resming a VM inside the initramfs that did not require us bypassing the daemon, that would be really great, but those are thoughts for future work. There are a lot of caveats that we had to workaround in the above approach: the "domxml-to-native" output was not very well sanitized..... it has bugs in it. We can try to post fixes to those fixes when we get around to them. Also, we had to deal with cgroup reconstruction. The cgroups are of course blown away when the new kernel takes over so we had to preserve them across the kexec so that libvirt is able to re-attach to a fully "valid" QEMU when libvirt comes back online. So, there's a lot to unpack there => Much too complex of a conversation for this more minimal patchset. - Michael