Qemu pause-before-switchover and libvirt live migration
Hello, I’m looking into the libvirt hooks that are available to make use of during a live migration, but there's only hooks on migration begin and migration end. On qemu directly I’m able to use pause-before-switchover migration capability to run a custom operation just before it moves to the destination, but within libvirt it seems that while this is leveraged internally, it’s not currently exposed to the client in a way I can use. Does anyone have any suggestions on how I could expose this to the client in a generic and sensible way? Thanks, Rogerio
On Thu, Oct 16, 2025 at 14:43:56 +0000, Rogério Vinhal Nunes via Devel wrote:
Hello,
I’m looking into the libvirt hooks that are available to make use of during a live migration, but there's only hooks on migration begin and migration end. On qemu directly I’m able to use pause-before-switchover migration capability to run a custom operation just before it moves to the destination, but within libvirt it seems that while this is leveraged internally, it’s not currently exposed to the client in a way I can use.
Does anyone have any suggestions on how I could expose this to the client in a generic and sensible way?
Yes this is currently not available. Can you elaborate what you'd want to use it for? E.g. what we don't support that could be solved by this. We prefer if things work without hooks as hooks are really clunky for users to setup properly and maybe your case can be improved by some native support.
We have some internal tools we need to setup/tear down on the time the domain stops on the source and starts on the destination, so I don't believe full native support would fit. An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook? I'm happy to work on a patch and I'm open for guidance on what to do here.
On 16 Oct 2025, at 18:44, Peter Krempa <pkrempa@redhat.com> wrote:
>
On Thu, Oct 16, 2025 at 14:43:56 +0000, Rogério Vinhal Nunes via Devel wrote: Hello,
I’m looking into the libvirt hooks that are available to make use of during a live migration, but there's only hooks on migration begin and migration end. On qemu directly I’m able to use pause-before-switchover migration capability to run a custom operation just before it moves to the destination, but within libvirt it seems that while this is leveraged internally, it’s not currently exposed to the client in a way I can use.
Does anyone have any suggestions on how I could expose this to the client in a generic and sensible way?
Yes this is currently not available.
Can you elaborate what you'd want to use it for? E.g. what we don't support that could be solved by this.
We prefer if things work without hooks as hooks are really clunky for users to setup properly and maybe your case can be improved by some native support.
Please don't top-post on technical lists. On Thu, Oct 16, 2025 at 18:47:36 +0000, Rogério Vinhal Nunes wrote:
We have some internal tools we need to setup/tear down on the time the domain stops on the source and starts on the destination, so I don't believe full native support would fit.
I've asked for clarification because libvirt already provides hooks that can fit your description. Specifically on the source of the migration you'll get: Two hook calls: $VMNAME stopped end - $VMNAME release end migrated Both of these happen when the VM is already running on the destination, but right after the migration. On the destination you get: $VMNAME migrate begin - $VMNAME prepare begin - $VMNAME start begin - $VMNAME started begin - IIRC 'migrate' and 'prepare' happen when setting up the migration. 'start' happens right before starting the qemu process. At this point the migration will progress. 'started' happens after the migration was complete. So this provides means to setup and tear down resources. The open question still is whether you need that to happen precisely at switchover time. Thus the request for what you want to actually do. A hook at switchover time is obviously possible but until now we didn't yet get a good reason to have one. A reason against is that it introduces latency for the switchover.
An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook?
Once again specifying what you actually want to do would be helpful. E.g. I can suggest that you can migrate the VM as paused, which ensures that once the migration completes it will not continue execution on the destination, which could give you the chance for additional synchronisation.
I'm happy to work on a patch and I'm open for guidance on what to do here.
On 16 Oct 2025, at 18:44, Peter Krempa <pkrempa@redhat.com> wrote:
>
On Thu, Oct 16, 2025 at 14:43:56 +0000, Rogério Vinhal Nunes via Devel wrote: Hello,
I’m looking into the libvirt hooks that are available to make use of during a live migration, but there's only hooks on migration begin and migration end. On qemu directly I’m able to use pause-before-switchover migration capability to run a custom operation just before it moves to the destination, but within libvirt it seems that while this is leveraged internally, it’s not currently exposed to the client in a way I can use.
Does anyone have any suggestions on how I could expose this to the client in a generic and sensible way?
Yes this is currently not available.
Can you elaborate what you'd want to use it for? E.g. what we don't support that could be solved by this.
We prefer if things work without hooks as hooks are really clunky for users to setup properly and maybe your case can be improved by some native support.
Sent from my iPhone
On 17 Oct 2025, at 12:59, Peter Krempa <pkrempa@redhat.com> wrote:
> Please don't top-post on technical lists.
On Thu, Oct 16, 2025 at 18:47:36 +0000, Rogério Vinhal Nunes wrote:
We have some internal tools we need to setup/tear down on the time the domain stops on the source and starts on the destination, so I don't believe full native support would fit.
I've asked for clarification because libvirt already provides hooks that can fit your description.
Specifically on the source of the migration you'll get:
Two hook calls:
$VMNAME stopped end - $VMNAME release end migrated
Both of these happen when the VM is already running on the destination, but right after the migration.
On the destination you get:
$VMNAME migrate begin - $VMNAME prepare begin - $VMNAME start begin - $VMNAME started begin -
IIRC 'migrate' and 'prepare' happen when setting up the migration. 'start' happens right before starting the qemu process. At this point the migration will progress. 'started' happens after the migration was complete.
So this provides means to setup and tear down resources.
The open question still is whether you need that to happen precisely at switchover time. Thus the request for what you want to actually do.
A hook at switchover time is obviously possible but until now we didn't yet get a good reason to have one. A reason against is that it introduces latency for the switchover.
I need to synchronise our internal storage solution at the switchover time because I need to unmap the resource from source and map on the destination. It would be a behaviour similar to what happens with NBD, but it's backed by our internal backend.
An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook?
Once again specifying what you actually want to do would be helpful.
E.g. I can suggest that you can migrate the VM as paused, which ensures that once the migration completes it will not continue execution on the destination, which could give you the chance for additional synchronisation.
For us it's important to have the least amount of interruption as possible, so we're very keen on a live migration here.
I'm happy to work on a patch and I'm open for guidance on what to do here.
On 16 Oct 2025, at 18:44, Peter Krempa <pkrempa@redhat.com> wrote:
>
On Thu, Oct 16, 2025 at 14:43:56 +0000, Rogério Vinhal Nunes via Devel wrote: Hello,
I’m looking into the libvirt hooks that are available to make use of during a live migration, but there's only hooks on migration begin and migration end. On qemu directly I’m able to use pause-before-switchover migration capability to run a custom operation just before it moves to the destination, but within libvirt it seems that while this is leveraged internally, it’s not currently exposed to the client in a way I can use.
Does anyone have any suggestions on how I could expose this to the client in a generic and sensible way?
Yes this is currently not available.
Can you elaborate what you'd want to use it for? E.g. what we don't support that could be solved by this.
We prefer if things work without hooks as hooks are really clunky for users to setup properly and maybe your case can be improved by some native support.
On Fri, Oct 17, 2025 at 13:47:54 +0000, Rogério Vinhal Nunes wrote:
On 17 Oct 2025, at 12:59, Peter Krempa <pkrempa@redhat.com> wrote:
On Thu, Oct 16, 2025 at 18:47:36 +0000, Rogério Vinhal Nunes wrote:
[...]
IIRC 'migrate' and 'prepare' happen when setting up the migration. 'start' happens right before starting the qemu process. At this point the migration will progress. 'started' happens after the migration was complete.
So this provides means to setup and tear down resources.
The open question still is whether you need that to happen precisely at switchover time. Thus the request for what you want to actually do.
A hook at switchover time is obviously possible but until now we didn't yet get a good reason to have one. A reason against is that it introduces latency for the switchover.
I need to synchronise our internal storage solution at the switchover time because I need to unmap the resource from source and map on the destination. It would be a behaviour similar to what happens with NBD, but it's backed by our internal backend.
So with NBD/NFS and others it works in a way where both the source and destination open the storage itself. QEMU internally ensures that it hands over the state cleanly and doesn't write from the source after handover. Can't your storage do that? That way you could do the setup before migration on the destination and tear-down after migration on source, thus eliminating the extra unbounded latency at switchover?
An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook?
Once again specifying what you actually want to do would be helpful.
E.g. I can suggest that you can migrate the VM as paused, which ensures that once the migration completes it will not continue execution on the destination, which could give you the chance for additional synchronisation.
For us it's important to have the least amount of interruption as possible, so we're very keen on a live migration here.
That's the reason I think a synchronous hook, which will block the migration from switching over while the hook is executing, is not a great idea.
On 20 Oct 2025, at 09:36, Peter Krempa <pkrempa@redhat.com> wrote:
> On Fri, Oct 17, 2025 at 13:47:54 +0000, Rogério Vinhal Nunes wrote:
On 17 Oct 2025, at 12:59, Peter Krempa <pkrempa@redhat.com> wrote:
On Thu, Oct 16, 2025 at 18:47:36 +0000, Rogério Vinhal Nunes wrote:
[...]
IIRC 'migrate' and 'prepare' happen when setting up the migration. 'start' happens right before starting the qemu process. At this point the migration will progress. 'started' happens after the migration was complete.
So this provides means to setup and tear down resources.
The open question still is whether you need that to happen precisely at switchover time. Thus the request for what you want to actually do.
A hook at switchover time is obviously possible but until now we didn't yet get a good reason to have one. A reason against is that it introduces latency for the switchover.
I need to synchronise our internal storage solution at the switchover time because I need to unmap the resource from source and map on the destination. It would be a behaviour similar to what happens with NBD, but it's backed by our internal backend.
So with NBD/NFS and others it works in a way where both the source and destination open the storage itself. QEMU internally ensures that it hands over the state cleanly and doesn't write from the source after handover.
Can't your storage do that? That way you could do the setup before migration on the destination and tear-down after migration on source, thus eliminating the extra unbounded latency at switchover? The problem is that the way it's currently designed it relies on cached writes that can be propagated after the domain starts on the destination, so we need the hook to, at least, flush the source before the destination becomes rw.
An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook?
Once again specifying what you actually want to do would be helpful.
E.g. I can suggest that you can migrate the VM as paused, which ensures that once the migration completes it will not continue execution on the destination, which could give you the chance for additional synchronisation.
For us it's important to have the least amount of interruption as possible, so we're very keen on a live migration here.
That's the reason I think a synchronous hook, which will block the migration from switching over while the hook is executing, is not a great idea. The hook is supposed to take order of ms whilst the migration of the memory is supposed to take many seconds. I believe that pausing the domain will be worse in manners of interruption. WRT to migrations that don't rely on it, we could add a migration flag that enables this.
On Mon, Oct 20, 2025 at 08:52:18 +0000, Rogério Vinhal Nunes wrote:
On 20 Oct 2025, at 09:36, Peter Krempa <pkrempa@redhat.com> wrote: On Fri, Oct 17, 2025 at 13:47:54 +0000, Rogério Vinhal Nunes wrote:
On 17 Oct 2025, at 12:59, Peter Krempa <pkrempa@redhat.com> wrote:
On Thu, Oct 16, 2025 at 18:47:36 +0000, Rogério Vinhal Nunes wrote:
[...]
So with NBD/NFS and others it works in a way where both the source and destination open the storage itself. QEMU internally ensures that it hands over the state cleanly and doesn't write from the source after handover.
Can't your storage do that? That way you could do the setup before migration on the destination and tear-down after migration on source, thus eliminating the extra unbounded latency at switchover?
The problem is that the way it's currently designed it relies on cached writes that can be propagated after the domain starts on the destination, so we need the hook to, at least, flush the source before the destination becomes rw.
So qemu internally uses posix_fadvise(POSIX_FADV_DONTNEED) (see qemu commit dd577a26ff03b6829721b1ffbbf9e7c411b72378) to drop caches before migration to specifically support migration with caching enabled. Won't that work for your storage? Alternatively the usual approach before that and in many cases the still suggested option is to bypass caching on the host side e.g. by using cache='none' for the disks. Would either of the above work? Or is the cache inherent to the storage?
An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook?
Once again specifying what you actually want to do would be helpful.
E.g. I can suggest that you can migrate the VM as paused, which ensures that once the migration completes it will not continue execution on the destination, which could give you the chance for additional synchronisation.
For us it's important to have the least amount of interruption as possible, so we're very keen on a live migration here.
That's the reason I think a synchronous hook, which will block the migration from switching over while the hook is executing, is not a great idea.
The hook is supposed to take order of ms whilst the migration of the memory is supposed to take many seconds. I believe that pausing the domain will be worse in manners of interruption. WRT to migrations that don't rely on it, we could add a migration flag that enables this.
The hook, as it happens on a critical path would need some form of positive action from the user/management app when it is about to be used so that we can avoid calling it for VMs which don't need it. A flag would likely work, but I was thinking of actually doing two hooks. The first one would be called in one of the preparation steps. The return value for that hook would be used as an indication that the switchover hook should be invoked. This way the hook can exist purely in hook form, and thus not require changes to the applications using said deployment. If nothing of what I've suggested aleviates your need for the hook, feel free to contriubute it. In the code, hooks are invoked via virHookCall(). You'll have to find the appropriate places where to place them. 'docs/hooks.rst' documents the user-facing side of the hook, so your new actions will need to be added. Feel free to post another thread if you have any further questions about the design. Our contributor guidelines are at https://www.libvirt.org/hacking.html including AI policy (emphasized due to your email address).
Hello Rogerio, did you find any way to run your platform code at-or-close to switchover time before starting the new QEMU process? Anything you could share? Thanks, Claudio On 10/20/25 10:52, Rogério Vinhal Nunes via Devel wrote:
On 20 Oct 2025, at 09:36, Peter Krempa <pkrempa@redhat.com> wrote:
> On Fri, Oct 17, 2025 at 13:47:54 +0000, Rogério Vinhal Nunes wrote:
On 17 Oct 2025, at 12:59, Peter Krempa <pkrempa@redhat.com> wrote:
On Thu, Oct 16, 2025 at 18:47:36 +0000, Rogério Vinhal Nunes wrote:
[...]
IIRC 'migrate' and 'prepare' happen when setting up the migration. 'start' happens right before starting the qemu process. At this point the migration will progress. 'started' happens after the migration was complete.
So this provides means to setup and tear down resources.
The open question still is whether you need that to happen precisely at switchover time. Thus the request for what you want to actually do.
A hook at switchover time is obviously possible but until now we didn't yet get a good reason to have one. A reason against is that it introduces latency for the switchover.
I need to synchronise our internal storage solution at the switchover time because I need to unmap the resource from source and map on the destination. It would be a behaviour similar to what happens with NBD, but it's backed by our internal backend.
So with NBD/NFS and others it works in a way where both the source and destination open the storage itself. QEMU internally ensures that it hands over the state cleanly and doesn't write from the source after handover.
Can't your storage do that? That way you could do the setup before migration on the destination and tear-down after migration on source, thus eliminating the extra unbounded latency at switchover? The problem is that the way it's currently designed it relies on cached writes that can be propagated after the domain starts on the destination, so we need the hook to, at least, flush the source before the destination becomes rw.
An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook?
Once again specifying what you actually want to do would be helpful.
E.g. I can suggest that you can migrate the VM as paused, which ensures that once the migration completes it will not continue execution on the destination, which could give you the chance for additional synchronisation.
For us it's important to have the least amount of interruption as possible, so we're very keen on a live migration here.
That's the reason I think a synchronous hook, which will block the migration from switching over while the hook is executing, is not a great idea. The hook is supposed to take order of ms whilst the migration of the memory is supposed to take many seconds. I believe that pausing the domain will be worse in manners of interruption. WRT to migrations that don't rely on it, we could add a migration flag that enables this.
Hey Claudio, We ended up managing to use the connection on the destination as a seal the block device for the source. We'll still validate at scale if this works out well before we get back to this.
On 28 Jan 2026, at 17:42, Claudio Fontana <cfontana@suse.de> wrote:
Hello Rogerio,
did you find any way to run your platform code at-or-close to switchover time before starting the new QEMU process?
Anything you could share?
Thanks,
Claudio
On 10/20/25 10:52, Rogério Vinhal Nunes via Devel wrote:
On 20 Oct 2025, at 09:36, Peter Krempa <pkrempa@redhat.com> wrote:
> On Fri, Oct 17, 2025 at 13:47:54 +0000, Rogério Vinhal Nunes wrote:
On 17 Oct 2025, at 12:59, Peter Krempa <pkrempa@redhat.com> wrote:
> On Thu, Oct 16, 2025 at 18:47:36 +0000, Rogério Vinhal Nunes wrote:
[...]
IIRC 'migrate' and 'prepare' happen when setting up the migration. 'start' happens right before starting the qemu process. At this point the migration will progress. 'started' happens after the migration was complete.
So this provides means to setup and tear down resources.
The open question still is whether you need that to happen precisely at switchover time. Thus the request for what you want to actually do.
A hook at switchover time is obviously possible but until now we didn't yet get a good reason to have one. A reason against is that it introduces latency for the switchover.
I need to synchronise our internal storage solution at the switchover time because I need to unmap the resource from source and map on the destination. It would be a behaviour similar to what happens with NBD, but it's backed by our internal backend.
So with NBD/NFS and others it works in a way where both the source and destination open the storage itself. QEMU internally ensures that it hands over the state cleanly and doesn't write from the source after handover.
Can't your storage do that? That way you could do the setup before migration on the destination and tear-down after migration on source, thus eliminating the extra unbounded latency at switchover? The problem is that the way it's currently designed it relies on cached writes that can be propagated after the domain starts on the destination, so we need the hook to, at least, flush the source before the destination becomes rw.
An alternative could be to have an option to wait for a resume operation to progress as a client-defined migration flag exposing the pre-switchover state. This way maybe we could work it as a client feature rather than a hook?
Once again specifying what you actually want to do would be helpful.
E.g. I can suggest that you can migrate the VM as paused, which ensures that once the migration completes it will not continue execution on the destination, which could give you the chance for additional synchronisation.
For us it's important to have the least amount of interruption as possible, so we're very keen on a live migration here.
That's the reason I think a synchronous hook, which will block the migration from switching over while the hook is executing, is not a great idea. The hook is supposed to take order of ms whilst the migration of the memory is supposed to take many seconds. I believe that pausing the domain will be worse in manners of interruption. WRT to migrations that don't rely on it, we could add a migration flag that enables this.
On 10/16/25 19:44, Peter Krempa via Devel wrote:
On Thu, Oct 16, 2025 at 14:43:56 +0000, Rogério Vinhal Nunes via Devel wrote:
Hello,
I’m looking into the libvirt hooks that are available to make use of during a live migration, but there's only hooks on migration begin and migration end. On qemu directly I’m able to use pause-before-switchover migration capability to run a custom operation just before it moves to the destination, but within libvirt it seems that while this is leveraged internally, it’s not currently exposed to the client in a way I can use.
Does anyone have any suggestions on how I could expose this to the client in a generic and sensible way?
Yes this is currently not available.
I would also be interested in this.
Can you elaborate what you'd want to use it for? E.g. what we don't support that could be solved by this.
There is internal state I need to update in the platform that is much better done just before the switchover. In addition to that, currently the way libvirt handles migration stats from QEMU is not satisfactory for me, so I need to query qemu directly.
We prefer if things work without hooks as hooks are really clunky for users to setup properly and maybe your case can be improved by some native support.
Ciao, CLaudio
On Fri, Oct 17, 2025 at 11:45:09 +0200, Claudio Fontana wrote:
On 10/16/25 19:44, Peter Krempa via Devel wrote:
On Thu, Oct 16, 2025 at 14:43:56 +0000, Rogério Vinhal Nunes via Devel wrote:
Hello,
I’m looking into the libvirt hooks that are available to make use of during a live migration, but there's only hooks on migration begin and migration end. On qemu directly I’m able to use pause-before-switchover migration capability to run a custom operation just before it moves to the destination, but within libvirt it seems that while this is leveraged internally, it’s not currently exposed to the client in a way I can use.
Does anyone have any suggestions on how I could expose this to the client in a generic and sensible way?
Yes this is currently not available.
I would also be interested in this.
Can you elaborate what you'd want to use it for? E.g. what we don't support that could be solved by this.
There is internal state I need to update in the platform that is much better done just before the switchover.
In addition to that, currently the way libvirt handles migration stats from QEMU is not satisfactory for me, so I need to query qemu directly.
Could you be more specific in both points?
participants (3)
-
Claudio Fontana -
Peter Krempa -
Rogério Vinhal Nunes