support for live migration with PCI passthrough devices
Hi guys, I'm wondering whether libvirt supports live migration for the VM with PCI passthrough devices. or it must be assumed before live migration that all passthrough devices be unplugged? If so, all unplugged devices should be manually hot-plugged to the VM after migration?? Thanks.
On 8/25/20 1:40 PM, Henry lol wrote:
Hi guys,
I'm wondering whether libvirt supports live migration for the VM with PCI passthrough devices. or it must be assumed before live migration that all passthrough devices be unplugged?
Unfortunately, this is still not supported. The problem is that PCI devices themselves are not capable of dumping their internal state and restoring from it on destination. There is a long thread started last month that discuss what the interface should look like, but at this point I guess we are still far away from it: https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
If so, all unplugged devices should be manually hot-plugged to the VM after migration??
This is the usual mode of operation, yes. Michal
On 8/25/20 8:56 AM, Michal Privoznik wrote:
On 8/25/20 1:40 PM, Henry lol wrote:
Hi guys,
I'm wondering whether libvirt supports live migration for the VM with PCI passthrough devices. or it must be assumed before live migration that all passthrough devices be unplugged?
Unfortunately, this is still not supported. The problem is that PCI devices themselves are not capable of dumping their internal state and restoring from it on destination.
There is a long thread started last month that discuss what the interface should look like, but at this point I guess we are still far away from it:
https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
If so, all unplugged devices should be manually hot-plugged to the VM after migration??
This is the usual mode of operation, yes.
There is one exception to this - if the device to be migrated is the VF of an SRIOV-capable network card, then there is machinery in qemu to automatically unplug the device before migration starts, and then automatically plug in a new similar device on the other end after the guest has started there. This all works in concert with a virtio-net emulated network device that is bonded together in the guest (linux only) to provide a single network device in the guest - when the VF is visible and working, it will be preferred for all traffic, but if the VF is missing then the virtio-net emulated device will be used. The result is that under normal operation the vfio-assigned VF is used for all network traffic, but during migration when the VF has been unplugged, the virtio-net device is used, so there is no disruption in network traffic, just a bit of degraded performance during migration. Here is a description of how this is configured in a libvirt domain definition: https://www.libvirt.org/formatdomain.html#teaming-a-virtio-hostdev-nic-pair This is all fairly new, and unfortunately doesn't work properly for all SRIOV cards.
On Tue, Aug 25, 2020 at 02:56:49PM +0200, Michal Privoznik wrote:
On 8/25/20 1:40 PM, Henry lol wrote:
Hi guys,
I'm wondering whether libvirt supports live migration for the VM with PCI passthrough devices. or it must be assumed before live migration that all passthrough devices be unplugged?
Unfortunately, this is still not supported. The problem is that PCI devices themselves are not capable of dumping their internal state and restoring from it on destination.
There is a long thread started last month that discuss what the interface should look like, but at this point I guess we are still far away from it:
https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
Just a little note, even with such an interface in place it will still take time for vendors of conventional physical devices to adopt it, so realistically the main focus here are mediated devices and SRIOV-capable devices. Erik
On 8/26/20 8:27 AM, Erik Skultety wrote:
On Tue, Aug 25, 2020 at 02:56:49PM +0200, Michal Privoznik wrote:
On 8/25/20 1:40 PM, Henry lol wrote:
Hi guys,
I'm wondering whether libvirt supports live migration for the VM with PCI passthrough devices. or it must be assumed before live migration that all passthrough devices be unplugged?
Unfortunately, this is still not supported. The problem is that PCI devices themselves are not capable of dumping their internal state and restoring from it on destination.
There is a long thread started last month that discuss what the interface should look like, but at this point I guess we are still far away from it:
https://www.redhat.com/archives/libvir-list/2020-July/msg00675.html
Just a little note, even with such an interface in place it will still take time for vendors of conventional physical devices to adopt it, so realistically the main focus here are mediated devices and SRIOV-capable devices.
Yeah that. That reminds me that at one of KVM Forums I've seen a demo of migration with a PCI device plugged in whole time, but it was all experimental. /me goes and tries to find it. à voilà: https://www.youtube.com/watch?v=ZAzv0c-fdAc Michal
participants (4)
-
Erik Skultety -
Henry lol -
Laine Stump -
Michal Privoznik