Re: [libvirt] [RFC 0/7] Live Migration with Pass-through Devices proposal

On 04/17/2015 04:53 AM, Chen Fan wrote: This functionality has been on my mind/bug list for a long time, but I haven't been able to pursue it much. See this BZ, along with the original patches submitted by Shradha Shah from SolarFlare: https://bugzilla.redhat.com/show_bug.cgi?id=896716 (I was a bit optimistic in my initial review of the patches - there are actually a lot of issues that weren't handled by those patches.) An interesting idea, but I think that is a 2nd level enhancement, not necessary initially (and maybe not ever, due to the high possibility of it being extremely difficult to get right in 100% of the cases). This isn't quite making sense - the bond will be on the guest, which may not have netcf installed. Anyway, I think it should be up to the guest's own system network config to have the bond already setup. If you try to impose it from outside that infrastructure, you run too much risk of running afoul of something on the guest (e.g. NetworkManager) Correct. This is the most important part. But not just unplugging it, you also need to wait until the unplug operation completes (it is asynchronous). (After this point, the emulated NIC that is part of the bond would get all of the traffic). Why does the MAC address need to be different? Are you suggesting doing this with passed-through non-SRIOV NICs? An SRIOV virtual function gets its MAC address from the libvirt config, so it's very simple to use the same MAC address across the migration. Any network card that would be able to do this on any sort of useful scale will be SRIOV-capable (or should be replaced with one that is - some of them are not that expensive). I never looked at the details of how SolarFlare's code handled the guest side (they have/had their own patchset they maintained for some older version of libvirt which integrated with some sort of enhanced bonding driver on the guests). I assumed the bond driver could handle this already, but have to say I never investigated. Putting this into <hostdev> is the wrong approach, for two reasons: 1) it doesn't account for the device to be used being in a different address on the source and destination hosts, 2) the <interface> element already has much of the config you need, and an interface type supporting hostdev passthrough. It has been possible to do passthrough of an SRIOV VF via <interface type='hostdev'> for a long time now and, even better, via an <interface type='network'> where the network pointed to contains a pool of VFs - As long as the source and destination hosts both have networks with the same name, libvirt will be able to find a currently available device on the destination as it migrates from one host to another instead of relying on both hosts having the exact same device at the exact same address on the host and destination (and also magically unused by any other guest). This page explains the use of a "hostdev network" which has a pool of devices: http://wiki.libvirt.org/page/Networking#Assignment_from_a_pool_of_SRIOV_V... This was designed specifically with the idea in mind that one day it would be possible to migrate a domain with a hostdev device (as long as the guest could handle the hostdev device being temporarily unplugged during the migration). Again, I think that this level of detail about the guest network config belongs on the guest, not in libvirt. ^^ this is the useful part but I don't think the right method is to make this action dependent on the device being a "bond". I think that in this respect Shradha's patches had a better idea - any hostdev (or, by implication <interface type='hostdev'> or, much more usefully <interface type='network'> pointing to a pool of VFs - could have an attribute "ephemeral". If ephemeral was "yes", then the device would always be unplugged prior to migration and re-plugged when migration was completed (the same thing should be done when saving/restoring a domain which also can't currently be done with a domain that has a passthrough device). For that matter, this could be a general-purpose thing (although probably most useful for hostdevs) - just make it possible for *any* hotpluggable device to be "ephemeral"; the meaning of this would be that every device marked as ephemeral should be unplugged prior to migration or save (and libvirt should wait for qemu to notify that the unplug is completed), and re-plugged right after the guest is restarted. (possibly it should be implemented as an <ephemeral> *element* rather than attribute, so that options could be specified). After that is implemented and works properly, then it might be the time to think about auto-creating the bond (although again, my opinion is that this is getting a bit too intrusive into the guest (and making it more likely to fail - I know from long experience with netcf that it is all too easy for some other service on the system (ahem) to mess up all your hard work); I think it would be better to just let the guest deal with setting up a bond in its system network config, and if the bond driver can't handle having a device in the bond unplugging and plugging, then the bond driver should be enhanced).