Re: [libvirt] RFC: PCI-Passthrough of SRIOV VF's new forward mode

On 02/08/2012 06:18 AM, Shradha Shah wrote: The two approaches are complementary. For example, just as in current libvirt you can directly define <interface type='bridge'>, you can also define <interface type='network'> and have the network be <forward mode='bridge'>. In the end, the interface ends up being connected to a non-libvirt-managed bridge device. In this case, you could define <interface type='hostdev'> and directly specify the host device in the interface definition, or you could define <interface type='network'>, then have a pool of devices in the network definition to choose from. In both cases, you would still end up with a network device that was assigned to the guest via passthrough. And just like interface type='bridge'and type='direct' were first implemented directly, and then later support for that type of interface was added to networks, I think it's logical to do it the same way here, since the latter can make use of the code written for the former. Yep. Agreed. BTW, I'm leaning towards <forward mode='hostdev'> because what I'm working on will work equally well for USB or PCI devices. I'm still unclear how the hybrid mode works (although I'm slowly getting a better picture) Yes, in cases where the installation is large enough people want to migrate. Of course, there is the problem that the guest may have the PCI device in some state at the time of migration that qemu doesn't know about, and so the destination device can't be put into that exact state (for example, maybe a packet has been written to the device's memory but not yet transmitted). As a matter of fact, migration is currently forbidden if there are any attached hostdevs - you must first detach them all, then migrate, then attach new devices. Have you actually experimented with migration of a guest that has one of your devices assigned to the guest via PCI passthrough? Yes. That's the reason that the ability to define networks with pools of network devices was added. It doesn't eliminate the usefulness of being able to directly define it in the <interface> if you have a simple setup. Interesting. I guess it makes sense to not waste time with a VF driver, since a host is likely to gain anything from using more than one network device per port anyway, so the only practical use of the VFs (especially 127 of them!) is by virtual guests. Well, that answers my previous question about migration :-) So I assume you perform an additional operation on top of libvirt migration, i.e. you first detach the VF passthrough device, then migrate, then at the other end you attach the new passthrough device, right? Interesting. This will mean that a single entry in the <devices> will use two PCI addresses. How are you currently dealing with that? Just dynamically using whatever free address is available? In general we like to maintain the stability of guest-side PCI addresses, and failure to do so causes problems (e.g. MS Windows may require re-activation). The need for 2 PCI addresses in a single device entry will require some thought. Also, I'm realizing that this model will require the ability to detach "half of a device" (since you need to be able to detach the VF passthrough device without detaching the virtio device). One other thing related to the XML - are you planning on always specifying <model type='virtio'> in the <interface> config? If so, the <network> definition doesn't need to say anything about whether its devices are to be used for normal hostdev, or hybrid - when the domain allocates a device from the network, it will then just look at <model> and decide whether to do a plain passthrough, or a hybrid mode passthrough. This way guests that don't have support for hybrid mode can use the same network as those that do (they'll just leave out <model>) To allow for any type of hostdev (and allow re-using existing hostdev management code), I've figured on something like this in the NetDef (and similar in the ActualNetDef): struct _virDomainNetDef { enum virDomainNetType type; unsigned char mac[VIR_MAC_BUFLEN]; ... union { ... struct { char *linkdev; int mode; /* enum virMacvtapMode from util/macvtap.h */ virNetDevVPortProfilePtr virtPortProfile; } direct; ** struct { ** virDomainHostdevDef def; ** virNetDevVPortProfilePtr virtPortProfile; } hostdev; } data; struct { bool sndbuf_specified; unsigned long sndbuf; } tune; ... char *ifname; virDomainDeviceInfo info; ... }; The HostdevDef would hold the PCI address information (or USB - not important but it's already in there, so...) along with info about the device's state prior to being assigned to the guest (you don't require that for your application, but for general use by other cards it is a requirement). The *really* nice thing about doing it this way, though, is that a pointer to this hostdevdef can just be plopped right into the domain's hostdevs[] array, and will then be included in things like scans of all passthrough devices to determine which devices are in use, etc. there's a bit more info about this in: https://www.redhat.com/archives/libvir-list/2012-January/msg01379.html adding the hostdev type to the data union, we also want to add one called hybrid: struct { char *linkdev; int mode; /* enum virMacvtapMode from util/macvtap.h */ virDomainHostdevDef def; virDomainDeviceInfo hostdevSourceInfo; virNetDevVPortProfilePtr virtPortProfile; } hybrid; (This also coincidentally solves the problem of needing to reserve two pci addresses on the guest - the address used for the virtio device will be in netdef->info, and the address for the passthrough device will be in netdef->data.hybrid.hostdevSourceInfo.) Believe me, we've heard about it :-) I'll send you separate mail off-list.