On Mon, 2011-08-22 at 05:17 -0400, Laine Stump wrote: I read this is to avoid wasting MAC addresses from the vendor's pool which might never be used RHEL 6.1 seems to use the PCI id to manage the inteface name in /etc/udev/rules.d/70-persistent-net.rules: # PCI device 0x8086:0x10ed (ixgbevf) SUBSYSTEM=="net", ACTION=="add", ATTR{dev_id}=="0x0", KERNELS=="0000:15:10.0", ATTR{type}=="1", KERNEL=="eth*", NAME="eth8" type='hostdev'? If I understand this correctly, this outlines an "implicit" pci passthrough and there is no need to provide an explicit <hostdev/> element in the domain xml. Guest configs using an explicit <hostdev/> element would still expose the problem outlined above, correct? Any plans for those?

On Aug 23, 2011, at 12:50 PM, Daniel P. Berrange wrote: I guess there is a issue with PCI-passtrough here, If the VEPA link is set up prior to VM start then that information is lost when the VM OS resets the device during initialization. Only on NICs with an integrated bridge can this setup be persistent because the bridge can handle the VLAN tagging and port setup. I see a major drawback with storing MAC adresses in <hostdev> elements: It would require great care to make sure that MAC adresses are unique across a big datacenter. The same is true for network setups, the available options are becomming more and more confusing. Regards, D.Herrendoerfer

On 08/23/2011 06:50 AM, Daniel P. Berrange wrote: Actually, I was thinking that the proper <hostdev> *would* be added to the live XML as non-persistent. This way all PCI devices currently assigned to the guest could still be retrieved by looking at the <hostdev> elements, but the specific PCI device used for this particular instance wouldn't need to be hardcoded into the config XML. (I think the ability to grab a free ethernet device from a pool at runtime, rather than having hardcoded devices, is an important feature of this proposed method of dealing with pci passthrough ethernet devices. I suppose a management app could be written to handle that allocation, and rewrite the domain config, but it seems like something that libvirt should be able to handle). I agree that we don't want to make management applications look for PCI devices scattered all over the config. Likewise I think it would be nice if applications don't have to go looking all over the place for MAC addresses. And now that I've heard port profiles need to be associated with these devices too, I'm wondering what will be next... having that type of high level information in a <hostdev> doesn't seem very appealing to me. I think it would be much cleaner if it could remain in <interface> (or in a <portgroup> of a network definition). I think with non-persistent <hostdev> elements auto-generated based on <interface>/<network> definitions, we can get the best of both worlds - a complete list of all PCI devices allocated to the guest is still available in one place, but we can leverage a lot of code already in the network interface management stuff - interface pools, portgroups, etc. (unfortunately, we'll never be able to take advantage of bandwidth management or nwfilters, but there's really no solution to that short of installing an agent in the guest - by the time you get to that point, I think it's probably time to acknowledge that PCI passthrough of network devices just isn't a great general purpose solution, and use an actual QEMU network device instead) Nice. I didn't realize that. The description of how a management app handles the situation actually fits quite well with my proposal - the non-persistent hostdev would be unplugged, and after migration is completed, the normal codepath for initializing network device plumbing for the qemu process on the destination host would automatically reserve and plug in a new pci device. I had actually been considering from the beginning that a <hostdev> element would end up in the live XML (after being created based on the <interface> (and the <network> it references) while the guest is starting up). This keeps network device config out of hostdev space, and hostdev config out of network device space (and fits in with the idea of eliminating host-specific config info from the domain config (since the actual PCI device to be used isn't in the domain XML, but is instead determined at domain startup.) If it's acceptable to add non-persistent <hostdev>s to the live XML, the main open item I see is that the management apps trying to migrate a guest containing them will need to understand that these transient <hostdev> devices will have replacements automatically plugged in on the destination by the networking code. For that matter, the management app shouldn't be unplugging them either (and neither should "virsh detach-device", for example), because they will require extra code not normally run during a PCI hot-unplug (to disassociate the port profile, and return the ethernet device to the network's pool) (So maybe the hostdev does need some reference back to the higher level device definition (in this case <interface>) after all. Bah.) (Another potential problem area I see is with the relative sequencing of unplugging/disassociating/plugging/associating these devices during a migration - for standard network devices I think the unplugging on the source host doesn't happen until after the migration is complete, but for PCI passthrough devices it must happen before the migration starts. But I may again be trying to think up a solution to a problem that is irrelevant).

On 08/24/2011 05:48 AM, Daniel P. Berrange wrote: Ah, well that clinches it - definitely an absolute requirement :-( Yep, you've convinced me (at least that the guest-side PCI info needs to be persistent, and the proper place for that is in a <hostdev>). I'm still not comfortable with having all that other extra info in the <hostdev> though. Let me take a few days to think about it and maybe come up with a logical method of having the hostdev reference back to the higher level info rather than containing it directly. If I can't think of something that doesn't look like a kludge, then we'll just have to do it by including mac address, port profile, etc as subelements of <hostdev>.