On 08/05/2017 01:14 PM, Laine Stump wrote:
On 08/03/2017 02:33 AM, Moshe Levi wrote:
Hi Laine,
I have a few question before I can give my opinion. I the Mellanox Card Dual Port that support one PCI with 2 PF is ConnectX-3 and ConnectX-3 Pro. (maybe others cards I will check this) The ConnectX-4 Dual Port and above is implemented with 2 PCI devices per 2 PF. So is the "multiple netdevs for a single PCI device" hardware model completely deprecated, and will never show up again in new products?
If that's the case, maybe I shouldn't burden libvirt's config with all this new config that will only be used for legacy hardware. Perhaps a better approach would be to stick with the current config, and make it work properly for VFIO device assignment when a ConnectX-3 card is configured in single port mode - even that doesn't work correctly now[*] but it's a more easily solved problem, and can be done with no config changes.
Opinions about this? If it's a deadend and the existing legacy hardware can be used in a reasonable manner by setting it to single port mode, I don't want to add externally-visible knobs to libvirt.
[*]If a VF is configured to be "port 2 only", libvirt would still try to get/set the MAC and vlan tag with a netlink message to the *port 1* netdev of the PF, so it would be saving/setting the wrong (nonexistent?) VF netdev.
I just tried this and find I was partially mistaken. For example, lets say that I have a PF with netdevs p2p1 (port 1) and p2p2 (port 2), and I set it up to have the first 4 VFs on port 1 and the 2nd 4 on port 2. It turns out that I can set the MAC/vlan tag using either p2p1 *or* p2p2 and it will be set in the correct place. However, if I try to *get* the current MAC/vlan tag for a VF that uses port 2 via the PF netdev for port 1, then I will just get back 0. This might seem confusing/misleading, but it actually makes the job of fixing device assignment for single port mode a bit simpler!
I did send patches yesterday (any reviews/testing appreciated!) that make everything work properly when saving/setting/restoring the VF netdev MAC/vlan tags on dual port cards used for macvtap passthrough:
https://www.redhat.com/archives/libvir-list/2017-August/msg00170.html
Those don't fix the situation when doing VFIO device assignment, but they do at least make macvtap passthrough work correctly (for all VF netdevs, even when the netdevs are dual port!), and are the first step in getting it right for VFIO.