Re: [libvirt] PCI passthrough/SR-IOV on Cavium cn889x

Thank you all for getting back so quickly. Some responses inline -----Original Message----- From: sendmail <justsendmailnothingelse(a)gmail.com&gt; On Behalf Of Laine Stump Sent: Wednesday, March 21, 2018 7:54 PM To: libvirt <libvir-list(a)redhat.com&gt; Cc: Ciprian Barbu <Ciprian.Barbu(a)enea.com>; Alexandru Avadanii <Alexandru.Avadanii(a)enea.com>; Alex Williamson <Alex.Williamson(a)redhat.com&gt; Subject: Re: [libvirt] PCI passthrough/SR-IOV on Cavium cn889x On 03/21/2018 11:46 AM, Ciprian Barbu wrote: I see that while I was typing my own "really long" message, that Alex pointed out in a response that you could use <hostdev> rather than <interface type='hostdev'> if you don't need to configure the MAC address or vlan tag of the VF from within libvirt. If that's the case, you can ignore the rest of my message, but otherwise read on :-) The PF netdev name is needed because the netlink messages to get/set the VF MAC address and vlan tag are sent to the PF netdev. A message to set the MAC and vlan tag for VF 2 of PF "enpblah' would be something like this:      RTM_SETLINK/NLM_F_REQUEST-------+      | ifindex=-1                    |      | family=AF_UNSPEC              |      | IFLA_IFNAME------------------+|      | | enpblah                    ||      | +----------------------------+|      | IFLA_VFINFO_LIST-------------+|      | | IFLA_VFINFO---------------+||      | | | IFLA_VF_MAC------------+|||      | | | | vf=2                 ||||      | | | | mac=de:ad:be:ef:c0:55||||      | | | +----------------------+|||      | | | IFLA_VF_VLAN-----------+|||      | | | | vf=2                 ||||      | | | | vlanid=42            ||||      | | | +----------------------+|||      | | +------------------------+|||      | +---------------------------+||      +-------------------------------+ I *think* (although I can't say for certain since the original code was written by someone else, and I've never tried it the other way) that we could achieve the same result by filling in ifindex with the index of "enpblah" (instead of -1), then leaving out the IFLA_IFNAME attribute, but I haven't found any way of specifying the target of a netlink message other than with its ifindex or its ifname. When you say "use the sysfs path", what exactly do you mean? Is there a way to save/set the VF MAC addresses and vlan tags via sysfs? Or (better) a way to address the netlink message to the PF if it has no netdev name or ifindex? Maybe the drivers are setup so that an RTM_SETLINK request send to a "primary VF" would be able to get/set VF_INFO for "Secondary VFs" associated with the same PF? I'm just pulling ideas out of thin air here... This is the first time I've heard of an SRIOV network device where the PF wasn't bound to a netdev driver and so had no netdev name or ifindex. I guess this is describing the card you're talking about?   https://dpdk.org/doc/guides/nics/thunderx.html I have to say that it does *not* give me the warm fuzzies that it apparently requires setting /sys/module/vfio/parameters/enable_unsafe_noiommu_mode=1 in order to work (or did I misunderstand that part). If the hardware exists, and if users need to be able to set each VF's MAC address and vlan tag via libvirt config, then we (the royal Open Source "we" :-) need to make it work somehow. If there is no other way to address a netlink message to the PF telling it to set the MAC address and vlan tag of a VF, then that may be needed. If it can be saved/set in some other *standard* way, then perhaps libvirt can grow support for it. Long emails with actual information are always preferable to an endless chain of short mails that reveal the situation in tiny bits and pieces :-)