Re: bridge + SR-IOV guests with KVM

On 8/27/20 4:12 AM, Philipp Rosenberger wrote: Do I correctly assume that you're attaching the PF of the SRIOV card to the bridge? (in particular, the PF of the same port that the VFs are from) I think you missed a paragraph here - I'm inferring that at this point you meant to say "But when I have both guests connected with an assigned VF and guests connected via an virtio-net device connected to the bridge via a tap device, the VF-connect guests cannot communicate with the bridge-connected guests." Is that correct, or am I inferring too much? By default all the ports on a Linux host bridge should have flood and learning turned on, and I would have thought that (if manually adding a mac address to the linux host bridge is enough to make the traffic flow) that having flood+learning turnes on would be enough to get the bridge to learn the proper port for traffic destined to the VF Really, my first assumption would have been that the switch screwing up was the switch in the SRIOV card incorrectly sending traffic for the bridged guests directly out the PF's physical port instead of rather than the Linux host bridge, and that the solution to make everything work would be to somehow add the MAC address of the *bridged* guests into the fdb of the SRIOV card. [at this point I read further through your message and follow the link to the slides...] Ooohh..... from slide 33, I see that *is* what's being done. It sounds like that's what you *are* doing - adding an fdb entry to the internal switch in the SRIOV card, *not* to the Linux host bridge (as I had thought right up until 3 paragraphs ago), correct? It again is interesting though - makes it sound like the switch in the SRIOV card has no learning and no flood. That definitely is what's said in the comments from the Intel forum. But isn't that the opposite of what they're saying on slide 33 of the presentation you linked to? My understanding is that it's saying that you need to add the *bridged* guest interface's MAC address to the fdb on the SRIOV card's switch. Ah, okay, this verifies my first assumption (that the bridge is attached to the PF). It also verifying that you did what's suggested in slide 33 of the presentation, not what was suggested in the Intel forum post. Is that correct? Well, if you create a libvirt network for your bridge device, something like this: <network> <name>bridgenet</name> <bridge name='br0'/> <forward mode='bridge'/> </network> (this is an "unmanaged" network, i.e. libvirt expects the bridge to already exist, doesn't add any iptables rules or dnsmasq instance - it just creates tap devices and connects them to the already-existing bridge), and then configure all your guest interfaces with: <interface type='network'> <source network='bridgenet'/> ... </interface> then a network hook will be called each time one of these interfaces is added or removed, and that script can add/remove the fdb entry. Hooks are described here: https://libvirt.org/hooks.html If you have an executable file named /etc/libvirt/hooks/network (or a file of your own naming in /etc/libvirt/hooks/network.d if your libvirt is 6.5.0 or newer) it will be called anytime a guest interface is added or removed from a libvirt network. The arguments will describe the current action, and stdin of the script will receive the full XML config of the network, as well as the xml for a <networkport>, which contains the interface's MAC address among other things. This should be enough information to derive the proper "bridge fdb add" command. (you'll want a similar clause in the same hook script that takes action when the interface is removed from the network). If you have trouble with the script, you can look in the #virt channel on irc.oftc.net - if it's a weekday and the right time of day, you may get immediate help (I'm actually curious to hear how this works out. It *almost* sounds like something that could be integrated into the standard config of libvirt, although I'm not sure how to make it easily consumable). (NB: you could also use the network hook script to perform some action when the VFs are added/removed from guests).