Re: Could you please help with questions about the net failover feature

On 7/8/20 1:30 AM, Stefan Assmann wrote: > On 2020-07-06 10:01, Laine Stump wrote: >> On 7/6/20 5:10 AM, Yalan Zhang wrote: >>> Hi Laine, >>> >>> For the feature testing before, I only test the linux bridge setting as >>> in 2), it works. >>> Now I tried 1), to use macvtap bridge mode connected to the PF, it can >>> not work as the hostdev interface can not get dhcp ip address on the >>> guest. >>> Check on host, the /var/log/messages and dmesg both says: >>> >>> "Jul  6 04:54:45 dell-per730-xx kernel: ixgbe 0000:82:00.1 >>> enp130s0f1: 1 >>> Spoofed packets detected >>> ...... >>> Jul  6 04:56:17 dell-per730-xx kernel: ixgbe 0000:82:00.1 enp130s0f1: 1 >>> Spoofed packets detected >>> Jul  6 04:56:54 dell-per730-xx kernel: ixgbe 0000:82:00.1 enp130s0f1: 1 >>> Spoofed packets detected >>> " >>> (enp130s0f1 is the PF's interface name, and 0000:82:00.1 is the PF's >>> pci >>> address) >>> # rpm -q kernel >>> kernel-4.18.0-193.4.1.el8_2.x86_64 >>> >>> Could you please help to confirm if this is a kernel bug?  Thank you >>> very much! >> Interesting. I'm not sure if this is expected behavior, or if it's >> improper >> behavior and it just hasn't been tested before (obviously based on my >> earlier recommendation, I think it *should* be able to work like >> this, and I >> *thought* I had tried it, but maybe I just imagined it :-/). >> >> I'm Cc'ing Stefan Assmann to see if he has an opinion on whether or >> not this >> should work. For his convenience, here is a summary of the config: >> The setup >> is that there is a bridge-mode macvtap interface on the PF, and one >> of the >> VF's has been given the same MAC address as the macvtap. the macvtap >> interface is connected to an emulated NIC in the guest, and the VF is >> assigned to the guest with VFIO. IIUC, the problem is using the same mac address for macvtap and for a VF.  This is what's causing the spoofed packets.

> >> I'll try this today on my setup, which uses I350 cards (igb driver). >> >> >>> >>> >>> You have two choices for the backup virtio interface: >>> >>> 1) it can be a macvtap device connected to the PF of the same SRIOV >>> device. >>> >>> 2) it can be a standard tap device connected to a Linux host bridge >>> (created outside libvirt in the host system network config) that is >>> attached to the PF (or alternately one of the VFs that isn't being used >>> for VMs, or to another physical ethernet adapter on the host that is >>> connected to the same network. >>> >>> >>> >>> >>> ------- >>> Best Regards, >>> Yalan Zhang >>> IRC: yalzhang >>> >>> >>> On Sun, Mar 22, 2020 at 6:50 AM Laine Stump <laine(a)redhat.com >>> <mailto:laine@redhat.com>> wrote: >>> >>>      On 3/21/20 1:08 AM, Yalan Zhang wrote: >>> >>>       > In my understanding, the standby and primary hostdev interface >>>      may be in >>>       > different subnet. >>> >>>      There is only one hostdev device in the team pair (that will be >>> the one >>>      with <teaming type='transient'.../> since it needs to be unplugged >>>      during migration). The other device must be a virtio device >>> (the one >>>      with <teaming type='persistent'/>). And no, they cannot be on >>> different >>>      subnets. They must both connect into the same ethernet "collision >>>      domain", such that the guest could assign the same IP address >>> to either >>>      of them and be able to communicate on the network. >>> >>>      There is some explanation of the use case for this option. and >>> some >>>      example config, here: >>> >>>      https://www.libvirt.org/formatdomain.html#elementsTeaming >>> >>>       > I'm not sure whether it is correct. Could you please help to >>>      explain? >>>       > Thank you in advance. >>>       > >>>       > For example, primary hostdev is connected to vf-pool with >>>      <pf='eth0'/>, >>>       > while the standby is connected to NAT network with " forward >>>      dev='eth0'". >>>       > The standby interface will get ip as 192.168.122.x, but after >>>      NAT, it >>>       > will be in the same subnet of the vf. >>>        > >>>       > So after the VF is unplugged, the packet will still >>> broadcast in the >>>       > same subnet, and the vm will get the packet as the standby >>> share the >>>       > same mac. Right? >>> >>>      No, not right :-) >>> >>>      The VF of an SRIOV network adapter is connected directly to the >>>      physical >>>      network, and will have an IP address that is on that network. Tap >>>      devices plugged into the default network (or any other libvirt >>> network >>>      based on a bridge device that is created/managed by libvirt) >>> have no >>>      direct connection to the physical network, and are on a different >>>      subnet. The fact that traffic from the guest *seems* to be >>> coming from >>>      an IP on the physical subnet is meaningless. The *guest* needs >>> to be >>>      able to use both NICs using the same IP address, and anything >>> plugged >>>      into the default network will need to have an IP address on a >>> different >>>      subnet from the perspective of the guest. >>> >>>      You have two choices for the backup virtio interface: >>> >>>      1) it can be a macvtap device connected to the PF of the same >>> SRIOV >>>      device. >>> >>>      2) it can be a standard tap device connected to a Linux host >>> bridge >>>      (created outside libvirt in the host system network config) >>> that is >>>      attached to the PF (or alternately one of the VFs that isn't >>> being used >>>      for VMs, or to another physical ethernet adapter on the host >>> that is >>>      connected to the same network. >>> >>> >>>      It is simplest to have the same name refer to the connection on >>> the >>>      source and destination hosts of a migration. That can be >>> handled by >>>      creating a libvirt network to refer to the bridge device created >>>      outside >>>      libvirt (or to the PF directly if you're going to use macvtap. >>> >>>      For example, if you're going to use macvtap, and the PF's name >>> on the >>>      host is ens4f0, you'd just create this network: >>> >>>          <network> >>>            <name>persistent-net</name> >>>            <forward mode='bridge'> >>>              <interface dev='ens4f0'/> >>>            </forward> >>>           <network> >>> >>>      any guest interface with this: >>> >>>             <interface type='network'> >>>               <source network='persistent-net'/> >>>               <mac address='00:11:22:33:44:55'/> >>>               <model type='virtio'/> >>>               <teaming type='persistent'/> >>>               <alias name='ua-backup0'/> >>>             </interface> >>> >>>      will get a macvtap device that's connected to ens4f0 in bridge >>> mode. >>> >>>      Or, if your host has a bridge device called br0 that is directly >>>      attached to the physical network (in whatever manner, it doesn't >>>      matter), you can define the network this way: >>> >>>          <network> >>>            <name>persistent-net</name> >>>            <bridge name='br0'/> >>>            <forward mode='bridge'/> >>>           <network> >>> >>>      The XML for the guest interface would be the same. >>> >>>      Then for the vfio (transient) interface, you could also define a >>>      network: >>> >>>           <network> >>>             <name>transient-net</name> >>>             <forward mode='hostdev'> >>>               <pf dev='ens4f0'/> >>>             </forward> >>>           </network> >>> >>>      and instead of using <interface type='hostdev'> in the guest >>> config, >>>      you >>>      would use this: >>> >>> >>>            <interface type='network'> >>>              <source network='transient-net'/> >>>              <model type='virtio'/>    [1] >>>              <mac address='00:11:22:33:44:55'/> >>>              <teaming type='transient' persistent='ua-backup0'/> >>>           </interface> >>> >>>      Even if the device names change on the other host (the >>> destination of >>>      the migration), as long as the other host has networks named >>>      "persistent-net" and "transient-net" that are of similar types >>> (macvtap >>>      or bridged for persistent-net, and hostdev for transient-net) then >>>      libvirt will be able to migrate the guest from one host to the >>> other >>>      with no mangling of the XML required. >>> >>>