macvtap with disconnected physical interface
Dear list, I just discovered the hard way that if the lower lever physical interface of a macvlan bridge is disconnected (ie: by unplugging the eth cable, resulting in no carrier), inter-guest network traffic from all virtual machines bound to the disconnected interface is dropped. This behavior surprises me, as with the classic bridges I can disconnect the underlying physical interface without causing any harm to inter-guest traffic. Am I doing something wrong, or this really is the expected behavior? If so, can I force the macvtap interfaces to bridge traffic even when the underlying physical interface is disconnected? Regards. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
On Mon, May 02, 2022 at 03:42:05AM +0200, Gionatan Danti wrote:
Dear list, I just discovered the hard way that if the lower lever physical interface of a macvlan bridge is disconnected (ie: by unplugging the eth cable, resulting in no carrier), inter-guest network traffic from all virtual machines bound to the disconnected interface is dropped.
This behavior surprises me, as with the classic bridges I can disconnect the underlying physical interface without causing any harm to inter-guest traffic.
Am I doing something wrong, or this really is the expected behavior? If so, can I force the macvtap interfaces to bridge traffic even when the underlying physical interface is disconnected?
Can you share the <interface> configuration for your guest NIC so we can see how it is setup. With regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
On 5/3/22 4:31 AM, Daniel P. Berrangé wrote:
On Mon, May 02, 2022 at 03:42:05AM +0200, Gionatan Danti wrote:
Dear list, I just discovered the hard way that if the lower lever physical interface of a macvlan bridge is disconnected (ie: by unplugging the eth cable, resulting in no carrier), inter-guest network traffic from all virtual machines bound to the disconnected interface is dropped.
This behavior surprises me, as with the classic bridges I can disconnect the underlying physical interface without causing any harm to inter-guest traffic.
Am I doing something wrong, or this really is the expected behavior? If so, can I force the macvtap interfaces to bridge traffic even when the underlying physical interface is disconnected?
Can you share the <interface> configuration for your guest NIC so we can see how it is setup.
I can't say that I've ever tried this, since my only reason for using macvtap is to provide the guests with direct connectivity to the physical network, and unplugging the physdev negates that. The behavior you describe doesn't surprise me all that much though, since the physical device in the case of a host bridge isn't an integral part of the bridge (it's just one more device attached to a port), while the physical device and macvlan bridge a much more closely associated. I'm Cc'ing Michael Tsirkin to see if he has more authoritative information on whether or not the macvtaps connected to a macvlan bridge can communicate amongst themselves when the physdev is disconnected. In the meantime, is there a reason you don't want to just use a standard host bridge that's not connected to any physdev? The one thing I can think of is that you might not want to allow communication between the host and guests, but as long as the bridge itself isn't given an IP address, that won't be possible (at least at the level of IP).
Il 2022-05-03 14:58 Laine Stump ha scritto:
I can't say that I've ever tried this, since my only reason for using macvtap is to provide the guests with direct connectivity to the physical network, and unplugging the physdev negates that. The behavior you describe doesn't surprise me all that much though, since the physical device in the case of a host bridge isn't an integral part of the bridge (it's just one more device attached to a port), while the physical device and macvlan bridge a much more closely associated.
Yeah, it sound very plausible.
I'm Cc'ing Michael Tsirkin to see if he has more authoritative information on whether or not the macvtaps connected to a macvlan bridge can communicate amongst themselves when the physdev is disconnected.
Thanks.
In the meantime, is there a reason you don't want to just use a standard host bridge that's not connected to any physdev? The one thing I can think of is that you might not want to allow communication between the host and guests, but as long as the bridge itself isn't given an IP address, that won't be possible (at least at the level of IP).
I generally use plain bridge for my KVM setup. Specifically, when using VLANs I setup the following: eth -> eth.xx -> bridge -> vnet This time, however, I need *both* a trunk-enabled VM (a virtual firewall) and other segregated virtual machines. A "plain" bridge setup would be something as: eth -> bridge -> bridge.xx -> bridge -> vnet Notice the two bridges, needed because bridge.xx is a VLAN interface when no vnet can be directly attached. To avoid the double bridges, I tried the following: eth -> bridge -> bridge.xx -> macvtap It seems to work very well but, during testing, I discovered that if the interface under the macvtap one (in this case the bridge itself) goes down, inter-guest networking is lost. As a side note, in the specific scenario I described above, such issues can not really happen: as a vnet interface is going to be always bound to the first bridge, it will be *always* up due to the vnet interface itself being always up (irrespective of the physical link status) and forcing the bridge up. However, working so well, I thought to change my classical bridge setup with a macvtap based one even for simpler installation. In short, going from: eth -> bridge -> vnet to: eth -> macvtap But this very simple setup is going deny all guest traffic should the physical interface become disconnected. A very crude solution would be to issues "ip link set macvtap0 protodown off" when the physical link goes down, but I wonder if a better solution exists. That said, is replacing classical bridges with macvtap interfaces a bad idea? Anything I should know before doing that? Regards. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
On Tue, May 03, 2022 at 11:15:26PM +0200, Gionatan Danti wrote:
Il 2022-05-03 14:58 Laine Stump ha scritto:
I can't say that I've ever tried this, since my only reason for using macvtap is to provide the guests with direct connectivity to the physical network, and unplugging the physdev negates that. The behavior you describe doesn't surprise me all that much though, since the physical device in the case of a host bridge isn't an integral part of the bridge (it's just one more device attached to a port), while the physical device and macvlan bridge a much more closely associated.
Yeah, it sound very plausible.
I'm Cc'ing Michael Tsirkin to see if he has more authoritative information on whether or not the macvtaps connected to a macvlan bridge can communicate amongst themselves when the physdev is disconnected.
Thanks.
In the meantime, is there a reason you don't want to just use a standard host bridge that's not connected to any physdev? The one thing I can think of is that you might not want to allow communication between the host and guests, but as long as the bridge itself isn't given an IP address, that won't be possible (at least at the level of IP).
I generally use plain bridge for my KVM setup. Specifically, when using VLANs I setup the following: eth -> eth.xx -> bridge -> vnet
This time, however, I need *both* a trunk-enabled VM (a virtual firewall) and other segregated virtual machines. A "plain" bridge setup would be something as: eth -> bridge -> bridge.xx -> bridge -> vnet
Notice the two bridges, needed because bridge.xx is a VLAN interface when no vnet can be directly attached. To avoid the double bridges, I tried the following: eth -> bridge -> bridge.xx -> macvtap
It seems to work very well but, during testing, I discovered that if the interface under the macvtap one (in this case the bridge itself) goes down, inter-guest networking is lost. As a side note, in the specific scenario I described above, such issues can not really happen: as a vnet interface is going to be always bound to the first bridge, it will be *always* up due to the vnet interface itself being always up (irrespective of the physical link status) and forcing the bridge up.
well this is kind of by design. if we don't force guest's link down it will think it can access internet, with all packets being lost. I see how you might want that in some cases - maybe we should consider an option to disable this behaviour.
However, working so well, I thought to change my classical bridge setup with a macvtap based one even for simpler installation. In short, going from: eth -> bridge -> vnet to: eth -> macvtap
But this very simple setup is going deny all guest traffic should the physical interface become disconnected. A very crude solution would be to issues "ip link set macvtap0 protodown off" when the physical link goes down, but I wonder if a better solution exists.
That said, is replacing classical bridges with macvtap interfaces a bad idea? Anything I should know before doing that? Regards.
-- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Il 2022-05-04 00:03 Michael S. Tsirkin ha scritto:
well this is kind of by design. if we don't force guest's link down it will think it can access internet, with all packets being lost.
I understand. But in my opinion, inter-guest communication should be preserved because: a) this is how the classical bridge mode works; b) modern guest have internal checks for internet connection; c) guest-to-guest communication should not be impaired by (lack of) internet access.
I see how you might want that in some cases - maybe we should consider an option to disable this behaviour.
It would be great - especially as default behavior. In the meantime, do we have some virsh command (or libvirt xml snippet) to work around the issue, rather than messing with "ip link protodown off"? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Il 2022-05-03 23:15 Gionatan Danti ha scritto:
I generally use plain bridge for my KVM setup. Specifically, when using VLANs I setup the following: eth -> eth.xx -> bridge -> vnet
This time, however, I need *both* a trunk-enabled VM (a virtual firewall) and other segregated virtual machines. A "plain" bridge setup would be something as: eth -> bridge -> bridge.xx -> bridge -> vnet
Notice the two bridges, needed because bridge.xx is a VLAN interface when no vnet can be directly attached. To avoid the double bridges, I tried the following: eth -> bridge -> bridge.xx -> macvtap
It seems to work very well but, during testing, I discovered that if the interface under the macvtap one (in this case the bridge itself) goes down, inter-guest networking is lost. As a side note, in the specific scenario I described above, such issues can not really happen: as a vnet interface is going to be always bound to the first bridge, it will be *always* up due to the vnet interface itself being always up (irrespective of the physical link status) and forcing the bridge up.
However, working so well, I thought to change my classical bridge setup with a macvtap based one even for simpler installation. In short, going from: eth -> bridge -> vnet to: eth -> macvtap
But this very simple setup is going deny all guest traffic should the physical interface become disconnected. A very crude solution would be to issues "ip link set macvtap0 protodown off" when the physical link goes down, but I wonder if a better solution exists.
That said, is replacing classical bridges with macvtap interfaces a bad idea? Anything I should know before doing that? Regards.
Hi all, any comment / suggestion on the steps described above? Does a simpler approach exists? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Il 2022-05-09 08:42 Gionatan Danti ha scritto:
Hi all, any comment / suggestion on the steps described above? Does a simpler approach exists?
Hi list, any insight on bridge vs macvlan setup previously described? Am I over-thinking it? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
Il 2022-05-03 10:31 Daniel P. Berrangé ha scritto:
Can you share the <interface> configuration for your guest NIC so we can see how it is setup.
Here we go... <interface type='direct'> <mac address='xx:xx:xx:xx:xx:xx'/> <source dev='enp0s31f6' mode='bridge'/> <target dev='macvtap0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> # ifconfig Both the physical and macvtap interfaces are disconnected (note how RUNNING is missing): enp0s31f6: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether xx:xx:xx:xx:xx:xx txqueuelen 1000 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device interrupt 16 memory 0xf4300000-f4320000 macvtap0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500 ether xx:xx:xx:xx:xx:xx txqueuelen 500 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 # ip link 2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000 link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff 6: macvtap0@enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 500 link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff I can bring up the macvtap interface with the following command (notice the RUNNING/UP flag): # ip link set macvtap0 protodown off # ifconfig macvtap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::redacted prefixlen 64 scopeid 0x20<link> ether xx:xx:xx:xx:xx:xx txqueuelen 500 (Ethernet) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 42 bytes 6001 (6.0 KB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 # ip link 6: macvtap0@enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 500 link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff However, at the next connection/disconnection of the physical link, the macvtap interface will go down again. Regards. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti@assyoma.it - info@assyoma.it GPG public key ID: FF5F32A8
participants (4)
-
Daniel P. Berrangé -
Gionatan Danti -
Laine Stump -
Michael S. Tsirkin