8:42 p.m.
Dear list,
I just discovered the hard way that if the lower lever physical
interface of a macvlan bridge is disconnected (ie: by unplugging the eth
cable, resulting in no carrier), inter-guest network traffic from all
virtual machines bound to the disconnected interface is dropped.
This behavior surprises me, as with the classic bridges I can disconnect
the underlying physical interface without causing any harm to
inter-guest traffic.
Am I doing something wrong, or this really is the expected behavior? If
so, can I force the macvtap interfaces to bridge traffic even when the
underlying physical interface is disconnected?
Regards.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
3:31 a.m.
On Mon, May 02, 2022 at 03:42:05AM +0200, Gionatan Danti wrote:
Dear list,
I just discovered the hard way that if the lower lever physical interface of
a macvlan bridge is disconnected (ie: by unplugging the eth cable, resulting
in no carrier), inter-guest network traffic from all virtual machines bound
to the disconnected interface is dropped.
This behavior surprises me, as with the classic bridges I can disconnect the
underlying physical interface without causing any harm to inter-guest
traffic.
Am I doing something wrong, or this really is the expected behavior? If so,
can I force the macvtap interfaces to bridge traffic even when the
underlying physical interface is disconnected?
Can you share the <interface> configuration for your guest NIC so we
can see how it is setup.
With regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
7:58 a.m.
On 5/3/22 4:31 AM, Daniel P. Berrangé wrote:
On Mon, May 02, 2022 at 03:42:05AM +0200, Gionatan Danti wrote:
> Dear list,
> I just discovered the hard way that if the lower lever physical interface of
> a macvlan bridge is disconnected (ie: by unplugging the eth cable, resulting
> in no carrier), inter-guest network traffic from all virtual machines bound
> to the disconnected interface is dropped.
>
> This behavior surprises me, as with the classic bridges I can disconnect the
> underlying physical interface without causing any harm to inter-guest
> traffic.
>
> Am I doing something wrong, or this really is the expected behavior? If so,
> can I force the macvtap interfaces to bridge traffic even when the
> underlying physical interface is disconnected?
Can you share the <interface> configuration for your guest NIC so we
can see how it is setup.
I can't say that I've ever tried this, since my only reason for using
macvtap is to provide the guests with direct connectivity to the
physical network, and unplugging the physdev negates that. The behavior
you describe doesn't surprise me all that much though, since the
physical device in the case of a host bridge isn't an integral part of
the bridge (it's just one more device attached to a port), while the
physical device and macvlan bridge a much more closely associated.
I'm Cc'ing Michael Tsirkin to see if he has more authoritative
information on whether or not the macvtaps connected to a macvlan bridge
can communicate amongst themselves when the physdev is disconnected.
In the meantime, is there a reason you don't want to just use a standard
host bridge that's not connected to any physdev? The one thing I can
think of is that you might not want to allow communication between the
host and guests, but as long as the bridge itself isn't given an IP
address, that won't be possible (at least at the level of IP).
4:15 p.m.
Il 2022-05-03 14:58 Laine Stump ha scritto:
I can't say that I've ever tried this, since my only reason
for using
macvtap is to provide the guests with direct connectivity to the
physical network, and unplugging the physdev negates that. The
behavior you describe doesn't surprise me all that much though, since
the physical device in the case of a host bridge isn't an integral
part of the bridge (it's just one more device attached to a port),
while the physical device and macvlan bridge a much more closely
associated.
Yeah, it sound very plausible.
I'm Cc'ing Michael Tsirkin to see if he has more
authoritative
information on whether or not the macvtaps connected to a macvlan
bridge can communicate amongst themselves when the physdev is
disconnected.
Thanks.
In the meantime, is there a reason you don't want to just use a
standard host bridge that's not connected to any physdev? The one
thing I can think of is that you might not want to allow communication
between the host and guests, but as long as the bridge itself isn't
given an IP address, that won't be possible (at least at the level of
IP).
I generally use plain bridge for my KVM setup. Specifically, when using
VLANs I setup the following:
eth -> eth.xx -> bridge -> vnet
This time, however, I need *both* a trunk-enabled VM (a virtual
firewall) and other segregated virtual machines. A "plain" bridge setup
would be something as:
eth -> bridge -> bridge.xx -> bridge -> vnet
Notice the two bridges, needed because bridge.xx is a VLAN interface
when no vnet can be directly attached. To avoid the double bridges, I
tried the following:
eth -> bridge -> bridge.xx -> macvtap
It seems to work very well but, during testing, I discovered that if the
interface under the macvtap one (in this case the bridge itself) goes
down, inter-guest networking is lost. As a side note, in the specific
scenario I described above, such issues can not really happen: as a vnet
interface is going to be always bound to the first bridge, it will be
*always* up due to the vnet interface itself being always up
(irrespective of the physical link status) and forcing the bridge up.
However, working so well, I thought to change my classical bridge setup
with a macvtap based one even for simpler installation. In short, going
from:
eth -> bridge -> vnet
to:
eth -> macvtap
But this very simple setup is going deny all guest traffic should the
physical interface become disconnected. A very crude solution would be
to issues "ip link set macvtap0 protodown off" when the physical link
goes down, but I wonder if a better solution exists.
That said, is replacing classical bridges with macvtap interfaces a bad
idea? Anything I should know before doing that?
Regards.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
5:03 p.m.
On Tue, May 03, 2022 at 11:15:26PM +0200, Gionatan Danti wrote:
Il 2022-05-03 14:58 Laine Stump ha scritto:
> I can't say that I've ever tried this, since my only reason for using
> macvtap is to provide the guests with direct connectivity to the
> physical network, and unplugging the physdev negates that. The
> behavior you describe doesn't surprise me all that much though, since
> the physical device in the case of a host bridge isn't an integral
> part of the bridge (it's just one more device attached to a port),
> while the physical device and macvlan bridge a much more closely
> associated.
Yeah, it sound very plausible.
> I'm Cc'ing Michael Tsirkin to see if he has more authoritative
> information on whether or not the macvtaps connected to a macvlan
> bridge can communicate amongst themselves when the physdev is
> disconnected.
Thanks.
> In the meantime, is there a reason you don't want to just use a
> standard host bridge that's not connected to any physdev? The one
> thing I can think of is that you might not want to allow communication
> between the host and guests, but as long as the bridge itself isn't
> given an IP address, that won't be possible (at least at the level of
> IP).
I generally use plain bridge for my KVM setup. Specifically, when using
VLANs I setup the following:
eth -> eth.xx -> bridge -> vnet
This time, however, I need *both* a trunk-enabled VM (a virtual firewall)
and other segregated virtual machines. A "plain" bridge setup would be
something as:
eth -> bridge -> bridge.xx -> bridge -> vnet
Notice the two bridges, needed because bridge.xx is a VLAN interface when no
vnet can be directly attached. To avoid the double bridges, I tried the
following:
eth -> bridge -> bridge.xx -> macvtap
It seems to work very well but, during testing, I discovered that if the
interface under the macvtap one (in this case the bridge itself) goes down,
inter-guest networking is lost. As a side note, in the specific scenario I
described above, such issues can not really happen: as a vnet interface is
going to be always bound to the first bridge, it will be *always* up due to
the vnet interface itself being always up (irrespective of the physical link
status) and forcing the bridge up.
well this is kind of by design. if we don't force guest's link
down it will think it can access internet, with all packets
being lost.
I see how you might want that in some cases - maybe we
should consider an option to disable this behaviour.
However, working so well, I thought to change my classical bridge
setup with
a macvtap based one even for simpler installation. In short, going from:
eth -> bridge -> vnet
to:
eth -> macvtap
But this very simple setup is going deny all guest traffic should the
physical interface become disconnected. A very crude solution would be to
issues "ip link set macvtap0 protodown off" when the physical link goes
down, but I wonder if a better solution exists.
That said, is replacing classical bridges with macvtap interfaces a bad
idea? Anything I should know before doing that?
Regards.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
2:24 a.m.
Il 2022-05-04 00:03 Michael S. Tsirkin ha scritto:
well this is kind of by design. if we don't force guest's
link
down it will think it can access internet, with all packets
being lost.
I understand. But in my opinion, inter-guest communication should be
preserved because:
a) this is how the classical bridge mode works;
b) modern guest have internal checks for internet connection;
c) guest-to-guest communication should not be impaired by (lack of)
internet access.
I see how you might want that in some cases - maybe we
should consider an option to disable this behaviour.
It would be great - especially as default behavior.
In the meantime, do we have some virsh command (or libvirt xml snippet)
to work around the issue, rather than messing with "ip link protodown
off"?
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
1:42 a.m.
Il 2022-05-03 23:15 Gionatan Danti ha scritto:
I generally use plain bridge for my KVM setup. Specifically, when
using VLANs I setup the following:
eth -> eth.xx -> bridge -> vnet
This time, however, I need *both* a trunk-enabled VM (a virtual
firewall) and other segregated virtual machines. A "plain" bridge
setup would be something as:
eth -> bridge -> bridge.xx -> bridge -> vnet
Notice the two bridges, needed because bridge.xx is a VLAN interface
when no vnet can be directly attached. To avoid the double bridges, I
tried the following:
eth -> bridge -> bridge.xx -> macvtap
It seems to work very well but, during testing, I discovered that if
the interface under the macvtap one (in this case the bridge itself)
goes down, inter-guest networking is lost. As a side note, in the
specific scenario I described above, such issues can not really
happen: as a vnet interface is going to be always bound to the first
bridge, it will be *always* up due to the vnet interface itself being
always up (irrespective of the physical link status) and forcing the
bridge up.
However, working so well, I thought to change my classical bridge
setup with a macvtap based one even for simpler installation. In
short, going from:
eth -> bridge -> vnet
to:
eth -> macvtap
But this very simple setup is going deny all guest traffic should the
physical interface become disconnected. A very crude solution would be
to issues "ip link set macvtap0 protodown off" when the physical link
goes down, but I wonder if a better solution exists.
That said, is replacing classical bridges with macvtap interfaces a
bad idea? Anything I should know before doing that?
Regards.
Hi all,
any comment / suggestion on the steps described above? Does a simpler
approach exists?
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
12:29 p.m.
Il 2022-05-09 08:42 Gionatan Danti ha scritto:
Hi all,
any comment / suggestion on the steps described above? Does a simpler
approach exists?
Hi list,
any insight on bridge vs macvlan setup previously described? Am I
over-thinking it?
Thanks.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
3:59 p.m.
Il 2022-05-03 10:31 Daniel P. Berrangé ha scritto:
Can you share the <interface> configuration for your guest NIC
so we
can see how it is setup.
Here we go...
<interface type='direct'>
<mac address='xx:xx:xx:xx:xx:xx'/>
<source dev='enp0s31f6' mode='bridge'/>
<target dev='macvtap0'/>
<model type='virtio'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x01'
slot='0x00'
function='0x0'/>
</interface>
# ifconfig
Both the physical and macvtap interfaces are disconnected (note how
RUNNING is missing):
enp0s31f6: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether xx:xx:xx:xx:xx:xx txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
device interrupt 16 memory 0xf4300000-f4320000
macvtap0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
ether xx:xx:xx:xx:xx:xx txqueuelen 500 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# ip link
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc
fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
6: macvtap0@enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500
qdisc noqueue state LOWERLAYERDOWN mode DEFAULT group default qlen 500
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
I can bring up the macvtap interface with the following command (notice
the RUNNING/UP flag):
# ip link set macvtap0 protodown off
# ifconfig
macvtap0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet6 fe80::redacted prefixlen 64 scopeid 0x20<link>
ether xx:xx:xx:xx:xx:xx txqueuelen 500 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 42 bytes 6001 (6.0 KB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
# ip link
6: macvtap0@enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc
noqueue state UP mode DEFAULT group default qlen 500
link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
However, at the next connection/disconnection of the physical link, the
macvtap interface will go down again.
Regards.
--
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti(a)assyoma.it - info(a)assyoma.it
GPG public key ID: FF5F32A8
799
days inactive
816
days old
8 comments
4 participants
participants (4)
-
Daniel P. Berrangé -
Gionatan Danti -
Laine Stump -
Michael S. Tsirkin