9:23 a.m.
Hi all,
I'm observing an issue that as soon as libvirt starts, UPD broadcasts
going through physical network (and unrelated to any virtualization) get
broken. Specifically, windows neighbourhood browsing through samba's
nmbd starts suffering badly (Samba is running on this same box).
At the moment I'm running a quite outdated version 1.2.9 of libvirt, but
other than this issue, it does its job pretty well, so I'd first
consider some patching/backporting rather than totally replacing it with
a new one. Anyway, I first need to better understand what is going on
and what is wrong with it.
This could also be related somewhat to
https://www.redhat.com/archives/libvir-list/2013-September/msg01311.html
but I suppose it is not exactly that thing.
I've already figured the source of trouble is anyway related to these
rules added:
-A POSTROUTING -o br0 -j MASQUERADE
-A POSTROUTING -o enp0s25 -j MASQUERADE
-A POSTROUTING -o virbr2_nic -j MASQUERADE
-A POSTROUTING -o vnet0 -j MASQUERADE
Here, virbr2_nic and vnet0 are used by libvirt for arranging network
configurations for VMs, ok. However, br0 is a main interface of this
host with primary ip address, with enp0s25 being a physical nic of this
host, and it is used for all sorts of regular (unrelated to
virtualization) communications. Also, br0 is used for attaching bridged
(as opposed to NATed) VMs managed by libvirt.
Clearly, libvirt somehow chooses to set up masquerading for literally
all existing network interfaces here (except lo), but I can't see a real
reason for the first two rules in the list above. Furthermore, they
corrupt UDP broadcats coming from outside and reaching this host
(through enp0s25/br0) such that source address gets replaced by this
hosts primary address (as per masquerading). I've verified this by
arranging a hand-crafted UDP listener and printing the respective source
addresses as seen by normal userspace.
Now I've discovered that I can "eliminate" the problem by either:
1. Removing "-A POSTROUTING -o br0 -j MASQUERADE" (manually)
2. Inserting "-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.255/32 -j
ACCEPT"
(Of course correcting rules by hand is not a solution, just a test)
So question is, how the correct rules should ideally look like? And, is
this issue known/fixed in most current libvirt?
Thank you,
Regards,
Nikolai
2 p.m.
On 6/28/19 10:23 AM, Nikolai Zhubr wrote:
Hi all,
I'm observing an issue that as soon as libvirt starts, UPD broadcasts
going through physical network (and unrelated to any virtualization) get
broken. Specifically, windows neighbourhood browsing through samba's
nmbd starts suffering badly (Samba is running on this same box).
At the moment I'm running a quite outdated version 1.2.9 of libvirt, but
other than this issue, it does its job pretty well, so I'd first
consider some patching/backporting rather than totally replacing it with
a new one. Anyway, I first need to better understand what is going on
and what is wrong with it.
This could also be related somewhat to
https://www.redhat.com/archives/libvir-list/2013-September/msg01311.html
but I suppose it is not exactly that thing.
I've already figured the source of trouble is anyway related to these
rules added:
-A POSTROUTING -o br0 -j MASQUERADE
-A POSTROUTING -o enp0s25 -j MASQUERADE
-A POSTROUTING -o virbr2_nic -j MASQUERADE
-A POSTROUTING -o vnet0 -j MASQUERADE
*None* of those rules were added by libvirt (unless your build of
libvirt, in addition to being ancient, has also been heavily hacked by a
third party with downstream-only patches, although I can't imagine how
the rules you show could possibly be a useful addition).
Detailed documentation of what iptables rules are added by libvirt can
be found here:
https://libvirt.org/firewall.html
(particularly in the "virtual network" section).
The masquerade rules added by libvirt are based on the IP address of the
NATed subnet, e.g.:
A LIBVIRT_PRT -s 192.168.12.0/24 ! -d 192.168.12.0/24 -p tcp \
-j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.12.0/24 ! -d 192.168.12.0/24 -p udp \
-j MASQUERADE --to-ports 1024-65535
-A LIBVIRT_PRT -s 192.168.12.0/24 ! -d 192.168.12.0/24 \
-j MASQUERADE
(this is from a less ancient version of libvirt that puts the rules on
its own private chain, but in the version you are using the rule would
be the same, except that "LIBVIRT_PRT" would be replaced with
"POSTROUTING")
You can verify my "counter-claim" by running "virsh net-destroy" for
all
of your libvirt networks, and seeing that the offending rules haven't
been removed.
In short, you need to look elsewhere for the culprit.
Here, virbr2_nic and vnet0 are used by libvirt for arranging network
configurations for VMs, ok. However, br0 is a main interface of this
host with primary ip address, with enp0s25 being a physical nic of this
host, and it is used for all sorts of regular (unrelated to
virtualization) communications. Also, br0 is used for attaching bridged
(as opposed to NATed) VMs managed by libvirt.
Clearly, libvirt somehow chooses to set up masquerading for literally
all existing network interfaces here (except lo),
It's clear that the rules are there. It's not clear that they were added
by libvirt.
but I can't see a real
reason for the first two rules in the list above. Furthermore, they
corrupt UDP broadcats coming from outside and reaching this host
(through enp0s25/br0) such that source address gets replaced by this
hosts primary address (as per masquerading). I've verified this by
arranging a hand-crafted UDP listener and printing the respective source
addresses as seen by normal userspace.
Now I've discovered that I can "eliminate" the problem by either:
1. Removing "-A POSTROUTING -o br0 -j MASQUERADE" (manually)
2. Inserting "-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.255/32 -j
ACCEPT"
(Of course correcting rules by hand is not a solution, just a test)
So question is, how the correct rules should ideally look like? And, is
this issue known/fixed in most current libvirt?
Except for putting the libvirt-added rules in their own private chains
(appearing in libvirt 5.1.0, released on Feb 1, 2019), the iptables
rules added by libvirt to support its virtual networks didn't materially
change in > 10 years. Your email is the first time I've ever seen such
rules attributed to libvirt so, as I said above, I think you need to
take a deeper dive into your host system's config.
Good luck!
11:26 a.m.
Hi Daniel and Laine,
[...]
> -A POSTROUTING -o br0 -j MASQUERADE
> -A POSTROUTING -o enp0s25 -j MASQUERADE
> -A POSTROUTING -o virbr2_nic -j MASQUERADE
> -A POSTROUTING -o vnet0 -j MASQUERADE
*None* of those rules were added by libvirt (unless your build of
[...]
You can verify my "counter-claim" by running "virsh
net-destroy" for all
of your libvirt networks, and seeing that the offending rules haven't
been removed.
In short, you need to look elsewhere for the culprit.
Yes, found it. You were both right, essentially.
The offending rules were added by a firewall in response to new
interfaces created by libvirt dynamically, due to some dubious relict
settings left in the firewall. (Silly me.)
So this it not an issue of libvirt indeed!
Tons of thanks for the quick and precise hit!
Regards,
Nikolai
>
> Here, virbr2_nic and vnet0 are used by libvirt for arranging network
> configurations for VMs, ok. However, br0 is a main interface of this
> host with primary ip address, with enp0s25 being a physical nic of
> this host, and it is used for all sorts of regular (unrelated to
> virtualization) communications. Also, br0 is used for attaching
> bridged (as opposed to NATed) VMs managed by libvirt.
>
> Clearly, libvirt somehow chooses to set up masquerading for literally
> all existing network interfaces here (except lo),
It's clear that the rules are there. It's not clear that they were added
by libvirt.
> but I can't see a real reason for the first two rules in the list
> above. Furthermore, they corrupt UDP broadcats coming from outside and
> reaching this host (through enp0s25/br0) such that source address gets
> replaced by this hosts primary address (as per masquerading). I've
> verified this by arranging a hand-crafted UDP listener and printing
> the respective source addresses as seen by normal userspace.
>
> Now I've discovered that I can "eliminate" the problem by either:
>
> 1. Removing "-A POSTROUTING -o br0 -j MASQUERADE" (manually)
> 2. Inserting "-A POSTROUTING -s 192.168.0.0/24 -d 192.168.0.255/32 -j
> ACCEPT"
> (Of course correcting rules by hand is not a solution, just a test)
>
> So question is, how the correct rules should ideally look like? And,
> is this issue known/fixed in most current libvirt?
Except for putting the libvirt-added rules in their own private chains
(appearing in libvirt 5.1.0, released on Feb 1, 2019), the iptables
rules added by libvirt to support its virtual networks didn't materially
change in > 10 years. Your email is the first time I've ever seen such
rules attributed to libvirt so, as I said above, I think you need to
take a deeper dive into your host system's config.
Good luck!
11:27 a.m.
On Fri, Jul 05, 2019 at 07:26:41PM +0300, Nikolai Zhubr wrote:
Hi Daniel and Laine,
[...]
> > -A POSTROUTING -o br0 -j MASQUERADE
> > -A POSTROUTING -o enp0s25 -j MASQUERADE
> > -A POSTROUTING -o virbr2_nic -j MASQUERADE
> > -A POSTROUTING -o vnet0 -j MASQUERADE
>
> *None* of those rules were added by libvirt (unless your build of
[...]
> You can verify my "counter-claim" by running "virsh net-destroy"
for all
> of your libvirt networks, and seeing that the offending rules haven't
> been removed.
>
> In short, you need to look elsewhere for the culprit.
Yes, found it. You were both right, essentially.
The offending rules were added by a firewall in response to new interfaces
created by libvirt dynamically, due to some dubious relict settings left in
the firewall. (Silly me.)
So this it not an issue of libvirt indeed!
Tons of thanks for the quick and precise hit!
No worries, thanks for confirming the root cause you found too.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1848
days inactive
1855
days old
3 comments
3 participants
participants (3)
-
Daniel P. Berrangé -
Laine Stump -
Nikolai Zhubr