[libvirt-users] guest A from virbr0 can talk to guest B in virbr1 but not vice versa
Hello, I came across an interesting problem in my home lab a few weeks ago as I'm prepping for my RHCE exam using Michael Jang study guide. I've been at this for days now, and I still can't wrap my head around how two or more virtual networks in default NAT configuration are even allowed to communicate with each other despite what the libvirt documentation said. Here's the excerpt I'm referring to in the wiki link here: http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
By default, guests that are connected via a virtual network with <forward mode='nat'/> can make any outgoing network connection they like. Incoming connections are allowed from the host, and from other guests connected to the same libvirt network, but all other incoming connections are blocked by iptables rules.
Also here's another assertion from 'The virtual network driver' section in http://libvirt.org/firewall.html:
type=nat
Allow inbound related to an established connection. Allow outbound, but only from our expected subnet. Allow traffic between guests. Deny all other inbound. Deny all other outbound.
I have three virtual networks with the following configs: ----------------------------------------------------------------------------- <network connections='1'> <name>default</name> <uuid>9c6796be-d54e-42bc-bcbe-2e4feee7154a</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:5a:5d:0e'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network> <network connections='1'> <name>outsider</name> <uuid>247e380a-8795-466a-b94a-5be2d05267bb</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:7f:a1:fb'/> <domain name='outsider'/> <ip address='192.168.100.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.100.2' end='192.168.100.254'/> </dhcp> </ip> </network> <network connections='1'> <name>besider</name> <uuid>cc714cce-dbba-452d-b2bf-d36084dcb723</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr2' stp='on' delay='0'/> <mac address='52:54:00:59:67:7f'/> <domain name='besider'/> <ip address='192.168.110.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.110.2' end='192.168.110.254'/> </dhcp> </ip> </network> ---------------------------------------------------------------------------- Here is the output of the 'FORWARD' iptables chain rules on my host (still using firewall-cmd): ------------------------------------------------------------------------ Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 8967 14M ACCEPT all -- * virbr2 0.0.0.0/0 192.168.110.0/24 ctstate RELATED,ESTABLISHED 5262 279K ACCEPT all -- virbr2 * 192.168.110.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr2 virbr2 0.0.0.0/0 0.0.0.0/0 70 5832 REJECT all -- * virbr2 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr2 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 8510 13M ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 5177 275K ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0 61 5100 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 8612 13M ACCEPT all -- * virbr1 0.0.0.0/0 192.168.100.0/24 ctstate RELATED,ESTABLISHED 5172 273K ACCEPT all -- virbr1 * 192.168.100.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr1 virbr1 0.0.0.0/0 0.0.0.0/0 0 0 REJECT all -- * virbr1 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr1 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_direct all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_IN_ZONES_SOURCE all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_IN_ZONES all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_OUT_ZONES_SOURCE all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_OUT_ZONES all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited -------------------------------------------------------------------------- I have a VM in each network: nest1.example.com (virbr0) - 192.168.122.50 nest2.example.org (virbr1) - 192.168.100.100 nest3.example.net (virbr2) - 192.168.110.25 I'm quite aware the above iptables rules were added by libvirt, but I'm still managing the firewall primarily through the *firewall-cmd* command.
From what I gathered...
nest3 can ping nest1 and nest2. Nest3 ping nest1 and made a hit here: 8967 14M ACCEPT all -- * virbr2 0.0.0.0/0 192.168.110.0/24 ctstate RELATED,ESTABLISHED 5262 279K ACCEPT all -- virbr2 * 192.168.110.0/24 0.0.0.0/0 Nest3 ping nest2 and made a hit here: 8967 14M ACCEPT all -- * virbr2 0.0.0.0/0 192.168.110.0/24 ctstate RELATED,ESTABLISHED 5262 279K ACCEPT all -- virbr2 * 192.168.110.0/24 0.0.0.0/0 ---------------------------------------------------------- ---------------------------------------------------------- Nest1 can ping nest2, but cannot ping nest3. Nest1 ping nest2 and made a hit here: 8510 13M ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 5177 275K ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0 Nest1 ping nest3 and made a hit here: 70 5832 REJECT all -- * virbr2 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable ---------------------------------------------------------- ---------------------------------------------------------- Nest2 cannot ping nest1 and nest3. Nest2 ping nest1 and made a hit here: 61 5100 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable Nest2 ping test3 and made a hit here: 70 5832 REJECT all -- * virbr2 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable ----------------------------------------------------------
From my observation, I see that the order of the virtual networks in the iptables FORWARD chain makes a difference. Each chunk associated with the virtual network in the chain, consisting of five lines, is exactly as described in the aforementioned link (http://libvirt.org/firewall.html). The virtual network in the top chunk of the chain can communicate with virtually all networks as opposed to the network in the last chunk that is consistent with the intention of the original design.
I'm using CentOS 7.3 with libvirt 2.0. I even tried reproducing this with CentOS 6.9 as I thought it was possible the firewalld may have influenced the change, but I was still getting similar result. Right now, I'm not certain if this is already a reported known bug, but I'm highly convinced this configuration is unofficially unsupported for quite a while. Can someone confirm this? This is my very first mailing list submission ever, and I apologize in advance if I couldn't figure out how to conveniently search up a similar discussion as this one in the archive. If this is in fact near impossible to implement in accordance to the intended design, then I'd like to have this confirmation publicly included in the docs. Thanks, Travis Johnson
On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
Hello,
I came across an interesting problem in my home lab a few weeks ago as I'm prepping for my RHCE exam using Michael Jang study guide. I've been at this for days now, and I still can't wrap my head around how two or more virtual networks in default NAT configuration are even allowed to communicate with each other despite what the libvirt documentation said.
Here's the excerpt I'm referring to in the wiki link here: http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
By default, guests that are connected via a virtual network with <forward mode='nat'/> can make any outgoing network connection they like. Incoming connections are allowed from the host, and from other guests connected to the same libvirt network, but all other incoming connections are blocked by iptables rules.
Also here's another assertion from 'The virtual network driver' section in http://libvirt.org/firewall.html:
type=nat
Allow inbound related to an established connection. Allow outbound, but only from our expected subnet. Allow traffic between guests. Deny all other inbound. Deny all other outbound.
I have three virtual networks with the following configs: ----------------------------------------------------------------------------- <network connections='1'> <name>default</name> <uuid>9c6796be-d54e-42bc-bcbe-2e4feee7154a</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:5a:5d:0e'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> </dhcp> </ip> </network>
<network connections='1'> <name>outsider</name> <uuid>247e380a-8795-466a-b94a-5be2d05267bb</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr1' stp='on' delay='0'/> <mac address='52:54:00:7f:a1:fb'/> <domain name='outsider'/> <ip address='192.168.100.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.100.2' end='192.168.100.254'/> </dhcp> </ip> </network>
<network connections='1'> <name>besider</name> <uuid>cc714cce-dbba-452d-b2bf-d36084dcb723</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='virbr2' stp='on' delay='0'/> <mac address='52:54:00:59:67:7f'/> <domain name='besider'/> <ip address='192.168.110.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.110.2' end='192.168.110.254'/> </dhcp> </ip> </network> ----------------------------------------------------------------------------
Here is the output of the 'FORWARD' iptables chain rules on my host (still using firewall-cmd): ------------------------------------------------------------------------ Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 8967 14M ACCEPT all -- * virbr2 0.0.0.0/0 192.168.110.0/24 ctstate RELATED,ESTABLISHED 5262 279K ACCEPT all -- virbr2 * 192.168.110.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr2 virbr2 0.0.0.0/0 0.0.0.0/0 70 5832 REJECT all -- * virbr2 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr2 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 8510 13M ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 5177 275K ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr0 virbr0 0.0.0.0/0 0.0.0.0/0 61 5100 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 8612 13M ACCEPT all -- * virbr1 0.0.0.0/0 192.168.100.0/24 ctstate RELATED,ESTABLISHED 5172 273K ACCEPT all -- virbr1 * 192.168.100.0/24 0.0.0.0/0 0 0 ACCEPT all -- virbr1 virbr1 0.0.0.0/0 0.0.0.0/0 0 0 REJECT all -- * virbr1 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 REJECT all -- virbr1 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable 0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_direct all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_IN_ZONES_SOURCE all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_IN_ZONES all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_OUT_ZONES_SOURCE all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 FORWARD_OUT_ZONES all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID 0 0 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited --------------------------------------------------------------------------
I have a VM in each network: nest1.example.com (virbr0) - 192.168.122.50 nest2.example.org (virbr1) - 192.168.100.100 nest3.example.net (virbr2) - 192.168.110.25
I'm quite aware the above iptables rules were added by libvirt, but I'm still managing the firewall primarily through the *firewall-cmd* command.
From what I gathered...
nest3 can ping nest1 and nest2.
Nest3 ping nest1 and made a hit here: 8967 14M ACCEPT all -- * virbr2 0.0.0.0/0 192.168.110.0/24 ctstate RELATED,ESTABLISHED 5262 279K ACCEPT all -- virbr2 * 192.168.110.0/24 0.0.0.0/0
Nest3 ping nest2 and made a hit here: 8967 14M ACCEPT all -- * virbr2 0.0.0.0/0 192.168.110.0/24 ctstate RELATED,ESTABLISHED 5262 279K ACCEPT all -- virbr2 * 192.168.110.0/24 0.0.0.0/0 ----------------------------------------------------------
---------------------------------------------------------- Nest1 can ping nest2, but cannot ping nest3.
Nest1 ping nest2 and made a hit here: 8510 13M ACCEPT all -- * virbr0 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED 5177 275K ACCEPT all -- virbr0 * 192.168.122.0/24 0.0.0.0/0
Nest1 ping nest3 and made a hit here: 70 5832 REJECT all -- * virbr2 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable ----------------------------------------------------------
---------------------------------------------------------- Nest2 cannot ping nest1 and nest3.
Nest2 ping nest1 and made a hit here: 61 5100 REJECT all -- * virbr0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
Nest2 ping test3 and made a hit here: 70 5832 REJECT all -- * virbr2 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable ----------------------------------------------------------
From my observation, I see that the order of the virtual networks in the iptables FORWARD chain makes a difference. Each chunk associated with the virtual network in the chain, consisting of five lines, is exactly as described in the aforementioned link (http://libvirt.org/firewall.html). The virtual network in the top chunk of the chain can communicate with virtually all networks as opposed to the network in the last chunk that is consistent with the intention of the original design.
I'm using CentOS 7.3 with libvirt 2.0. I even tried reproducing this with CentOS 6.9 as I thought it was possible the firewalld may have influenced the change, but I was still getting similar result.
Right now, I'm not certain if this is already a reported known bug, but I'm highly convinced this configuration is unofficially unsupported for quite a while. Can someone confirm this? This is my very first mailing list submission ever, and I apologize in advance if I couldn't figure out how to conveniently search up a similar discussion as this one in the archive. If this is in fact near impossible to implement in accordance to the intended design, then I'd like to have this confirmation publicly included in the docs.
Thanks for reporting this. It's clearly a bug in libvirt. The rules are in this order: all rules for virbr0 all rules for virbr1 all rules for virbr2 But what we should do instead is: input rules for all networks local rules for all networks output rules for all networks reject rules for all networks The problem is that we do not know how other rules look like. So what we might need to do is create chains where rules for the first network are, then only append network rules into those chains. Would you mind filing a bug for this issue, so we can properly track it and don't forget about it? I'll have a look at it in the meantime, but don't promise anything since I'm not that familiar with that part of the codebase.
Thanks,
Travis Johnson
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander wrote:
On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
Hello,
I came across an interesting problem in my home lab a few weeks ago as I'm prepping for my RHCE exam using Michael Jang study guide. I've been at this for days now, and I still can't wrap my head around how two or more virtual networks in default NAT configuration are even allowed to communicate with each other despite what the libvirt documentation said.
Here's the excerpt I'm referring to in the wiki link here: http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
By default, guests that are connected via a virtual network with <forward mode='nat'/> can make any outgoing network connection they like. Incoming connections are allowed from the host, and from other guests connected to the same libvirt network, but all other incoming connections are blocked by iptables rules.
I did not read this properly...
Also here's another assertion from 'The virtual network driver' section in http://libvirt.org/firewall.html:
type=nat
Allow inbound related to an established connection. Allow outbound, but only from our expected subnet. Allow traffic between guests. Deny all other
I would expect the 'traffic between guests' to mean only guests on the same network. Also that lines up with what's written above.
inbound. Deny all other outbound.
[...]
Thanks for reporting this. It's clearly a bug in libvirt. The rules are in this order:
all rules for virbr0 all rules for virbr1 all rules for virbr2
But what we should do instead is:
input rules for all networks local rules for all networks output rules for all networks reject rules for all networks
The problem is that we do not know how other rules look like. So what we might need to do is create chains where rules for the first network are, then only append network rules into those chains.
Would you mind filing a bug for this issue, so we can properly track it and don't forget about it? I'll have a look at it in the meantime, but don't promise anything since I'm not that familiar with that part of the codebase.
Actually, now that I'm thinking about it, the problem is not that we disallow the communication from virbr1 to virbr0, but the problem is that we allow the connection from virbr0 to everywhere. Maybe the solution would be: for each network, insert the rules on top of the forward chain and for each started network, explicitly reject that one. So that after one network starts it would look like this (I'm writing this from memory just to illustrate the idea, not actually looking up how stuff looks): ACCEPT any virbr0 RELATED,ESTABLISHED ACCEPT virbr0 virbr0 ACCEPT virbr0 any src:192.168.122.0/24 REJECT any virbr0 REJECT virbr0 any And after second network is started, we'd have: # This one is new: REJECT virbr1 virbr0 # These ones would be normally at the end, IIRC: ACCEPT any virbr1 RELATED,ESTABLISHED ACCEPT virbr1 virbr1 ACCEPT virbr1 any src:192.168.122.0/24 REJECT any virbr1 REJECT virbr1 any # These are left as they were: ACCEPT any virbr0 RELATED,ESTABLISHED ACCEPT virbr0 virbr0 ACCEPT virbr0 any src:192.168.122.0/24 REJECT any virbr0 REJECT virbr0 any Cc'ing Laine so that he can weigh in as he has way more knowledge of this part of the code =)
Thanks,
Travis Johnson
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On 06/20/2017 05:27 AM, Martin Kletzander wrote:
On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander wrote:
On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
Hello,
I came across an interesting problem in my home lab a few weeks ago as I'm prepping for my RHCE exam using Michael Jang study guide. I've been at this for days now, and I still can't wrap my head around how two or more virtual networks in default NAT configuration are even allowed to communicate with each other despite what the libvirt documentation said.
Here's the excerpt I'm referring to in the wiki link here: http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections:
By default, guests that are connected via a virtual network with <forward mode='nat'/> can make any outgoing network connection they like. Incoming connections are allowed from the host, and from other guests connected to the same libvirt network, but all other incoming connections are blocked by iptables rules.
I did not read this properly...
Also here's another assertion from 'The virtual network driver' section in http://libvirt.org/firewall.html:
type=nat
Allow inbound related to an established connection. Allow outbound, but only from our expected subnet. Allow traffic between guests. Deny all other
I would expect the 'traffic between guests' to mean only guests on the same network. Also that lines up with what's written above.
That is correct.
inbound. Deny all other outbound.
[...]
Thanks for reporting this. It's clearly a bug in libvirt. The rules are in this order:
all rules for virbr0 all rules for virbr1 all rules for virbr2
But what we should do instead is:
input rules for all networks local rules for all networks output rules for all networks reject rules for all networks
That doesn't achieve the desired results (as you figured out later, according to your followup comments.
The problem is that we do not know how other rules look like. So what we might need to do is create chains where rules for the first network are, then only append network rules into those chains.
Would you mind filing a bug for this issue, so we can properly track it and don't forget about it? I'll have a look at it in the meantime, but don't promise anything since I'm not that familiar with that part of the codebase.
Actually, now that I'm thinking about it, the problem is not that we disallow the communication from virbr1 to virbr0, but the problem is that we allow the connection from virbr0 to everywhere.
Yep! That's actually a known bug that has been there "forever". It has always been brought up in the context of "I want communication between my NATed networks, and it only works in one direction!", rather than "I want to block communication between my NATed networks, but it gets past in one direction!". When people learn that it's (kind of) blocked, they work around it by adding a new network that is shared by all the guests. Because all of the reports have been in this direction, fixing it properly has been very low on (my) priority list.
Maybe the solution would be: for each network, insert the rules on top of the forward chain and for each started network, explicitly reject that one. So that after one network starts it would look like this (I'm writing this from memory just to illustrate the idea, not actually looking up how stuff looks):
ACCEPT any virbr0 RELATED,ESTABLISHED ACCEPT virbr0 virbr0 ACCEPT virbr0 any src:192.168.122.0/24 REJECT any virbr0 REJECT virbr0 any
And after second network is started, we'd have:
# This one is new: REJECT virbr1 virbr0
Yeah, it's (mostly) this rule that makes the solution complicated. Essentially you need one of these rules for each pair of virtual networks, so if you have 10 networks, you'll need 45 rules, for 11 you'll need 55 (in general, for "n" networks, you'll need (n * (n-1) / 2) extra rules). Aside from the extra packet processing overhead caused by the high number of rules needed (which may be completely insignificant), and the necessity to add/remove rules at specific labels rather than just inserting them all at the beginning, there is the issue that currently each iptables rule added for a libvirt network is dependent only on that network, so it's simple to determine which rules to remove when a network is destroyed. If we start adding rules that are dependent on two networks, then proper cleanup will be more complicated. None of that should prevent a properly motivated person from writing the code to do it, of course!
# These ones would be normally at the end, IIRC: ACCEPT any virbr1 RELATED,ESTABLISHED ACCEPT virbr1 virbr1 ACCEPT virbr1 any src:192.168.122.0/24 REJECT any virbr1 REJECT virbr1 any # These are left as they were: ACCEPT any virbr0 RELATED,ESTABLISHED ACCEPT virbr0 virbr0 ACCEPT virbr0 any src:192.168.122.0/24 REJECT any virbr0 REJECT virbr0 any
Cc'ing Laine so that he can weigh in as he has way more knowledge of this part of the code =)
Yep!
That's actually a known bug that has been there "forever". It has always been brought up in the context of "I want communication between my NATed networks, and it only works in one direction!", rather than "I want to block communication between my NATed networks, but it gets past in one direction!". When people learn that it's (kind of) blocked, they work around it by adding a new network that is shared by all the guests. Because all of the reports have been in this direction, fixing it properly has been very low on (my) priority list.
Thanks for confirming the bug as I suspected, Laine! I knew it was too good to be true that I'd be the first to discover this. Thanks for your huge help, Martin. I tried searching for this age-old you may be speaking of and I was only able to find this: https://bugzilla.redhat.com/show_bug.cgi?id=533193 Also found another somewhat-related bug affecting iptables (port-forwarding to VM): https://bugzilla.redhat.com/show_bug.cgi?id=1079088 If it's not too much trouble, can you provide me the ID number so I can track it? Aside from the extra packet processing overhead caused by the high
number of rules needed (which may be completely insignificant), and the necessity to add/remove rules at specific labels rather than just inserting them all at the beginning, there is the issue that currently each iptables rule added for a libvirt network is dependent only on that network, so it's simple to determine which rules to remove when a network is destroyed. If we start adding rules that are dependent on two networks, then proper cleanup will be more complicated.
None of that should prevent a properly motivated person from writing the code to do it, of course!
At least now I know this is quite feasible to fix. Thank you both for the well-elaborated responses. Cheers, -Travis On Tue, Jun 20, 2017 at 12:46 PM, Laine Stump <laine@laine.org> wrote:
On 06/20/2017 05:27 AM, Martin Kletzander wrote:
On Tue, Jun 20, 2017 at 10:05:19AM +0200, Martin Kletzander wrote:
On Tue, Jun 20, 2017 at 02:26:59AM -0400, Travis S. Johnson wrote:
Hello,
I came across an interesting problem in my home lab a few weeks ago as I'm prepping for my RHCE exam using Michael Jang study guide. I've been at this for days now, and I still can't wrap my head around how two or more virtual networks in default NAT configuration are even allowed to communicate with each other despite what the libvirt documentation said.
Here's the excerpt I'm referring to in the wiki link here: http://wiki.libvirt.org/page/Networking#Forwarding_ Incoming_Connections:
By default, guests that are connected via a virtual network with <forward mode='nat'/> can make any outgoing network connection they like. Incoming connections are allowed from the host, and from other guests connected to the same libvirt network, but all other incoming connections are blocked by iptables rules.
I did not read this properly...
Also here's another assertion from 'The virtual network driver' section in http://libvirt.org/firewall.html:
type=nat
Allow inbound related to an established connection. Allow outbound,
but
only from our expected subnet. Allow traffic between guests. Deny all other
I would expect the 'traffic between guests' to mean only guests on the same network. Also that lines up with what's written above.
That is correct.
inbound. Deny all other outbound.
[...]
Thanks for reporting this. It's clearly a bug in libvirt. The rules are in this order:
all rules for virbr0 all rules for virbr1 all rules for virbr2
But what we should do instead is:
input rules for all networks local rules for all networks output rules for all networks reject rules for all networks
That doesn't achieve the desired results (as you figured out later, according to your followup comments.
The problem is that we do not know how other rules look like. So what we might need to do is create chains where rules for the first network are, then only append network rules into those chains.
Would you mind filing a bug for this issue, so we can properly track it and don't forget about it? I'll have a look at it in the meantime, but don't promise anything since I'm not that familiar with that part of the codebase.
Actually, now that I'm thinking about it, the problem is not that we disallow the communication from virbr1 to virbr0, but the problem is that we allow the connection from virbr0 to everywhere.
Yep!
That's actually a known bug that has been there "forever". It has always been brought up in the context of "I want communication between my NATed networks, and it only works in one direction!", rather than "I want to block communication between my NATed networks, but it gets past in one direction!". When people learn that it's (kind of) blocked, they work around it by adding a new network that is shared by all the guests. Because all of the reports have been in this direction, fixing it properly has been very low on (my) priority list.
Maybe the solution would be: for each network, insert the rules on top of the forward chain and for each started network, explicitly reject that one. So that after one network starts it would look like this (I'm writing this from memory just to illustrate the idea, not actually looking up how stuff looks):
ACCEPT any virbr0 RELATED,ESTABLISHED ACCEPT virbr0 virbr0 ACCEPT virbr0 any src:192.168.122.0/24 REJECT any virbr0 REJECT virbr0 any
And after second network is started, we'd have:
# This one is new: REJECT virbr1 virbr0
Yeah, it's (mostly) this rule that makes the solution complicated. Essentially you need one of these rules for each pair of virtual networks, so if you have 10 networks, you'll need 45 rules, for 11 you'll need 55 (in general, for "n" networks, you'll need (n * (n-1) / 2) extra rules).
Aside from the extra packet processing overhead caused by the high number of rules needed (which may be completely insignificant), and the necessity to add/remove rules at specific labels rather than just inserting them all at the beginning, there is the issue that currently each iptables rule added for a libvirt network is dependent only on that network, so it's simple to determine which rules to remove when a network is destroyed. If we start adding rules that are dependent on two networks, then proper cleanup will be more complicated.
None of that should prevent a properly motivated person from writing the code to do it, of course!
# These ones would be normally at the end, IIRC: ACCEPT any virbr1 RELATED,ESTABLISHED ACCEPT virbr1 virbr1 ACCEPT virbr1 any src:192.168.122.0/24 REJECT any virbr1 REJECT virbr1 any # These are left as they were: ACCEPT any virbr0 RELATED,ESTABLISHED ACCEPT virbr0 virbr0 ACCEPT virbr0 any src:192.168.122.0/24 REJECT any virbr0 REJECT virbr0 any
Cc'ing Laine so that he can weigh in as he has way more knowledge of this part of the code =)
participants (3)
-
Laine Stump -
Martin Kletzander -
Travis S. Johnson