[libvirt] Don't add iptables rules when creating networks
Hi, I just found out that libvirt always add some iptables rules if it creates a natted (or routed) network. There were a couple of mailing list posts about this so I'm pretty sure this is not news to you. I don't want to go into the debate if your approach is sensible or not (I guess there are some use cases where I kind of like it). However on my server machine I really need full control over my (rather complicated) firewall settings. Currently the newly added rules really create a lot of problems for me. For example if I manage to have a good configuration after startup and then start a libvirt network afterwards, it will inject its rules at the start of the FORWARD queue (even though the same parameters are already present at the end!). On every net start there will be more duplicated rules and they will take preference over my existing rules. Besides that specific issue I think this is only one tiny problem compared to others (central configuration of firewall rules, auditing requirements, ...). Therefore I would like to have some kind 'power user' flag that prevents libvirt from adding any filter rules. I'm fine with activating it manually as long as I don't have to patch libvirt. fs
After some more hours of trying to fix libvirt's behavior, I decided to fix this issue by patching libvirt. Patch for 0.6.3 attached. If there's hope getting something like that into the git repository, I can port it to 0.7 easily. fs
On Sun, Dec 20, 2009 at 07:19:06PM +0100, Felix Schwarz wrote:
Hi,
I just found out that libvirt always add some iptables rules if it creates a natted (or routed) network. There were a couple of mailing list posts about this so I'm pretty sure this is not news to you.
I don't want to go into the debate if your approach is sensible or not (I guess there are some use cases where I kind of like it). However on my server machine I really need full control over my (rather complicated) firewall settings.
Currently the newly added rules really create a lot of problems for me. For example if I manage to have a good configuration after startup and then start a libvirt network afterwards, it will inject its rules at the start of the FORWARD queue (even though the same parameters are already present at the end!). On every net start there will be more duplicated rules and they will take preference over my existing rules.
There shold never be duplicated rules. If you stop a libvirt virutal network, it will remove its previously added rules, so there should be no duplication next time it is started. If removal isn't working, that's a bug to be fixed. Can you outline how your desired configuration for libvirt NAT mode is different from what libvirt already does ? The goal for this is to be totally zero-conf, so that fact that you can't use the default setup shows something is lacking in our impl & I'd prefer to identify what that is rather than blindly disabling it. In addition the libvirt rules are written to try & ensure that they only impact traffic to/from the subnet that is configured in the libvirt network, to avoid causing problems for other rules you might have already configured.
Besides that specific issue I think this is only one tiny problem compared to others (central configuration of firewall rules, auditing requirements, ...).
Therefore I would like to have some kind 'power user' flag that prevents libvirt from adding any filter rules. I'm fine with activating it manually as long as I don't have to patch libvirt.
This isn't really something we want to support. As I mention above we want to make sure this works out of the box without manual config. The one change we do want to make to the setup, is to move all the rules into dedicated chains (libvirt_INPUT, libvirt_FORWARD, etc) so that we only add a single rule to the main INPUT/FORWARD chains. Regards, Daniel. -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Am 21.12.2009 13:04, schrieb Daniel P. Berrange:
There shold never be duplicated rules. If you stop a libvirt virutal network, it will remove its previously added rules, so there should be no duplication next time it is started. If removal isn't working, that's a bug to be fixed.
I had two different networks, one with nat, one routed. Only one is started with autostart. As soon as I start the other, I get additional (duplicated I think) rules.
Can you outline how your desired configuration for libvirt NAT mode is different from what libvirt already does ? The goal for this is to be totally zero-conf, so that fact that you can't use the default setup shows something is lacking in our impl& I'd prefer to identify what that is rather than blindly disabling it.
Actually my main interest is the routed mode, not NAT. This is my iptables after I started two networks (no other packet filter): # iptables --list Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.78.21 ACCEPT all -- 192.168.78.21 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable (...) My issues: 1) INPUT chain ACCEPTs DNS/dhcp from outside You might notice that the INPUT chain basically says that I ACCEPT all DNS/dhcp from all interfaces. I don't want that. As soon as I configure a packet filter (e.g. shorewall), libvirt's configuration will take precedence. 2) FORWARD contains general rules ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable These rules apply to all FORWARDed connections. I need *way* more control. 3) FORWARD ACCEPTs packets from all hosts ACCEPT all -- anywhere 192.168.78.21 ACCEPT all -- 192.168.78.21 anywhere Say I have routed libvirt network but I want to protect these hosts - only some specific hosts may reach them (e.g. a virtualized backend app server is only reachable by the frontend servers). With the generated iptables rules I can not do that. 4) No way to override rules All new iptables rules are pre-prepended when a new network is started (which may happen at any time), potentially circumventing all existing rules. 5) Company policies How do you keep firewall rules manageable/auditable in 'not extremly simple' situations? Many companies I know have a very strict policy that only one application is allowed to define rules (e.g. shorewall or a proprietary FW). I mean you @Red Hat should know stuff like that. If libvirt touches my carefully reviewed policies, it might open a lot of security issues. That being said I appreciate your approach to make it easy for simple cases and desktop end users. In fact, I'm using libvirt since Fedora 10 on a desktop with problems. Now with RHEL 5.4 I'm starting to use that on servers and here I need way more control. I guess there are a lot more use cases when you just need to disable automated iptables changes - just because libvirt does not have the whole picture.
Therefore I would like to have some kind 'power user' flag that prevents libvirt from adding any filter rules. I'm fine with activating it manually as long as I don't have to patch libvirt.
This isn't really something we want to support. As I mention above we want to make sure this works out of the box without manual config.
I can totally understand you - but how do you think you can deal with system security if libvirt just does not have all information? How can I use a libvirt host as a router, only giving specific IPs accesss to a routed network?
The one change we do want to make to the setup, is to move all the rules into dedicated chains (libvirt_INPUT, libvirt_FORWARD, etc) so that we only add a single rule to the main INPUT/FORWARD chains.
I'm afraid that this won't help in my situation: Still all the rules are prepended and I can not specify which rules should be inserted. fs
On Mon, Dec 21, 2009 at 03:42:17PM +0100, Felix Schwarz wrote:
Am 21.12.2009 13:04, schrieb Daniel P. Berrange:
There shold never be duplicated rules. If you stop a libvirt virutal network, it will remove its previously added rules, so there should be no duplication next time it is started. If removal isn't working, that's a bug to be fixed.
I had two different networks, one with nat, one routed. Only one is started with autostart. As soon as I start the other, I get additional (duplicated I think) rules.
Can you outline how your desired configuration for libvirt NAT mode is different from what libvirt already does ? The goal for this is to be totally zero-conf, so that fact that you can't use the default setup shows something is lacking in our impl& I'd prefer to identify what that is rather than blindly disabling it.
Actually my main interest is the routed mode, not NAT.
This is my iptables after I started two networks (no other packet filter):
# iptables --list Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps ACCEPT udp -- anywhere anywhere udp dpt:domain ACCEPT tcp -- anywhere anywhere tcp dpt:domain ACCEPT udp -- anywhere anywhere udp dpt:bootps ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere 192.168.78.21 ACCEPT all -- 192.168.78.21 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable ACCEPT all -- anywhere 192.168.122.0/24 state RELATED,ESTABLISHED ACCEPT all -- 192.168.122.0/24 anywhere ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
(...)
My issues: 1) INPUT chain ACCEPTs DNS/dhcp from outside
You might notice that the INPUT chain basically says that I ACCEPT all DNS/dhcp from all interfaces. I don't want that. As soon as I configure a packet filter (e.g. shorewall), libvirt's configuration will take precedence.
No it doesn't say that. You are missing the '-v' flag to list the rules. If you add that you'll see that the rules are *different* and they all explicitly include the name of the bridge interface associated with the libvirt network
2) FORWARD contains general rules ACCEPT all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
These rules apply to all FORWARDed connections. I need *way* more control.
Again, use the -v flag to iptables to see the full set of constraints. We explicitly try to make libvirt rules *only* apply to the libvirt created interfaces and not the rest of your interfaces/configuration.
3) FORWARD ACCEPTs packets from all hosts ACCEPT all -- anywhere 192.168.78.21 ACCEPT all -- 192.168.78.21 anywhere
Say I have routed libvirt network but I want to protect these hosts - only some specific hosts may reach them (e.g. a virtualized backend app server is only reachable by the frontend servers). With the generated iptables rules I can not do that.
4) No way to override rules All new iptables rules are pre-prepended when a new network is started (which may happen at any time), potentially circumventing all existing rules.
That is intentionale, and since the libvirt rules only match on the libvirt network interface name & ip range, it should not be opening any holes in your current config for other interfaces.
5) Company policies How do you keep firewall rules manageable/auditable in 'not extremly simple' situations? Many companies I know have a very strict policy that only one application is allowed to define rules (e.g. shorewall or a proprietary FW). I mean you @Red Hat should know stuff like that. If libvirt touches my carefully reviewed policies, it might open a lot of security issues.
I agree that corporate policy/compliance issues are probably the main stumbling block here. In some cases, improving the documentation about exactly what libvirt does with iptables may help. eg attempt to show that libvirt's rules won't open any holes in the firewall beyond what is explicitly intended This obviously won't be enough for everyone's policy/compliance needs though. In such strict managed deployments, I thing the libvirt virtua network functionality is simply not going to be possible to use. Once you've taken away the iptables setup, they there ceases to be much point in using this functionality as it is. There are other libvirt APIs that would suit better, such as the network interface management APIs we recently added.
That being said I appreciate your approach to make it easy for simple cases and desktop end users. In fact, I'm using libvirt since Fedora 10 on a desktop with problems. Now with RHEL 5.4 I'm starting to use that on servers and here I need way more control.
I guess there are a lot more use cases when you just need to disable automated iptables changes - just because libvirt does not have the whole picture.
We'd really like to make sure libvirt can get the bigger picture if at all possible. It is a core goal with libvirt functionality, that it be possible to use it all from a remote client without needing to login into the virt host as an a shell admin.
Therefore I would like to have some kind 'power user' flag that prevents libvirt from adding any filter rules. I'm fine with activating it manually as long as I don't have to patch libvirt.
This isn't really something we want to support. As I mention above we want to make sure this works out of the box without manual config.
I can totally understand you - but how do you think you can deal with system security if libvirt just does not have all information? How can I use a libvirt host as a router, only giving specific IPs accesss to a routed network?
Can you explain a little more about your routed setup ? In particular, are you trying to use the same IP address range for VMs and your LAN, and thus just route a handful of IPs ? Or are you having a separate subnet for the VMs ? I know libvirt won't cope with the former scenario currently, since as you say it would need to know which IPs to route. We can deal with the separate-subnet scenario though & that shouldn't require any per-IP setup on the virt host I wrote a blog post about the routed network scenario that we currently support & expect to work http://berrange.com/personal/diary/2009/12/routed-subnets-without-nat-for-li... As per the last paragraph, I'd like us to add extra support for Proxy-ARP (optionally with subnetting) to our routed config
The one change we do want to make to the setup, is to move all the rules into dedicated chains (libvirt_INPUT, libvirt_FORWARD, etc) so that we only add a single rule to the main INPUT/FORWARD chains.
I'm afraid that this won't help in my situation: Still all the rules are prepended and I can not specify which rules should be inserted.
Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Am 21.12.2009 16:00, schrieb Daniel P. Berrange:
My issues: 1) INPUT chain ACCEPTs DNS/dhcp from outside
You might notice that the INPUT chain basically says that I ACCEPT all DNS/dhcp from all interfaces. I don't want that. As soon as I configure a packet filter (e.g. shorewall), libvirt's configuration will take precedence.
No it doesn't say that. You are missing the '-v' flag to list the rules. If you add that you'll see that the rules are *different* and they all explicitly include the name of the bridge interface associated with the libvirt network
You're right - actually I did not check closely enough. Sorry for that.
I agree that corporate policy/compliance issues are probably the main stumbling block here. (...)
This obviously won't be enough for everyone's policy/compliance needs though. In such strict managed deployments, I thing the libvirt virtua network functionality is simply not going to be possible to use. Once you've taken away the iptables setup, they there ceases to be much point in using this functionality as it is. There are other libvirt APIs that would suit better, such as the network interface management APIs we recently added.
Which APIs do you think of? To me it looked like libvirt should become the default configuration layer whenever you do something with virtual machines (as it is configured by default, most configuration tools use it, ...). Therefore I tried to make my setup work with libvirt to make use of all that integration stuff...
Can you explain a little more about your routed setup ? In particular, are you trying to use the same IP address range for VMs and your LAN, and thus just route a handful of IPs ?
Basically yes: This is a server in a data center with a couple of IPs that are assigned by my provider (no subnet). So I assign one IP to my host and route the others to libvirt interfaces so that my VMs can provide public services as well. I need a routed setup due to MAC address filtering in the switches.
I know libvirt won't cope with the former scenario currently, since as you say it would need to know which IPs to route. We can deal with the separate-subnet scenario though& that shouldn't require any per-IP setup on the virt host
Actually there are not that many ipv4 addresses left so there are only 4 IPs included in my plan (used to be 1 + subnet with 6 usable IPs). Therefore I get only single IP addresses. fs
Hello,
Can you outline how your desired configuration for libvirt NAT mode is different from what libvirt already does ? The goal for this is to be totally zero-conf, so that fact that you can't use the default setup shows something is lacking in our impl & I'd prefer to identify what that is rather than blindly disabling it. In addition the libvirt rules are written to try & ensure that they only impact traffic to/from the subnet that is configured in the libvirt network, to avoid causing problems for other rules you might have already configured.
I opened a bug report[1] for this too, doing the right thing for out-of-the-box configuration is ok, but everything should be opt-out and manually configurable. I add sanity-check rules at top of my netfilter chains and when a libvirt network start it's not "protected" by theses rules. It's like my bug report on dnsmasq[2], I already have a complete DHCP/DNS-with-LDAP-backend configuration for the subnet, I don't need it but can not opt-out the feature. This disempower the user/administrator, which I think is bad. So, what I whould like to see: 1. Automatic configuration for out-of-the-box setup 2. Opt-out all the automatic configurations 3. Manually configurable, with pre-up(before), up(doing it), post-up(after) and their down counterparts. Please. Footnotes: [1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=568790 [2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=549183 -- Daniel Dehennin Récupérer ma clef GPG: gpg --keyserver pgp.mit.edu --recv-keys 0x6A2540D1
participants (3)
-
Daniel Dehennin -
Daniel P. Berrange -
Felix Schwarz