[libvirt] iptables and libvirt
Hi, I would like your feedback on the following idea. What if we could flexibly change the iptables rules for the different guests as they are deployed onto the node/host. The idea would be to do all of this within the iptables of the host leaving alone the iptables of the guests themselves. Here are some specifics: - Physical systems typically isolated using firewalls protecting well known ports. - With virt, on shared physical device, use a bridge to give full LAN access to vm - Or a virtual network which is an isolated bridge with no physical connection. Guest can talk to each other directly. Only NAT'd outbound. - The idea is to eventually make it easy to centrally set up iptable rules for guests that are applied in the host iptables. - We would have to be able to migrate the iptables rules and the state data with vm as it moves The benefits of this would be we could: - Create networking controls that provide same isolation as physical systems - Control which VMs can talk to which others Integration option: - Integration in virtd because it knows about the guests and their network parameters. Thanks for your feedback. Best regards, Karl
On Fri, 2009-02-06 at 13:36 -0500, Karl Wirth wrote:
What if we could flexibly change the iptables rules for the different guests as they are deployed onto the node/host. The idea would be to do all of this within the iptables of the host leaving alone the iptables of the guests themselves.
The first issue with this is that the host does not know the IP addresses in use by the guests; it might be possible to work around that with setting up rules matching on bridge ports in some cases. Secondly, network devices may be directly assigned to guests - in that case, we won't even see any of the packets the guest sends or receives. I also don't see how you can implement that in the general case, given what a management nightmare iptables is. The trouble is that in a general libvirt installation, we could have arbitrary iptables rules in effect that are not controlled by libvirt. To reliably say, for example, that we reliably block all ports for VM x, we'd either need to understand all the existing iptables rules, or insert our rules first in the appropriate chains and be confident that they will never conflict with any other manually set up rules. It would be nice to do this, to offer an additional layer of security, especially around insecure OS's; to pull that off in practice, you'd need to assume fairly tight control of the host (e.g., only use shared network interfaces, only deal with iptables rules set up by a known application) With that, iptables management belongs into a higher-level management app, like ovirt, not libvirt. David
On Sat, Feb 07, 2009 at 12:44:07AM +0000, David Lutterkort wrote:
On Fri, 2009-02-06 at 13:36 -0500, Karl Wirth wrote:
What if we could flexibly change the iptables rules for the different guests as they are deployed onto the node/host. The idea would be to do all of this within the iptables of the host leaving alone the iptables of the guests themselves.
The first issue with this is that the host does not know the IP addresses in use by the guests; it might be possible to work around that with setting up rules matching on bridge ports in some cases.
Actually I believe Karl's use case is that the host explicitly *does* know the IP the guest is /supposed/ to be using, and wants to prevent it spoofing someone else's IP.
Secondly, network devices may be directly assigned to guests - in that case, we won't even see any of the packets the guest sends or receives.
I think that's a tradeoff you decide to make when assigning the NIC directly, so we can just declare it out of scope.
I also don't see how you can implement that in the general case, given what a management nightmare iptables is. The trouble is that in a general libvirt installation, we could have arbitrary iptables rules in effect that are not controlled by libvirt. To reliably say, for example, that we reliably block all ports for VM x, we'd either need to understand all the existing iptables rules, or insert our rules first in the appropriate chains and be confident that they will never conflict with any other manually set up rules.
It is tricky, but I don't think it is any worse than the situation we are already in WRT to iptables usage by the virtual networking stuff. We're basically talking about adding deny rules against the TAP device associated with the guest. The complexity comes because a user might add further rules which then allow stuff we denied, or apps / commands may flush rules we've added. I think it is fine to just document the limitations / requirements in this case.
It would be nice to do this, to offer an additional layer of security, especially around insecure OS's; to pull that off in practice, you'd need to assume fairly tight control of the host (e.g., only use shared network interfaces, only deal with iptables rules set up by a known application)
With that, iptables management belongs into a higher-level management app, like ovirt, not libvirt.
NB, we do already have indirect iptables anti-spoofing support for the Xen driver in libvirt. If you specify a <ip address> tag for a Xen bridged network driver, the vif-bridge script will try to add rules to prevent the vnet device using a different IP address. I agree with your general point though, that when trying this in a general purpose OS deployment I don't think you can provide sufficient guarentees from a libvirt POV. There are simply too many other things that may break or otherwise badly interact with the iptables rules we're adding. At the very simplest level, 'service iptables restart' messes things up. In the context of a controlled host image, like the oVirt managed node, the mgmt app is in control of the host OS, and in such a scenario it may be practical for libvirt to add iptables rules for guests. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
Actually I believe Karl's use case is that the host explicitly *does* know the IP the guest is /supposed/ to be using, and wants to prevent it spoofing someone else's IP.
Yes. This is what I was thinking.
I agree with your general point though, that when trying this in a general purpose OS deployment I don't think you can provide sufficient guarentees from a libvirt POV. There are simply too many other things that may break or otherwise badly interact with the iptables rules we're adding. At the very simplest level, 'service iptables restart' messes things up.
In the context of a controlled host image, like the oVirt managed node, the mgmt app is in control of the host OS, and in such a scenario it may be practical for libvirt to add iptables rules for guests.
I was thinking of a fully managed node. Thanks for this feedback.
On Fri, Feb 06, 2009 at 01:36:23PM -0500, Karl Wirth wrote:
Hi,
I would like your feedback on the following idea.
What if we could flexibly change the iptables rules for the different guests as they are deployed onto the node/host. The idea would be to do all of this within the iptables of the host leaving alone the iptables of the guests themselves.
Here are some specifics: - Physical systems typically isolated using firewalls protecting well known ports. - With virt, on shared physical device, use a bridge to give full LAN access to vm - Or a virtual network which is an isolated bridge with no physical connection. Guest can talk to each other directly. Only NAT'd outbound. - The idea is to eventually make it easy to centrally set up iptable rules for guests that are applied in the host iptables. - We would have to be able to migrate the iptables rules and the state data with vm as it moves
These bullet points don't really state any clear goal / requirement. My first assumption is that you're looking for a way to stop a guest using another guests IP address. So called 'ip address anti-spoofing' in Xen terminology. You'd also need to prevent a guest spoofing another guest's MAC address for this to be worthwhile. Which comes down to a matter of adding iptables, ip6tables and ebtables rules against the TAP device i guess. Controlling guest <-> guest traffic as you mention below, becomes alot more complex problem because you're considering interactions between guests' TAP devices, and not just adding rules to control stuff coming in & out of a single TAP device.
The benefits of this would be we could: - Create networking controls that provide same isolation as physical systems - Control which VMs can talk to which others
This has rather alot of overlap with the stated goals of the sVirt project, though I don't think that explicitly addresses networking, mostly disk / host OS resources. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Karl Wirth wrote:
Hi,
I would like your feedback on the following idea.
What if we could flexibly change the iptables rules for the different guests as they are deployed onto the node/host. The idea would be to do all of this within the iptables of the host leaving alone the iptables of the guests themselves.
At first one thing: The firewall setup for EL-5 and EL-6 is using the same mechanism with accept rules first and reject rules afterwards. This means that adding an accept rule before the reject rule could open up the firewall.
Here are some specifics: - Physical systems typically isolated using firewalls protecting well known ports. - With virt, on shared physical device, use a bridge to give full LAN access to vm - Or a virtual network which is an isolated bridge with no physical connection. Guest can talk to each other directly. Only NAT'd outbound. - The idea is to eventually make it easy to centrally set up iptable rules for guests that are applied in the host iptables. - We would have to be able to migrate the iptables rules and the state data with vm as it moves
Migration od the state will be a problem for EL-5 and IPv6, because stateful firewalling in EL-5 is only possible with IPv4. This is due to using different netfilter interfaces for IPv4 and IPv6.
The benefits of this would be we could: - Create networking controls that provide same isolation as physical systems - Control which VMs can talk to which others
Integration option: - Integration in virtd because it knows about the guests and their network parameters.
Some Questions: 1) Should it be a static system with predefined rules or a fully dynamic system? 2) Will there be a configuration utility for the rules? 3) What do you want to do with user-customized firewalls? Thanks, Thomas
On Friday 13 February 2009 19:21:10 Thomas Woerner wrote: Hi, i just got the same idea, so here are some thoughts.
Some Questions: 3) What do you want to do with user-customized firewalls?
I want do to port forwarding for a nat network for statically configured ip addressed. A solution could be: <network> <name>sample</name> <forward dev='eth0' mode='nat'/> <bridge name='intbr0' stp='on' forwardDelay='0' /> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254' /> <host mac='AA:BB:CC:DD:EE:FF' ip='192.168.122.102'> <port proto='tcp' destip='EXT_IP' dport='80' /> <port proto='tcp' destip='EXT_IP' dport='2202' mapped_port='22'> </host> </dhcp> </ip> </network> The first mapping could be translatet in something like iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 80 \ -m state NEW -j ACCEPT iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 80 -j DNAT \ --to-destination 192.168.122.102 second one would be iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 22 \ -m state NEW -j ACCEPT iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 2202 -j DNAT \ --to-destination 192.168.122.102:22 Hope this makes it a little bit clearer what i want to do with user customized firewalls. Uli
Ulrich Dangel wrote:
On Friday 13 February 2009 19:21:10 Thomas Woerner wrote: Hi,
i just got the same idea, so here are some thoughts.
Some Questions: 3) What do you want to do with user-customized firewalls?
I want do to port forwarding for a nat network for statically configured ip addressed.
A solution could be: <network> <name>sample</name> <forward dev='eth0' mode='nat'/> <bridge name='intbr0' stp='on' forwardDelay='0' /> <ip address='192.168.122.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.122.2' end='192.168.122.254' /> <host mac='AA:BB:CC:DD:EE:FF' ip='192.168.122.102'> <port proto='tcp' destip='EXT_IP' dport='80' /> <port proto='tcp' destip='EXT_IP' dport='2202' mapped_port='22'> </host> </dhcp> </ip> </network>
The first mapping could be translatet in something like iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 80 \ -m state NEW -j ACCEPT iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 80 -j DNAT \ --to-destination 192.168.122.102
second one would be
iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 22 \ -m state NEW -j ACCEPT iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 2202 -j DNAT \ --to-destination 192.168.122.102:22
Hope this makes it a little bit clearer what i want to do with user customized firewalls.
What is EXT_IP here? These forward rules are exactly what "lokkit --forward-port=if=<interface>:port=<port>:proto=<protocol>[:toport=<destination port>][:toaddr=<destination address>]" already is doing. The configuration is written to /etc/sysconfig/system-config-firewall and /etc/sysconfig/iptables.
Uli
Thomas
One way to do this is to place a tiny VM (static kernel+very small initramfs [uClibc+busybox+iptables+dnsmasq]) between VM clusters and the host, rather than giving the host an IP on each cluster's bridge directly. The tool that launches it (via libvirt) appends extra files to the initramfs giving iptables rules to be run. I use this "virtual router" to NETMAP multiple clouds of VMs which all think they're using the same network space (say, 192.168.0.0/24) onto different subnets (say, 192.168.1.x and 192.168.2.x for the first two clusters), but also have added support for redirecting connections intended for specific targets to elsewhere, overriding DNS results for specific hosts, and other miscellaneous utility functions. Using a separate VM rather than iptables rules on the host was necessary in my use case because doing symmetrical NETMAP properly requires packets from the host to the clients to pass through the PREROUTING table -- which packets generated within a given host don't do. If anyone (libvirt/oVirt/whomever) is interested in incorporating this into their project (even as an entry in a contrib repository), let me know; I can't distribute binaries without going through some pain and suffering (setting up a SKU with my employer to ship a CD with sources to the kernel and the GPLed components of the initrd), but providing it in source form as a minor patch to someone else's project (and this *is* actually implemented in very little code -- a mixture of Python and busybox-friendly shell scripts totaling under 500 lines, so the "minor" label applies) should be clear sailing.
participants (6)
-
Charles Duffy -
Daniel P. Berrange -
David Lutterkort -
Karl Wirth -
Thomas Woerner -
Ulrich Dangel