12:36 p.m.
Hi,
I would like your feedback on the following idea.
What if we could flexibly change the iptables rules for the different
guests as they are deployed onto the node/host. The idea would be to do
all of this within the iptables of the host leaving alone the iptables
of the guests themselves.
Here are some specifics:
- Physical systems typically isolated using firewalls protecting well
known ports.
- With virt, on shared physical device, use a bridge to give full LAN
access to vm
- Or a virtual network which is an isolated bridge with no physical
connection. Guest can talk to each other directly. Only NAT'd outbound.
- The idea is to eventually make it easy to centrally set up iptable
rules for guests that are applied in the host iptables.
- We would have to be able to migrate the iptables rules and the state
data with vm as it moves
The benefits of this would be we could:
- Create networking controls that provide same isolation as physical systems
- Control which VMs can talk to which others
Integration option:
- Integration in virtd because it knows about the guests and their
network parameters.
Thanks for your feedback.
Best regards,
Karl
6:44 p.m.
On Fri, 2009-02-06 at 13:36 -0500, Karl Wirth wrote:
What if we could flexibly change the iptables rules for the
different
guests as they are deployed onto the node/host. The idea would be to do
all of this within the iptables of the host leaving alone the iptables
of the guests themselves.
The first issue with this is that the host does not know the IP
addresses in use by the guests; it might be possible to work around that
with setting up rules matching on bridge ports in some cases.
Secondly, network devices may be directly assigned to guests - in that
case, we won't even see any of the packets the guest sends or receives.
I also don't see how you can implement that in the general case, given
what a management nightmare iptables is. The trouble is that in a
general libvirt installation, we could have arbitrary iptables rules in
effect that are not controlled by libvirt. To reliably say, for example,
that we reliably block all ports for VM x, we'd either need to
understand all the existing iptables rules, or insert our rules first in
the appropriate chains and be confident that they will never conflict
with any other manually set up rules.
It would be nice to do this, to offer an additional layer of security,
especially around insecure OS's; to pull that off in practice, you'd
need to assume fairly tight control of the host (e.g., only use shared
network interfaces, only deal with iptables rules set up by a known
application)
With that, iptables management belongs into a higher-level management
app, like ovirt, not libvirt.
David
5:24 a.m.
On Sat, Feb 07, 2009 at 12:44:07AM +0000, David Lutterkort wrote:
On Fri, 2009-02-06 at 13:36 -0500, Karl Wirth wrote:
> What if we could flexibly change the iptables rules for the different
> guests as they are deployed onto the node/host. The idea would be to do
> all of this within the iptables of the host leaving alone the iptables
> of the guests themselves.
The first issue with this is that the host does not know the IP
addresses in use by the guests; it might be possible to work around that
with setting up rules matching on bridge ports in some cases.
Actually I believe Karl's use case is that the host explicitly *does*
know the IP the guest is /supposed/ to be using, and wants to prevent
it spoofing someone else's IP.
Secondly, network devices may be directly assigned to guests - in
that
case, we won't even see any of the packets the guest sends or receives.
I think that's a tradeoff you decide to make when assigning the NIC
directly, so we can just declare it out of scope.
I also don't see how you can implement that in the general case,
given
what a management nightmare iptables is. The trouble is that in a
general libvirt installation, we could have arbitrary iptables rules in
effect that are not controlled by libvirt. To reliably say, for example,
that we reliably block all ports for VM x, we'd either need to
understand all the existing iptables rules, or insert our rules first in
the appropriate chains and be confident that they will never conflict
with any other manually set up rules.
It is tricky, but I don't think it is any worse than the situation
we are already in WRT to iptables usage by the virtual networking
stuff. We're basically talking about adding deny rules against the
TAP device associated with the guest. The complexity comes because
a user might add further rules which then allow stuff we denied,
or apps / commands may flush rules we've added. I think it is fine
to just document the limitations / requirements in this case.
It would be nice to do this, to offer an additional layer of
security,
especially around insecure OS's; to pull that off in practice, you'd
need to assume fairly tight control of the host (e.g., only use shared
network interfaces, only deal with iptables rules set up by a known
application)
With that, iptables management belongs into a higher-level management
app, like ovirt, not libvirt.
NB, we do already have indirect iptables anti-spoofing support for
the Xen driver in libvirt. If you specify a <ip address> tag for a
Xen bridged network driver, the vif-bridge script will try to add
rules to prevent the vnet device using a different IP address.
I agree with your general point though, that when trying this in a general
purpose OS deployment I don't think you can provide sufficient guarentees
from a libvirt POV. There are simply too many other things that may break
or otherwise badly interact with the iptables rules we're adding. At the
very simplest level, 'service iptables restart' messes things up.
In the context of a controlled host image, like the oVirt managed node,
the mgmt app is in control of the host OS, and in such a scenario it
may be practical for libvirt to add iptables rules for guests.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
9:12 a.m.
Daniel P. Berrange wrote:
Actually I believe Karl's use case is that the host explicitly
*does*
know the IP the guest is /supposed/ to be using, and wants to prevent
it spoofing someone else's IP.
Yes. This is what I was thinking.
I agree with your general point though, that when trying this in a
general
purpose OS deployment I don't think you can provide sufficient guarentees
from a libvirt POV. There are simply too many other things that may break
or otherwise badly interact with the iptables rules we're adding. At the
very simplest level, 'service iptables restart' messes things up.
In the context of a controlled host image, like the oVirt managed node,
the mgmt app is in control of the host OS, and in such a scenario it
may be practical for libvirt to add iptables rules for guests.
I was thinking of a fully managed node.
Thanks for this feedback.
5:31 a.m.
On Fri, Feb 06, 2009 at 01:36:23PM -0500, Karl Wirth wrote:
Hi,
I would like your feedback on the following idea.
What if we could flexibly change the iptables rules for the different
guests as they are deployed onto the node/host. The idea would be to do
all of this within the iptables of the host leaving alone the iptables
of the guests themselves.
Here are some specifics:
- Physical systems typically isolated using firewalls protecting well
known ports.
- With virt, on shared physical device, use a bridge to give full LAN
access to vm
- Or a virtual network which is an isolated bridge with no physical
connection. Guest can talk to each other directly. Only NAT'd outbound.
- The idea is to eventually make it easy to centrally set up iptable
rules for guests that are applied in the host iptables.
- We would have to be able to migrate the iptables rules and the state
data with vm as it moves
These bullet points don't really state any clear goal / requirement.
My first assumption is that you're looking for a way to stop a guest
using another guests IP address. So called 'ip address anti-spoofing'
in Xen terminology. You'd also need to prevent a guest spoofing another
guest's MAC address for this to be worthwhile. Which comes down to
a matter of adding iptables, ip6tables and ebtables rules against
the TAP device i guess.
Controlling guest <-> guest traffic as you mention below, becomes alot
more complex problem because you're considering interactions between
guests' TAP devices, and not just adding rules to control stuff coming
in & out of a single TAP device.
The benefits of this would be we could:
- Create networking controls that provide same isolation as physical systems
- Control which VMs can talk to which others
This has rather alot of overlap with the stated goals of the sVirt
project, though I don't think that explicitly addresses networking,
mostly disk / host OS resources.
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
12:21 p.m.
Karl Wirth wrote:
Hi,
I would like your feedback on the following idea.
What if we could flexibly change the iptables rules for the different
guests as they are deployed onto the node/host. The idea would be to do
all of this within the iptables of the host leaving alone the iptables
of the guests themselves.
At first one thing: The firewall setup for EL-5 and EL-6 is using the
same mechanism with accept rules first and reject rules afterwards.
This means that adding an accept rule before the reject rule could open
up the firewall.
Here are some specifics:
- Physical systems typically isolated using firewalls protecting well
known ports.
- With virt, on shared physical device, use a bridge to give full LAN
access to vm
- Or a virtual network which is an isolated bridge with no physical
connection. Guest can talk to each other directly. Only NAT'd outbound.
- The idea is to eventually make it easy to centrally set up iptable
rules for guests that are applied in the host iptables.
- We would have to be able to migrate the iptables rules and the state
data with vm as it moves
Migration od the state will be a problem for EL-5 and IPv6, because
stateful firewalling in EL-5 is only possible with IPv4. This is due to
using different netfilter interfaces for IPv4 and IPv6.
The benefits of this would be we could:
- Create networking controls that provide same isolation as physical systems
- Control which VMs can talk to which others
Integration option:
- Integration in virtd because it knows about the guests and their
network parameters.
Some Questions:
1) Should it be a static system with predefined rules or a fully dynamic
system?
2) Will there be a configuration utility for the rules?
3) What do you want to do with user-customized firewalls?
Thanks,
Thomas
9:47 a.m.
On Friday 13 February 2009 19:21:10 Thomas Woerner wrote:
Hi,
i just got the same idea, so here are some thoughts.
Some Questions:
3) What do you want to do with user-customized firewalls?
I want do to port forwarding for a nat network for statically configured ip
addressed.
A solution could be:
<network>
<name>sample</name>
<forward dev='eth0' mode='nat'/>
<bridge name='intbr0' stp='on' forwardDelay='0' />
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254' />
<host mac='AA:BB:CC:DD:EE:FF' ip='192.168.122.102'>
<port proto='tcp' destip='EXT_IP' dport='80' />
<port proto='tcp' destip='EXT_IP' dport='2202'
mapped_port='22'>
</host>
</dhcp>
</ip>
</network>
The first mapping could be translatet in something like
iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 80 \
-m state NEW -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 80 -j DNAT \
--to-destination 192.168.122.102
second one would be
iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 22 \
-m state NEW -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 2202 -j DNAT \
--to-destination 192.168.122.102:22
Hope this makes it a little bit clearer what i want to do with user customized
firewalls.
Uli
7:43 a.m.
Ulrich Dangel wrote:
On Friday 13 February 2009 19:21:10 Thomas Woerner wrote:
Hi,
i just got the same idea, so here are some thoughts.
> Some Questions:
> 3) What do you want to do with user-customized firewalls?
I want do to port forwarding for a nat network for statically configured ip
addressed.
A solution could be:
<network>
<name>sample</name>
<forward dev='eth0' mode='nat'/>
<bridge name='intbr0' stp='on' forwardDelay='0' />
<ip address='192.168.122.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.122.2' end='192.168.122.254' />
<host mac='AA:BB:CC:DD:EE:FF' ip='192.168.122.102'>
<port proto='tcp' destip='EXT_IP' dport='80' />
<port proto='tcp' destip='EXT_IP' dport='2202'
mapped_port='22'>
</host>
</dhcp>
</ip>
</network>
The first mapping could be translatet in something like
iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 80 \
-m state NEW -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 80 -j DNAT \
--to-destination 192.168.122.102
second one would be
iptables -A FORWARD -i intbr0 -p tcp -d 192.168.122.102 --dport 22 \
-m state NEW -j ACCEPT
iptables -t nat -A PREROUTING -p tcp -s EXT_IP --dport 2202 -j DNAT \
--to-destination 192.168.122.102:22
Hope this makes it a little bit clearer what i want to do with user customized
firewalls.
What is EXT_IP here?
These forward rules are exactly what "lokkit
--forward-port=if=<interface>:port=<port>:proto=<protocol>[:toport=<destination
port>][:toaddr=<destination address>]" already is doing. The
configuration is written to /etc/sysconfig/system-config-firewall and
/etc/sysconfig/iptables.
Uli
Thomas
8:04 p.m.
One way to do this is to place a tiny VM (static kernel+very small
initramfs [uClibc+busybox+iptables+dnsmasq]) between VM clusters and the
host, rather than giving the host an IP on each cluster's bridge
directly. The tool that launches it (via libvirt) appends extra files to
the initramfs giving iptables rules to be run. I use this "virtual
router" to NETMAP multiple clouds of VMs which all think they're using
the same network space (say, 192.168.0.0/24) onto different subnets
(say, 192.168.1.x and 192.168.2.x for the first two clusters), but also
have added support for redirecting connections intended for specific
targets to elsewhere, overriding DNS results for specific hosts, and
other miscellaneous utility functions.
Using a separate VM rather than iptables rules on the host was necessary
in my use case because doing symmetrical NETMAP properly requires
packets from the host to the clients to pass through the PREROUTING
table -- which packets generated within a given host don't do.
If anyone (libvirt/oVirt/whomever) is interested in incorporating this
into their project (even as an entry in a contrib repository), let me
know; I can't distribute binaries without going through some pain and
suffering (setting up a SKU with my employer to ship a CD with sources
to the kernel and the GPLed components of the initrd), but providing it
in source form as a minor patch to someone else's project (and this *is*
actually implemented in very little code -- a mixture of Python and
busybox-friendly shell scripts totaling under 500 lines, so the "minor"
label applies) should be clear sailing.
5749
days inactive
5767
days old
8 comments
6 participants
participants (6)
-
Charles Duffy -
Daniel P. Berrange -
David Lutterkort -
Karl Wirth -
Thomas Woerner -
Ulrich Dangel