12:19 p.m.
Hi,
I just found out that libvirt always add some iptables rules if it creates a
natted (or routed) network. There were a couple of mailing list posts about
this so I'm pretty sure this is not news to you.
I don't want to go into the debate if your approach is sensible or not (I
guess there are some use cases where I kind of like it). However on my server
machine I really need full control over my (rather complicated) firewall
settings.
Currently the newly added rules really create a lot of problems for me. For
example if I manage to have a good configuration after startup and then start
a libvirt network afterwards, it will inject its rules at the start of the
FORWARD queue (even though the same parameters are already present at the
end!). On every net start there will be more duplicated rules and they will
take preference over my existing rules.
Besides that specific issue I think this is only one tiny problem compared to
others (central configuration of firewall rules, auditing requirements, ...).
Therefore I would like to have some kind 'power user' flag that prevents
libvirt from adding any filter rules. I'm fine with activating it manually as
long as I don't have to patch libvirt.
fs
4:52 p.m.
After some more hours of trying to fix libvirt's behavior, I decided to fix
this issue by patching libvirt. Patch for 0.6.3 attached. If there's hope
getting something like that into the git repository, I can port it to 0.7 easily.
fs
6:04 a.m.
On Sun, Dec 20, 2009 at 07:19:06PM +0100, Felix Schwarz wrote:
Hi,
I just found out that libvirt always add some iptables rules if it creates
a natted (or routed) network. There were a couple of mailing list posts
about this so I'm pretty sure this is not news to you.
I don't want to go into the debate if your approach is sensible or not (I
guess there are some use cases where I kind of like it). However on my
server machine I really need full control over my (rather complicated)
firewall settings.
Currently the newly added rules really create a lot of problems for me. For
example if I manage to have a good configuration after startup and then
start a libvirt network afterwards, it will inject its rules at the start
of the FORWARD queue (even though the same parameters are already present
at the end!). On every net start there will be more duplicated rules and
they will take preference over my existing rules.
There shold never be duplicated rules. If you stop a libvirt virutal network,
it will remove its previously added rules, so there should be no duplication
next time it is started. If removal isn't working, that's a bug to be fixed.
Can you outline how your desired configuration for libvirt NAT mode is
different from what libvirt already does ? The goal for this is to be
totally zero-conf, so that fact that you can't use the default setup
shows something is lacking in our impl & I'd prefer to identify what
that is rather than blindly disabling it. In addition the libvirt rules
are written to try & ensure that they only impact traffic to/from the
subnet that is configured in the libvirt network, to avoid causing problems
for other rules you might have already configured.
Besides that specific issue I think this is only one tiny problem
compared
to others (central configuration of firewall rules, auditing requirements,
...).
Therefore I would like to have some kind 'power user' flag that prevents
libvirt from adding any filter rules. I'm fine with activating it manually
as long as I don't have to patch libvirt.
This isn't really something we want to support. As I mention above we
want to make sure this works out of the box without manual config.
The one change we do want to make to the setup, is to move all the rules
into dedicated chains (libvirt_INPUT, libvirt_FORWARD, etc) so that we
only add a single rule to the main INPUT/FORWARD chains.
Regards,
Daniel.
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
8:42 a.m.
Am 21.12.2009 13:04, schrieb Daniel P. Berrange:
There shold never be duplicated rules. If you stop a libvirt virutal
network,
it will remove its previously added rules, so there should be no duplication
next time it is started. If removal isn't working, that's a bug to be fixed.
I had two different networks, one with nat, one routed. Only one is started
with autostart. As soon as I start the other, I get additional (duplicated I
think) rules.
Can you outline how your desired configuration for libvirt NAT mode
is
different from what libvirt already does ? The goal for this is to be
totally zero-conf, so that fact that you can't use the default setup
shows something is lacking in our impl& I'd prefer to identify what
that is rather than blindly disabling it.
Actually my main interest is the routed mode, not NAT.
This is my iptables after I started two networks (no other packet filter):
# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 192.168.78.21
ACCEPT all -- 192.168.78.21 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
ACCEPT all -- anywhere 192.168.122.0/24 state
RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
(...)
My issues:
1) INPUT chain ACCEPTs DNS/dhcp from outside
You might notice that the INPUT chain basically says that I ACCEPT all
DNS/dhcp from all interfaces. I don't want that. As soon as I configure a
packet filter (e.g. shorewall), libvirt's configuration will take precedence.
2) FORWARD contains general rules
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
These rules apply to all FORWARDed connections. I need *way* more control.
3) FORWARD ACCEPTs packets from all hosts
ACCEPT all -- anywhere 192.168.78.21
ACCEPT all -- 192.168.78.21 anywhere
Say I have routed libvirt network but I want to protect these hosts - only
some specific hosts may reach them (e.g. a virtualized backend app server is
only reachable by the frontend servers). With the generated iptables rules I
can not do that.
4) No way to override rules
All new iptables rules are pre-prepended when a new network is started (which
may happen at any time), potentially circumventing all existing rules.
5) Company policies
How do you keep firewall rules manageable/auditable in 'not extremly simple'
situations?
Many companies I know have a very strict policy that only one application is
allowed to define rules (e.g. shorewall or a proprietary FW). I mean you @Red
Hat should know stuff like that. If libvirt touches my carefully reviewed
policies, it might open a lot of security issues.
That being said I appreciate your approach to make it easy for simple cases
and desktop end users. In fact, I'm using libvirt since Fedora 10 on a desktop
with problems. Now with RHEL 5.4 I'm starting to use that on servers and here
I need way more control.
I guess there are a lot more use cases when you just need to disable automated
iptables changes - just because libvirt does not have the whole picture.
> Therefore I would like to have some kind 'power user'
flag that prevents
> libvirt from adding any filter rules. I'm fine with activating it manually
> as long as I don't have to patch libvirt.
This isn't really something we want to support. As I mention above we
want to make sure this works out of the box without manual config.
I can totally understand you - but how do you think you can deal with system
security if libvirt just does not have all information? How can I use a
libvirt host as a router, only giving specific IPs accesss to a routed network?
The one change we do want to make to the setup, is to move all the
rules
into dedicated chains (libvirt_INPUT, libvirt_FORWARD, etc) so that we
only add a single rule to the main INPUT/FORWARD chains.
I'm afraid that this won't help in my situation: Still all the rules are
prepended and I can not specify which rules should be inserted.
fs
9 a.m.
On Mon, Dec 21, 2009 at 03:42:17PM +0100, Felix Schwarz wrote:
Am 21.12.2009 13:04, schrieb Daniel P. Berrange:
>There shold never be duplicated rules. If you stop a libvirt virutal
>network,
>it will remove its previously added rules, so there should be no
>duplication
>next time it is started. If removal isn't working, that's a bug to be
>fixed.
I had two different networks, one with nat, one routed. Only one is started
with autostart. As soon as I start the other, I get additional (duplicated
I think) rules.
>Can you outline how your desired configuration for libvirt NAT mode is
>different from what libvirt already does ? The goal for this is to be
>totally zero-conf, so that fact that you can't use the default setup
>shows something is lacking in our impl& I'd prefer to identify what
>that is rather than blindly disabling it.
Actually my main interest is the routed mode, not NAT.
This is my iptables after I started two networks (no other packet filter):
# iptables --list
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 192.168.78.21
ACCEPT all -- 192.168.78.21 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
ACCEPT all -- anywhere 192.168.122.0/24 state
RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
(...)
My issues:
1) INPUT chain ACCEPTs DNS/dhcp from outside
You might notice that the INPUT chain basically says that I ACCEPT all
DNS/dhcp from all interfaces. I don't want that. As soon as I configure a
packet filter (e.g. shorewall), libvirt's configuration will take
precedence.
No it doesn't say that. You are missing the '-v' flag to list the rules.
If you add that you'll see that the rules are *different* and they all
explicitly include the name of the bridge interface associated with the
libvirt network
2) FORWARD contains general rules
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with
icmp-port-unreachable
These rules apply to all FORWARDed connections. I need *way* more control.
Again, use the -v flag to iptables to see the full set of constraints.
We explicitly try to make libvirt rules *only* apply to the libvirt
created interfaces and not the rest of your interfaces/configuration.
3) FORWARD ACCEPTs packets from all hosts
ACCEPT all -- anywhere 192.168.78.21
ACCEPT all -- 192.168.78.21 anywhere
Say I have routed libvirt network but I want to protect these hosts - only
some specific hosts may reach them (e.g. a virtualized backend app server
is only reachable by the frontend servers). With the generated iptables
rules I can not do that.
4) No way to override rules
All new iptables rules are pre-prepended when a new network is started
(which may happen at any time), potentially circumventing all existing
rules.
That is intentionale, and since the libvirt rules only match on the libvirt
network interface name & ip range, it should not be opening any holes in
your current config for other interfaces.
5) Company policies
How do you keep firewall rules manageable/auditable in 'not extremly
simple' situations?
Many companies I know have a very strict policy that only one application
is allowed to define rules (e.g. shorewall or a proprietary FW). I mean you
@Red Hat should know stuff like that. If libvirt touches my carefully
reviewed policies, it might open a lot of security issues.
I agree that corporate policy/compliance issues are probably the main
stumbling block here. In some cases, improving the documentation about
exactly what libvirt does with iptables may help. eg attempt to show
that libvirt's rules won't open any holes in the firewall beyond what
is explicitly intended
This obviously won't be enough for everyone's policy/compliance needs
though. In such strict managed deployments, I thing the libvirt virtua
network functionality is simply not going to be possible to use. Once
you've taken away the iptables setup, they there ceases to be much point
in using this functionality as it is. There are other libvirt APIs that
would suit better, such as the network interface management APIs we
recently added.
That being said I appreciate your approach to make it easy for simple
cases
and desktop end users. In fact, I'm using libvirt since Fedora 10 on a
desktop with problems. Now with RHEL 5.4 I'm starting to use that on
servers and here I need way more control.
I guess there are a lot more use cases when you just need to disable
automated iptables changes - just because libvirt does not have the whole
picture.
We'd really like to make sure libvirt can get the bigger picture if
at all possible. It is a core goal with libvirt functionality, that it
be possible to use it all from a remote client without needing to login
into the virt host as an a shell admin.
>> Therefore I would like to have some kind 'power
user' flag that prevents
>> libvirt from adding any filter rules. I'm fine with activating it
manually
>> as long as I don't have to patch libvirt.
>
> This isn't really something we want to support. As I mention above we
> want to make sure this works out of the box without manual config.
I can totally understand you - but how do you think you can deal with
system security if libvirt just does not have all information? How can I
use a libvirt host as a router, only giving specific IPs accesss to a
routed network?
Can you explain a little more about your routed setup ? In particular,
are you trying to use the same IP address range for VMs and your LAN,
and thus just route a handful of IPs ? Or are you having a separate
subnet for the VMs ? I know libvirt won't cope with the former scenario
currently, since as you say it would need to know which IPs to route.
We can deal with the separate-subnet scenario though & that shouldn't
require any per-IP setup on the virt host
I wrote a blog post about the routed network scenario that we currently
support & expect to work
http://berrange.com/personal/diary/2009/12/routed-subnets-without-nat-for...
As per the last paragraph, I'd like us to add extra support for Proxy-ARP
(optionally with subnetting) to our routed config
> The one change we do want to make to the setup, is to move all
the rules
> into dedicated chains (libvirt_INPUT, libvirt_FORWARD, etc) so that we
> only add a single rule to the main INPUT/FORWARD chains.
I'm afraid that this won't help in my situation: Still all the rules are
prepended and I can not specify which rules should be inserted.
Regards,
Daniel
--
|: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :|
|: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
8:06 a.m.
Am 21.12.2009 16:00, schrieb Daniel P. Berrange:
> My issues:
> 1) INPUT chain ACCEPTs DNS/dhcp from outside
>
> You might notice that the INPUT chain basically says that I ACCEPT all
> DNS/dhcp from all interfaces. I don't want that. As soon as I configure a
> packet filter (e.g. shorewall), libvirt's configuration will take
> precedence.
No it doesn't say that. You are missing the '-v' flag to list the rules.
If you add that you'll see that the rules are *different* and they all
explicitly include the name of the bridge interface associated with the
libvirt network
You're right - actually I did not check closely enough. Sorry for that.
I agree that corporate policy/compliance issues are probably the
main
stumbling block here. (...)
This obviously won't be enough for everyone's policy/compliance needs
though. In such strict managed deployments, I thing the libvirt virtua
network functionality is simply not going to be possible to use. Once
you've taken away the iptables setup, they there ceases to be much point
in using this functionality as it is. There are other libvirt APIs that
would suit better, such as the network interface management APIs we
recently added.
Which APIs do you think of? To me it looked like libvirt should become the
default configuration layer whenever you do something with virtual machines
(as it is configured by default, most configuration tools use it, ...).
Therefore I tried to make my setup work with libvirt to make use of all that
integration stuff...
Can you explain a little more about your routed setup ? In
particular,
are you trying to use the same IP address range for VMs and your LAN,
and thus just route a handful of IPs ?
Basically yes: This is a server in a data center with a couple of IPs that are
assigned by my provider (no subnet). So I assign one IP to my host and route
the others to libvirt interfaces so that my VMs can provide public services as
well.
I need a routed setup due to MAC address filtering in the switches.
I know libvirt won't cope with the former scenario
currently, since as you say it would need to know which IPs to route.
We can deal with the separate-subnet scenario though& that shouldn't
require any per-IP setup on the virt host
Actually there are not that many ipv4 addresses left so there are only 4 IPs
included in my plan (used to be 1 + subnet with 6 usable IPs). Therefore I get
only single IP addresses.
fs
6:24 a.m.
Hello,
Can you outline how your desired configuration for libvirt NAT mode
is
different from what libvirt already does ? The goal for this is to be
totally zero-conf, so that fact that you can't use the default setup
shows something is lacking in our impl & I'd prefer to identify what
that is rather than blindly disabling it. In addition the libvirt
rules are written to try & ensure that they only impact traffic
to/from the subnet that is configured in the libvirt network, to avoid
causing problems for other rules you might have already configured.
I opened a bug report[1] for this too, doing the right thing for
out-of-the-box configuration is ok, but everything should be opt-out and
manually configurable.
I add sanity-check rules at top of my netfilter chains and when a
libvirt network start it's not "protected" by theses rules.
It's like my bug report on dnsmasq[2], I already have a complete
DHCP/DNS-with-LDAP-backend configuration for the subnet, I don't need it
but can not opt-out the feature.
This disempower the user/administrator, which I think is bad.
So, what I whould like to see:
1. Automatic configuration for out-of-the-box setup
2. Opt-out all the automatic configurations
3. Manually configurable, with pre-up(before), up(doing it),
post-up(after) and their down counterparts.
Please.
Footnotes:
[1] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=568790
[2] http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=549183
--
Daniel Dehennin
Récupérer ma clef GPG:
gpg --keyserver pgp.mit.edu --recv-keys 0x6A2540D1
5306
days inactive
5481
days old
6 comments
3 participants
participants (3)
-
Daniel Dehennin -
Daniel P. Berrange -
Felix Schwarz