[libvirt] How to prevent libvirt from adding iptables rules?
Hi, I'm new to libvirt but not a complete neophite. I'm using libvirt and kvm in ubuntu with "vmbuilder". I'm creating a couple of VMs inside a host that is directly connected to internet with a public routeable address. Since I only have one public address, I won't use bridging. I'm using shorewall (www.shorewall.net) to configure my iptables rules. I intend to use DNAT to route specific ports in the host to one or other VM. With standard masquerading, I give the VMs access to the outside world. At first I used the 'default' network (with a different rfc1918 network)... everything was kinda working until I rebooted the host... at that point I lost connectivity between the outside world and the VMs. From inside the host I had no trouble connecting to the VMs. If I restarted shorewall (which actually cleans all iptables rules and regenerate them according to its configuration) everything works fine. After sending a report and some debugging in the shorewall mailing list, it was clear that libvirt was adding rules to iptables. After reading a bit (http://libvirt.org/formatnetwork.html#examplesPrivate) I created a new network called "isolated". I stopped default (and disabled its autostart), and defined and started isolated. This is the content of isolated.xml: <network> <name>isolated</name> <uuid>51cffbcc-88f5-4edc-a81c-1765c1045691</uuid> <bridge name='virbr%d' stp='on' forwardDelay='0' /> <ip address='10.3.14.1' netmask='255.255.255.0'> <dhcp> <range start='10.3.14.128' end='10.3.14.254' /> </dhcp> </ip> </network> I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started. Is there a way to convince libvirt not to add these rules? Feel free to ask for any data that I didn't send here. TIA. -- Mariano Absatz - "El Baby" el.baby@gmail.com www.clueless.com.ar -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- An expert is a person who has made all the mistakes that can be made in a very narrow field. Niels Bohr Danish physicist (1885 - 1962) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- * TagZilla 0.066 * http://tagzilla.mozdev.org
I'm sorry... is this not the right place to ask this kind of questions? Is there another more user-oriented list or forum? TIA On Tue, Mar 31, 2009 at 16:08, Mariano Absatz <el.baby@gmail.com> wrote:
Hi,
I'm new to libvirt but not a complete neophite.
I'm using libvirt and kvm in ubuntu with "vmbuilder".
I'm creating a couple of VMs inside a host that is directly connected to internet with a public routeable address. Since I only have one public address, I won't use bridging.
I'm using shorewall (www.shorewall.net) to configure my iptables rules.
I intend to use DNAT to route specific ports in the host to one or other VM.
With standard masquerading, I give the VMs access to the outside world.
At first I used the 'default' network (with a different rfc1918 network)... everything was kinda working until I rebooted the host... at that point I lost connectivity between the outside world and the VMs. From inside the host I had no trouble connecting to the VMs.
If I restarted shorewall (which actually cleans all iptables rules and regenerate them according to its configuration) everything works fine. After sending a report and some debugging in the shorewall mailing list, it was clear that libvirt was adding rules to iptables.
After reading a bit (http://libvirt.org/formatnetwork.html#examplesPrivate) I created a new network called "isolated". I stopped default (and disabled its autostart), and defined and started isolated.
This is the content of isolated.xml: <network> <name>isolated</name> <uuid>51cffbcc-88f5-4edc-a81c-1765c1045691</uuid> <bridge name='virbr%d' stp='on' forwardDelay='0' /> <ip address='10.3.14.1' netmask='255.255.255.0'> <dhcp> <range start='10.3.14.128' end='10.3.14.254' /> </dhcp> </ip> </network>
I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
Feel free to ask for any data that I didn't send here.
TIA.
-- Mariano Absatz - El Baby www.clueless.com.ar ######################### "An archaeologist is the best husband a woman can have. The older she gets the more intereste...
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
At first I used the 'default' network (with a different rfc1918 network)... everything was kinda working until I rebooted the host... at that point I lost connectivity between the outside world and the VMs. From inside the host I had no trouble connecting to the VMs.
If I restarted shorewall (which actually cleans all iptables rules and regenerate them according to its configuration) everything works fine. After sending a report and some debugging in the shorewall mailing list, it was clear that libvirt was adding rules to iptables.
Yes, the libvirt virtual network capability adds iptables to control traffic to/from the virtual network.
After reading a bit (http://libvirt.org/formatnetwork.html#examplesPrivate) I created a new network called "isolated". I stopped default (and disabled its autostart), and defined and started isolated.
This is the content of isolated.xml: <network> <name>isolated</name> <uuid>51cffbcc-88f5-4edc-a81c-1765c1045691</uuid> <bridge name='virbr%d' stp='on' forwardDelay='0' /> <ip address='10.3.14.1' netmask='255.255.255.0'> <dhcp> <range start='10.3.14.128' end='10.3.14.254' /> </dhcp> </ip> </network>
I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network. If this is a problem, then the best bet is to not use the virtual network capability. Instead create a bridge device yourself using distro network scripts, and do whatever routing/firewalling setup you need for shorwall to work Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
[...] I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network.
Messing with iptables rules isn't guaranteed to work either. Esp if the existing firewall is re-run. SuSEfirewall2 for example runs when interfaces come or go so it will kill any rules that someone added behind it's back. What kind of iptables rules do you need to install? cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg)
On Thu, Apr 02, 2009 at 10:16:13AM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
[...] I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network.
Messing with iptables rules isn't guaranteed to work either. Esp if the existing firewall is re-run. SuSEfirewall2 for example runs when interfaces come or go so it will kill any rules that someone added behind it's back.
We have a similar issue with the Fedora equivalent of SuSSfirewall, and it provides a mechanism for us to register the set of rules we want, so when it is re-run, it re-adds our rules. As a failsafe, sending SIGHUP to libvirtd will make it re-add its rules so if there's some post-config hook for SuSEfirewall, it could be made to SIGHUP the libvirtd daemon.
What kind of iptables rules do you need to install?
It depends on the particular config, but it is adding sets of rules against the IP range & bridge device config for the interface we add to allow / disallow forwarding of traffic. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
On Thu, Apr 02, 2009 at 10:16:13AM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
[...] I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network.
Messing with iptables rules isn't guaranteed to work either. Esp if the existing firewall is re-run. SuSEfirewall2 for example runs when interfaces come or go so it will kill any rules that someone added behind it's back.
We have a similar issue with the Fedora equivalent of SuSSfirewall, and it provides a mechanism for us to register the set of rules we want, so when it is re-run, it re-adds our rules.
SuSEfirewall2 does not have such a mechanism and TBH I pretty much dislike the idea of allowing applications to inject arbitrary rules. I'd prefer some higher level abstraction so it's left to the firewall to decide how to translate the request into actual iptables rules (or whatever else technology is used in the background). cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg)
On Mon, 2009-04-06 at 14:36 +0200, Ludwig Nussel wrote:
SuSEfirewall2 does not have such a mechanism and TBH I pretty much dislike the idea of allowing applications to inject arbitrary rules. I'd prefer some higher level abstraction so it's left to the firewall to decide how to translate the request into actual iptables rules (or whatever else technology is used in the background).
How would that be done ? I don't know of any tool that could do that, especially given the large number of existing firewall mgmt tools out there (s-c-firewall, SuSEfirewall2, whatever Debian/Ubuntu have, shorewall, pyroman, ...) David
David Lutterkort wrote:
On Mon, 2009-04-06 at 14:36 +0200, Ludwig Nussel wrote:
SuSEfirewall2 does not have such a mechanism and TBH I pretty much dislike the idea of allowing applications to inject arbitrary rules. I'd prefer some higher level abstraction so it's left to the firewall to decide how to translate the request into actual iptables rules (or whatever else technology is used in the background).
How would that be done ? I don't know of any tool that could do that,
Such a mechanism doesn't exist yet. That doesn't mean it can't be implemented in the future. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg)
On Mon, Apr 06, 2009 at 02:36:16PM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Thu, Apr 02, 2009 at 10:16:13AM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
[...] I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network.
Messing with iptables rules isn't guaranteed to work either. Esp if the existing firewall is re-run. SuSEfirewall2 for example runs when interfaces come or go so it will kill any rules that someone added behind it's back.
We have a similar issue with the Fedora equivalent of SuSSfirewall, and it provides a mechanism for us to register the set of rules we want, so when it is re-run, it re-adds our rules.
SuSEfirewall2 does not have such a mechanism and TBH I pretty much dislike the idea of allowing applications to inject arbitrary rules. I'd prefer some higher level abstraction so it's left to the firewall to decide how to translate the request into actual iptables rules (or whatever else technology is used in the background).
I don't much like it either, but currently there isn't any other viable way to provide good network connectivity out of the box, with zero configuration required by the user. In the perfect world we could delegate setup to NetworkManager, and indeed NM's latest connection sharing capabilities does very similar things with IPtables that libvirt does - we worked with the NM developers to make sure our stuff was compatible. So there's potentiall for more work with NM if someone's interested in pursuing that direction Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
On Mon, Apr 06, 2009 at 02:36:16PM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Thu, Apr 02, 2009 at 10:16:13AM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
[...] I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network.
Messing with iptables rules isn't guaranteed to work either. Esp if the existing firewall is re-run. SuSEfirewall2 for example runs when interfaces come or go so it will kill any rules that someone added behind it's back.
We have a similar issue with the Fedora equivalent of SuSSfirewall, and it provides a mechanism for us to register the set of rules we want, so when it is re-run, it re-adds our rules.
SuSEfirewall2 does not have such a mechanism and TBH I pretty much dislike the idea of allowing applications to inject arbitrary rules. I'd prefer some higher level abstraction so it's left to the firewall to decide how to translate the request into actual iptables rules (or whatever else technology is used in the background).
I don't much like it either, but currently there isn't any other viable way to provide good network connectivity out of the box, with zero configuration required by the user.
The idea of making firewalls magically do the right thing without any input from the user just doesn't work out. Anyways, I'm not denying that there currently is no way to have firewall rules adjusted without going low level and calling iptables yourself. That's why I'm interested in what kind of rules you need for what purpose. Maybe a better way to interface with SuSEfirewall2 springs into my mind then. Or maybe not, who knows :-)
In the perfect world we could delegate setup to NetworkManager, and indeed NM's latest connection sharing capabilities does very similar things with IPtables that libvirt does
*shudder* that's even more scary and doesn't belong there either. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg)
Daniel P. Berrange wrote:
On Mon, Apr 06, 2009 at 02:36:16PM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Thu, Apr 02, 2009 at 10:16:13AM +0200, Ludwig Nussel wrote:
Daniel P. Berrange wrote:
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
[...] I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules? No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network. Messing with iptables rules isn't guaranteed to work either. Esp if the existing firewall is re-run. SuSEfirewall2 for example runs when interfaces come or go so it will kill any rules that someone added behind it's back. We have a similar issue with the Fedora equivalent of SuSSfirewall, and it provides a mechanism for us to register the set of rules we want, so when it is re-run, it re-adds our rules. SuSEfirewall2 does not have such a mechanism and TBH I pretty much dislike the idea of allowing applications to inject arbitrary rules. I'd prefer some higher level abstraction so it's left to the firewall to decide how to translate the request into actual iptables rules (or whatever else technology is used in the background).
I don't much like it either, but currently there isn't any other viable way to provide good network connectivity out of the box, with zero configuration required by the user. In the perfect world we could delegate setup to NetworkManager, and indeed NM's latest connection sharing capabilities does very similar things with IPtables that libvirt does - we worked with the NM developers to make sure our stuff was compatible. So there's potentiall for more work with NM if someone's interested in pursuing that direction
Daniel
Can you please define the requirements for a firewall interface for libvirt? Thomas
(sorry, Daniel... I had only answered you instead of copying the list also) Daniel P. Berrange escribió el 01/04/09 09:41:
On Tue, Mar 31, 2009 at 04:08:24PM -0300, Mariano Absatz wrote:
At first I used the 'default' network (with a different rfc1918 network)... everything was kinda working until I rebooted the host... at that point I lost connectivity between the outside world and the VMs. From inside the host I had no trouble connecting to the VMs.
If I restarted shorewall (which actually cleans all iptables rules and regenerate them according to its configuration) everything works fine. After sending a report and some debugging in the shorewall mailing list, it was clear that libvirt was adding rules to iptables.
Yes, the libvirt virtual network capability adds iptables to control traffic to/from the virtual network.
After reading a bit (http://libvirt.org/formatnetwork.html#examplesPrivate) I created a new network called "isolated". I stopped default (and disabled its autostart), and defined and started isolated.
This is the content of isolated.xml: <network> <name>isolated</name> <uuid>51cffbcc-88f5-4edc-a81c-1765c1045691</uuid> <bridge name='virbr%d' stp='on' forwardDelay='0' /> <ip address='10.3.14.1' netmask='255.255.255.0'> <dhcp> <range start='10.3.14.128' end='10.3.14.254' /> </dhcp> </ip> </network>
I modified my VMs to use isolated rather than default, but rules keep being added to iptables when libvirt-bin is started.
Is there a way to convince libvirt not to add these rules?
No, libvirt needs to add the rules here because otherwise the guest virtual network would not be guarenteed to be isolated from the host network.
If this is a problem, then the best bet is to not use the virtual network capability. Instead create a bridge device yourself using distro network scripts, and do whatever routing/firewalling setup you need for shorwall to work
Daniel
I see.. so I can't just ask libvirt to create the bridge for me and not touch iptables rules... I chose "isolated" just hoping that would be the way of preventing the addition of iptables rules... The problem at this time is that, other than the rules I see libvirt adds are conflicting with my rules (since they are inserted at the top of INPUT and FORWARD before mine): Chain INPUT (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 ACCEPT udp -- vnet0 * 0.0.0.0/0 <http://0.0.0.0/0> 0.0.0.0/0 <http://0.0.0.0/0> udp dpt:53 0 0 ACCEPT tcp -- vnet0 * 0.0.0.0/0 <http://0.0.0.0/0> 0.0.0.0/0 <http://0.0.0.0/0> tcp dpt:53 0 0 ACCEPT udp -- vnet0 * 0.0.0.0/0 <http://0.0.0.0/0> 0.0.0.0/0 <http://0.0.0.0/0> udp dpt:67 0 0 ACCEPT tcp -- vnet0 * 0.0.0.0/0 <http://0.0.0.0/0> 0.0.0.0/0 <http://0.0.0.0/0> tcp dpt:67 Chain FORWARD (policy DROP 0 packets, 0 bytes) pkts bytes target prot opt in out source destination - 0 0 ACCEPT all -- vnet0 vnet0 0.0.0.0/0 <http://0.0.0.0/0> 0.0.0.0/0 <http://0.0.0.0/0> - 0 0 REJECT all -- * vnet0 0.0.0.0/0 <http://0.0.0.0/0> 0.0.0.0/0 <http://0.0.0.0/0> reject-with icmp-port-unreachable - 0 0 REJECT all -- vnet0 * 0.0.0.0/0 <http://0.0.0.0/0> 0.0.0.0/0 <http://0.0.0.0/0> reject-with icmp-port-unreachable Well... for the time being, I think I'll add a "shorewall restart" at the end of rc.local which will kill these rules and leave only the ones that shorewall generates... -- Mariano Absatz - "El Baby" el.baby@gmail.com www.clueless.com.ar -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Light travels faster than sound. This is why some people appear bright until you hear them speak. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- * TagZilla 0.066 * http://tagzilla.mozdev.org
participants (5)
-
Daniel P. Berrange -
David Lutterkort -
Ludwig Nussel -
Mariano Absatz -
Thomas Woerner