[libvirt-users] Networkfilters in Routed setup
Hello! Since i could not find any information on the internet about this subject, i'm going to try my luck on this list. I'm trying to setup network-filter on a routed setup. I have a root-server at Hetzner, a german hosting provider. Along with my server i ordered a (/28) subnet to be able to setup dedicated IPs for my virtual machines (KVM). My Server is running Ubuntu 12.04 with libvirt 0.9.8 . Since Hetzner does not allow any bridged traffic, i had to setup a routed network. Currently my (via libvirt) defined network looks like this: (lets assume my subnet is 1.2.3.64/28): <network> <name>hetzner-subnet-v4</name> <forward dev='eth0' mode='route'> <interface dev='eth0'/> </forward> <bridge name='route-br0' stp='off' delay='0' /> <mac address='52:54:00:F0:D0:AA'/> <ip address='1.2.3.65' netmask='255.255.255.240'></ip> </network> The network definition for all running VMs looks like this: <interface type='network'> <mac address='52:54:00:00:00:##'/> <source network='hetzner-subnet-v4'/> </interface> Without using Network-Filters, this setup is running as expected. All traffic is correctly forwarded to my virtual machines connected to "route-br0" and the following iptables-rules are created in the FORWARD Chain: Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) target prot opt in out source destination ACCEPT all -- eth0 route-br0 0.0.0.0/0 1.2.3.64/28 ACCEPT all -- route-br0 eth0 1.2.3.64/28 0.0.0.0/0 ACCEPT all -- route-br0 route-br0 0.0.0.0/0 0.0.0.0/0 REJECT all -- * route-br0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable REJECT all -- route-br0 * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable When i try to setup a network-filter for a VM (a modified version of http://libvirt.org/formatnwfilter.html last example): <filter name='server-x' chain='root'> <filterref filter='clean-traffic'/> <rule action='accept' direction='in' priority='500'> <all state='ESTABLISHED'/> </rule> <rule action='accept' direction='out' priority='500'> <all state='ESTABLISHED,RELATED'/> </rule> <rule action='accept' direction='in' priority='500'> <tcp state='NEW' dstportstart='22'/> </rule> <rule action='accept' direction='out' priority='500'> <all state='NEW'/> </rule> <rule action='drop' direction='inout' priority='500'> <all/> </rule> </filter> and adding the filter to my interface-definition of a VM using the following syntax: <filterref filter='server-x'> <parameter name='IP' value='1.2.3.70'/> </filterref> additional iptable-rules are getting created. The problematic rule seems to be the following: -A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX which should trigger the following rules: -A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT -A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT But this actually never happens. The FO-vnetX Chain never sees any packets and my syslog says: xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore Am i doing something wrong? I hope i did not write too much useless stuff here. I'm tried to figure it all out by myself, but im currently stuck. Lets hope some wise guys can help me out here. Maybe there is some documentation i have missed? Thanks! kind regards, Sebastian
On 02/14/2014 08:40 PM, h0rst wrote:
Hello!
Since i could not find any information on the internet about this subject, i'm going to try my luck on this list.
I'm trying to setup network-filter on a routed setup. I have a root-server at Hetzner, a german hosting provider. Along with my server i ordered a (/28) subnet to be able to setup dedicated IPs for my virtual machines (KVM). My Server is running Ubuntu 12.04 with libvirt 0.9.8 . Since Hetzner does not allow any bridged traffic,
You *really* should upgrade to a newer libvirt.
Without using Network-Filters, this setup is running as expected. All traffic is correctly forwarded to my virtual machines connected to "route-br0" and the following iptables-rules are created in the FORWARD Chain:
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) target prot opt in out source destination ACCEPT all -- eth0 route-br0 0.0.0.0/0 1.2.3.64/28 ACCEPT all -- route-br0 eth0 1.2.3.64/28 0.0.0.0/0 ACCEPT all -- route-br0 route-br0 0.0.0.0/0 0.0.0.0/0 REJECT all -- * route-br0 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable REJECT all -- route-br0 * 0.0.0.0/0
Those previous two rules are the ones added when you specify a forward dev. You don't need to do that - I would recommend removing the "dev='eth0'" from the <forward> element of the network, along with the "<interface dev='eth0'/>" subelement. This won't change operation at all, it will just make things slightly less confusing and misleading.
0.0.0.0/0 reject-with icmp-port-unreachable
When i try to setup a network-filter for a VM (a modified version of http://libvirt.org/formatnwfilter.html last example):
<filter name='server-x' chain='root'> <filterref filter='clean-traffic'/> <rule action='accept' direction='in' priority='500'> <all state='ESTABLISHED'/> </rule> <rule action='accept' direction='out' priority='500'> <all state='ESTABLISHED,RELATED'/> </rule> <rule action='accept' direction='in' priority='500'> <tcp state='NEW' dstportstart='22'/> </rule> <rule action='accept' direction='out' priority='500'> <all state='NEW'/> </rule> <rule action='drop' direction='inout' priority='500'> <all/> </rule> </filter>
and adding the filter to my interface-definition of a VM using the following syntax:
<filterref filter='server-x'> <parameter name='IP' value='1.2.3.70'/> </filterref>
additional iptable-rules are getting created. The problematic rule seems to be the following:
-A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX
which should trigger the following rules:
-A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT -A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT
But this actually never happens. The FO-vnetX Chain never sees any packets and my syslog says:
xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore
That somehow sounded familiar, so I looked it up in the git history and found this: http://libvirt.org/git/?p=libvirt.git;a=commit;h=65fb9d49cc9caae210977934b53... That patch was included in libvirt-1.0.2, just about a year ago.
Am i doing something wrong?
YOu need to upgrade your libvirt to at least 1.0.2 (preferably newer).
On Di, 2014-02-18 at 12:03 +0200, Laine Stump wrote:
You *really* should upgrade to a newer libvirt.
I know that version 0.9.8 is very old. But to be honest i tried to avoid upgrading and compiling a newer version since i don't know if it has any effects on running VMs (but i haven't checked this yet). Its a production server and i did not want to interrupt any services running on these VMs. However, i'm afraid that upgrading might be the only option if i want to avoid setting up iptables manually.
Those previous two rules are the ones added when you specify a forward dev. You don't need to do that - I would recommend removing the "dev='eth0'" from the <forward> element of the network, along with the "<interface dev='eth0'/>" subelement. This won't change operation at all, it will just make things slightly less confusing and misleading.
Thank you for pointing this out. I just did that. And after a reboot everything is still working as expected (yes. I just DID interrupt the services running in my VMs. So i guess i could even upgrade to a newer libvirt =) )
-A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX
which should trigger the following rules:
-A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT -A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT
But this actually never happens. The FO-vnetX Chain never sees any packets and my syslog says:
xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for non-bridged traffic is not supported anymore
That somehow sounded familiar, so I looked it up in the git history and found this:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=65fb9d49cc9caae210977934b53...
That patch was included in libvirt-1.0.2, just about a year ago.
After reading that i remove the following iptables rule: iptables -D libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX and manually added this rule: (the patch said that adding an extra argument (--physdev-is-bridged) is needed for rules like this): iptables -A libvirt-out -m physdev --physdev-is-bridged --physdev-out vnetX -g FO-vnetX Indeed this prevents my syslog from being spammed with the mentioned warning. However, this did not fix the problem. This rule never matches anything, and thus the FO-vnetX Chain never sees any packets. Using a testing rule like: iptables -A libvirt-out -d 1.2.3.70 -g FO-vnetX made everything work as expected. Well, this is definitely not the way it is expected to work since it does not do any "bridge port" matching. This all makes me think its not a libvirt specific problem and updating to a newer version will not fix my problem. Maybe there is somebody out there using a setup like mine and can show me the rules that a getting created with a newer version of libvirt? So long, thank you for all the usefull information! Kind regards, Sebastian
On 02/18/2014 06:31 AM, h0rst wrote:
On Di, 2014-02-18 at 12:03 +0200, Laine Stump wrote:
You *really* should upgrade to a newer libvirt.
I know that version 0.9.8 is very old. But to be honest i tried to avoid upgrading and compiling a newer version since i don't know if it has any effects on running VMs (but i haven't checked this yet).
There should be no problem upgrading to a newer libvirt. We take great pains to ensure that a newer version of libvirt can be reloaded and gracefully understand the XML recorded by older versions, with no loss to running VMs. While there have been bugs on this front, they get caught and patched quickly, so by updating to something like the latest Fedora stable build (currently 1.1.3.3), you are even more likely to avoid these sorts of problems when compared to upgrading all the way to the master branch of libvirt. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Di, 2014-02-18 at 16:06 -0700, Eric Blake wrote:
There should be no problem upgrading to a newer libvirt. We take great pains to ensure that a newer version of libvirt can be reloaded and gracefully understand the XML recorded by older versions, with no loss to running VMs. While there have been bugs on this front, they get caught and patched quickly, so by updating to something like the latest Fedora stable build (currently 1.1.3.3), you are even more likely to avoid these sorts of problems when compared to upgrading all the way to the master branch of libvirt.
After i forced myself to upgrade and compile a newer version of libvirt, (to be precise, i wanted to upgrade to the latest libvirt-1.2.1.tar.gz) i ran into some problems. I compiled libvirt with the following options: As a reminder, i'm running ""good"" old Ubuntu 12.04: ./configure --with-lxc --with-storage-lvm --prefix=/usr --localstatedir=/var --sysconfdir=/etc Compiling and installing worked perfectly after installing all missing dependencies. At first everything looked fine and all created networks and domains where still running. To be sure everything would survive a system restart, i rebooted. Thats when everything (or something) went wrong. After starting libvirtd i got the following error:
error: Failed to start network hetzner-subnet-v4 error: unsupported configuration: Publicly routable address 1.2.3.65 is prohibited. The version of dnsmasq on this host (2.59) doesn't support the bind-dynamic option or use SO_BINDTODEVICE on listening sockets, one of which is required for safe operation on a publicly routable subnet (see CVE-2012-3411). You must either upgrade dnsmasq, or use a private/local subnet range for this network (as described in RFC1918/RFC3484/RFC4193).
Since no VM was running at this point (because of the missing networks), i decided to quickly update to a newer version of DNSMASQ (2.68) and installed this to "/usr/local/sbin" and linked it to "/usr/sbin/dnsmasq" after removing the distribution specific packages. When i tried to start the hetzner-subnet-v4 network, i got the following error:
2014-02-19 14:11:58.636+0000: 7075: error : virCommandWait:2376 : internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root USER=root LOGNAME=root /usr/sbin/dnsmasq --version) unexpected exit status 1: libvirt: error : cannot execute binary /usr/sbin/dnsmasq: Permission denied
2014-02-19 14:11:58.636+0000: 7075: error : dnsmasqCapsRefreshInternal:747 : failed to run '/usr/sbin/dnsmasq --version': : Success error: Failed to start network hetzner-subnet-v4 error: failed to run '/usr/sbin/dnsmasq --version': : Success
However, running dnsmasq manually worked. Since that was the moment the first phonecalls started because users could not access their services on the VMs i quickly reverted everything to its previous state to get everything up and running again. I dont have any testing server, so i could not play around with it anymore (better to say not right now. I might have to wait until everyone is sleeping ;)). Does libvirt has any problems when accessing a softlink instead of a binary? Poorly that possibility came into my mind after reverting back to its original state! I'm really sorry to spam you guys with all my problems ;) Kind regards, Sebastian
On 02/19/2014 08:47 AM, h0rst wrote:
2014-02-19 14:11:58.636+0000: 7075: error : virCommandWait:2376 : internal error: Child process (LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root USER=root LOGNAME=root /usr/sbin/dnsmasq --version) unexpected exit status 1: libvirt: error : cannot execute binary /usr/sbin/dnsmasq: Permission denied
You'll need to figure out where the permission denied is coming from; perhaps the problem is that your symlink requires resolution to a directory that is not readable bu the
2014-02-19 14:11:58.636+0000: 7075: error : dnsmasqCapsRefreshInternal:747 : failed to run '/usr/sbin/dnsmasq --version': : Success error: Failed to start network hetzner-subnet-v4 error: failed to run '/usr/sbin/dnsmasq --version': : Success
These remaining error messages look awkward; we probably have issues in the code that overwrites the useful message when it shouldn't. But that's just cosmetic, compared to getting the real root cause of why libvirtd can't execute dnsmasq.
However, running dnsmasq manually worked. Since that was the moment the first phonecalls started because users could not access their services on the VMs i quickly reverted everything to its previous state to get everything up and running again. I dont have any testing server, so i could not play around with it anymore (better to say not right now. I might have to wait until everyone is sleeping ;)). Does libvirt has any problems when accessing a softlink instead of a binary? Poorly that possibility came into my mind after reverting back to its original state!
Libvirt shouldn't have any problems following a symlink, unless the symlink resolves to a path that doesn't have proper permissions. Can you paste actual terminal transcripts proving that you can manually execute /usr/sbin/dnsmasq --version? -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Mi, 2014-02-19 at 09:26 -0700, Eric Blake wrote:
Libvirt shouldn't have any problems following a symlink, unless the symlink resolves to a path that doesn't have proper permissions. Can you paste actual terminal transcripts proving that you can manually execute /usr/sbin/dnsmasq --version?
Yes. Sorry. I forgot to include those calls proving that in manually worked. user@hv:~$ ls -la /usr/sbin/dnsmasq lrwxrwxrwx 1 root root 23 Feb 19 18:26 /usr/sbin/dnsmasq -> /usr/local/sbin/dnsmasq user@hv:~$ ls -la /usr/local/sbin/dnsmasq -rwxr-xr-x 1 root root 302941 Feb 19 18:25 /usr/local/sbin/dnsmasq As unprivileged user: user@hv:~$ /usr/sbin/dnsmasq --version Dnsmasq version 2.68 Copyright (c) 2000-2013 Simon Kelley Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth This software comes with ABSOLUTELY NO WARRANTY. Dnsmasq is free software, and you are welcome to redistribute it under the terms of the GNU General Public License, version 2 or 3. As root: root@hv ~ # /usr/sbin/dnsmasq --version Dnsmasq version 2.68 Copyright (c) 2000-2013 Simon Kelley Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth This software comes with ABSOLUTELY NO WARRANTY. Dnsmasq is free software, and you are welcome to redistribute it under the terms of the GNU General Public License, version 2 or 3. I can do more test if the users aren't awake and cant complain about non working services and VMs ;) Kind regards, Sebastian
participants (3)
-
Eric Blake -
h0rst -
Laine Stump