12:40 p.m.
Hello!
Since i could not find any information on the internet about this subject, i'm going
to try my luck on this list.
I'm trying to setup network-filter on a routed setup. I have a root-server at Hetzner,
a german hosting provider.
Along with my server i ordered a (/28) subnet to be able to setup dedicated IPs for my
virtual machines (KVM).
My Server is running Ubuntu 12.04 with libvirt 0.9.8 . Since Hetzner does not allow any
bridged traffic, i had
to setup a routed network. Currently my (via libvirt) defined network looks like this:
(lets assume my subnet is 1.2.3.64/28):
<network>
<name>hetzner-subnet-v4</name>
<forward dev='eth0' mode='route'>
<interface dev='eth0'/>
</forward>
<bridge name='route-br0' stp='off' delay='0' />
<mac address='52:54:00:F0:D0:AA'/>
<ip address='1.2.3.65' netmask='255.255.255.240'></ip>
</network>
The network definition for all running VMs looks like this:
<interface type='network'>
<mac address='52:54:00:00:00:##'/>
<source network='hetzner-subnet-v4'/>
</interface>
Without using Network-Filters, this setup is running as expected. All traffic is correctly
forwarded to my virtual
machines connected to "route-br0" and the following iptables-rules are created
in the FORWARD Chain:
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
target prot opt in out source destination
ACCEPT all -- eth0 route-br0 0.0.0.0/0 1.2.3.64/28
ACCEPT all -- route-br0 eth0 1.2.3.64/28 0.0.0.0/0
ACCEPT all -- route-br0 route-br0 0.0.0.0/0 0.0.0.0/0
REJECT all -- * route-br0 0.0.0.0/0 0.0.0.0/0
reject-with icmp-port-unreachable
REJECT all -- route-br0 * 0.0.0.0/0 0.0.0.0/0
reject-with icmp-port-unreachable
When i try to setup a network-filter for a VM (a modified version of
http://libvirt.org/formatnwfilter.html last example):
<filter name='server-x' chain='root'>
<filterref filter='clean-traffic'/>
<rule action='accept' direction='in' priority='500'>
<all state='ESTABLISHED'/>
</rule>
<rule action='accept' direction='out' priority='500'>
<all state='ESTABLISHED,RELATED'/>
</rule>
<rule action='accept' direction='in' priority='500'>
<tcp state='NEW' dstportstart='22'/>
</rule>
<rule action='accept' direction='out' priority='500'>
<all state='NEW'/>
</rule>
<rule action='drop' direction='inout' priority='500'>
<all/>
</rule>
</filter>
and adding the filter to my interface-definition of a VM using the following syntax:
<filterref filter='server-x'>
<parameter name='IP' value='1.2.3.70'/>
</filterref>
additional iptable-rules are getting created. The problematic rule seems to be the
following:
-A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX
which should trigger the following rules:
-A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT
-A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT
But this actually never happens. The FO-vnetX Chain never sees any packets and my syslog
says:
xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for
non-bridged traffic is not supported anymore
Am i doing something wrong? I hope i did not write too much useless stuff here. I'm
tried to figure it all out by
myself, but im currently stuck. Lets hope some wise guys can help me out here. Maybe there
is some documentation i have missed?
Thanks!
kind regards,
Sebastian
4:03 a.m.
On 02/14/2014 08:40 PM, h0rst wrote:
Hello!
Since i could not find any information on the internet about this subject, i'm going
to try my luck on this list.
I'm trying to setup network-filter on a routed setup. I have a root-server at
Hetzner, a german hosting provider.
Along with my server i ordered a (/28) subnet to be able to setup dedicated IPs for my
virtual machines (KVM).
My Server is running Ubuntu 12.04 with libvirt 0.9.8 . Since Hetzner does not allow any
bridged traffic,
You *really* should upgrade to a newer libvirt.
Without using Network-Filters, this setup is running as expected. All
traffic is correctly forwarded to my virtual
machines connected to "route-br0" and the following iptables-rules are created
in the FORWARD Chain:
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
target prot opt in out source destination
ACCEPT all -- eth0 route-br0 0.0.0.0/0 1.2.3.64/28
ACCEPT all -- route-br0 eth0 1.2.3.64/28 0.0.0.0/0
ACCEPT all -- route-br0 route-br0 0.0.0.0/0 0.0.0.0/0
REJECT all -- * route-br0 0.0.0.0/0 0.0.0.0/0
reject-with icmp-port-unreachable
REJECT all -- route-br0 * 0.0.0.0/0
Those previous two rules are the ones added when you specify a forward
dev. You don't need to do that - I would recommend removing the
"dev='eth0'" from the <forward> element of the network, along with
the
"<interface dev='eth0'/>" subelement. This won't change
operation at
all, it will just make things slightly less confusing and misleading.
0.0.0.0/0 reject-with icmp-port-unreachable
When i try to setup a network-filter for a VM (a modified version of
http://libvirt.org/formatnwfilter.html last example):
<filter name='server-x' chain='root'>
<filterref filter='clean-traffic'/>
<rule action='accept' direction='in' priority='500'>
<all state='ESTABLISHED'/>
</rule>
<rule action='accept' direction='out' priority='500'>
<all state='ESTABLISHED,RELATED'/>
</rule>
<rule action='accept' direction='in' priority='500'>
<tcp state='NEW' dstportstart='22'/>
</rule>
<rule action='accept' direction='out' priority='500'>
<all state='NEW'/>
</rule>
<rule action='drop' direction='inout' priority='500'>
<all/>
</rule>
</filter>
and adding the filter to my interface-definition of a VM using the following syntax:
<filterref filter='server-x'>
<parameter name='IP' value='1.2.3.70'/>
</filterref>
additional iptable-rules are getting created. The problematic rule seems to be the
following:
-A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX
which should trigger the following rules:
-A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT
-A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT
But this actually never happens. The FO-vnetX Chain never sees any packets and my syslog
says:
xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for
non-bridged traffic is not supported anymore
That somehow sounded familiar, so I looked it up in the git history and
found this:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=65fb9d49cc9caae210977934...
That patch was included in libvirt-1.0.2, just about a year ago.
Am i doing something wrong?
YOu need to upgrade your libvirt to at least 1.0.2 (preferably newer).
7:31 a.m.
On Di, 2014-02-18 at 12:03 +0200, Laine Stump wrote:
You *really* should upgrade to a newer libvirt.
I know that version 0.9.8 is very old. But to be honest i tried to avoid upgrading and
compiling a newer version since i don't know if it has any effects on running VMs (but
i haven't checked this yet).
Its a production server and i did not want to interrupt any services running on these VMs.
However, i'm afraid that upgrading might be the only option if i want to avoid setting
up iptables manually.
Those previous two rules are the ones added when you specify a
forward
dev. You don't need to do that - I would recommend removing the
"dev='eth0'" from the <forward> element of the network, along
with the
"<interface dev='eth0'/>" subelement. This won't change
operation at
all, it will just make things slightly less confusing and misleading.
Thank you for pointing this out. I just did that. And after a reboot everything is still
working as expected (yes. I just DID interrupt the services running in my VMs. So i guess
i could even upgrade to a newer libvirt =) )
>
> -A libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX
>
> which should trigger the following rules:
>
> -A FO-vnetX -p all -m state --state ESTABLISHED -j ACCEPT
> -A FO-vnetX -p tcp --dport 22 -m state --state NEW -j ACCEPT
>
> But this actually never happens. The FO-vnetX Chain never sees any packets and my
syslog says:
>
> xt_physdev: using --physdev-out in the OUTPUT, FORWARD and POSTROUTING chains for
non-bridged traffic is not supported anymore
That somehow sounded familiar, so I looked it up in the git history
and
found this:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=65fb9d49cc9caae210977934...
That patch was included in libvirt-1.0.2, just about a year ago.
After reading that i remove the following iptables rule:
iptables -D libvirt-out -m physdev --physdev-out vnetX -g FO-vnetX
and manually added this rule: (the patch said that adding an extra argument
(--physdev-is-bridged) is needed for rules like this):
iptables -A libvirt-out -m physdev --physdev-is-bridged --physdev-out vnetX -g FO-vnetX
Indeed this prevents my syslog from being spammed with the mentioned warning. However,
this did not fix the problem. This rule never matches anything, and thus the FO-vnetX
Chain never sees any packets. Using a testing rule like:
iptables -A libvirt-out -d 1.2.3.70 -g FO-vnetX
made everything work as expected. Well, this is definitely not the way it is expected to
work since it does not do any "bridge port" matching. This all makes me think
its not a libvirt specific problem and updating to a newer version will not fix my
problem. Maybe there is somebody out there using a setup like mine and can show me the
rules that a getting created with a newer version of libvirt?
So long, thank you for all the usefull information!
Kind regards,
Sebastian
5:06 p.m.
On 02/18/2014 06:31 AM, h0rst wrote:
On Di, 2014-02-18 at 12:03 +0200, Laine Stump wrote:
> You *really* should upgrade to a newer libvirt.
I know that version 0.9.8 is very old. But to be honest i tried to avoid upgrading and
compiling a newer version since i don't know if it has any effects on running VMs (but
i haven't checked this yet).
There should be no problem upgrading to a newer libvirt. We take great
pains to ensure that a newer version of libvirt can be reloaded and
gracefully understand the XML recorded by older versions, with no loss
to running VMs. While there have been bugs on this front, they get
caught and patched quickly, so by updating to something like the latest
Fedora stable build (currently 1.1.3.3), you are even more likely to
avoid these sorts of problems when compared to upgrading all the way to
the master branch of libvirt.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
9:47 a.m.
On Di, 2014-02-18 at 16:06 -0700, Eric Blake wrote:
There should be no problem upgrading to a newer libvirt. We take
great
pains to ensure that a newer version of libvirt can be reloaded and
gracefully understand the XML recorded by older versions, with no loss
to running VMs. While there have been bugs on this front, they get
caught and patched quickly, so by updating to something like the latest
Fedora stable build (currently 1.1.3.3), you are even more likely to
avoid these sorts of problems when compared to upgrading all the way to
the master branch of libvirt.
After i forced myself to upgrade and compile a newer version of libvirt,
(to be precise, i wanted to upgrade to the latest libvirt-1.2.1.tar.gz)
i ran into some problems. I compiled libvirt with the following options:
As a reminder, i'm running ""good"" old Ubuntu 12.04:
./configure --with-lxc --with-storage-lvm --prefix=/usr --localstatedir=/var
--sysconfdir=/etc
Compiling and installing worked perfectly after installing all missing
dependencies. At first everything looked fine and all created networks
and domains where still running. To be sure everything would survive a
system restart, i rebooted. Thats when everything (or something) went
wrong. After starting libvirtd i got the following error:
>> error: Failed to start network hetzner-subnet-v4
>> error: unsupported configuration: Publicly routable address 1.2.3.65 is
prohibited.
>> The version of dnsmasq on this host (2.59) doesn't support the bind-dynamic
option
>> or use SO_BINDTODEVICE on listening sockets, one of which is required for safe
>> operation on a publicly routable subnet (see CVE-2012-3411). You must either
upgrade
>> dnsmasq, or use a private/local subnet range for this network (as described in
RFC1918/RFC3484/RFC4193).
Since no VM was running at this point (because of the missing networks),
i decided to quickly update to a newer version of DNSMASQ (2.68) and
installed this to "/usr/local/sbin" and linked it to
"/usr/sbin/dnsmasq"
after removing the distribution specific packages. When i tried to start
the hetzner-subnet-v4 network, i got the following error:
>> 2014-02-19 14:11:58.636+0000: 7075: error :
virCommandWait:2376 : internal error: Child process (LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root USER=root
LOGNAME=root /usr/sbin/dnsmasq --version) unexpected exit status 1: libvirt: error :
cannot execute binary /usr/sbin/dnsmasq: Permission denied
>> 2014-02-19 14:11:58.636+0000: 7075: error :
dnsmasqCapsRefreshInternal:747 : failed to run '/usr/sbin/dnsmasq --version': :
Success
>> error: Failed to start network hetzner-subnet-v4
>> error: failed to run '/usr/sbin/dnsmasq --version': : Success
However, running dnsmasq manually worked. Since that was the moment the first
phonecalls started because users could not access their services on the VMs
i quickly reverted everything to its previous state to get everything up and
running again. I dont have any testing server, so i could not play around with
it anymore (better to say not right now. I might have to wait until everyone is
sleeping ;)). Does libvirt has any problems when accessing a softlink instead
of a binary? Poorly that possibility came into my mind after reverting back to
its original state!
I'm really sorry to spam you guys with all my problems ;)
Kind regards,
Sebastian
10:26 a.m.
On 02/19/2014 08:47 AM, h0rst wrote:
>>> 2014-02-19 14:11:58.636+0000: 7075: error : virCommandWait:2376 : internal
error: Child process (LC_ALL=C
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root USER=root
LOGNAME=root /usr/sbin/dnsmasq --version) unexpected exit status 1: libvirt: error :
cannot execute binary /usr/sbin/dnsmasq: Permission denied
You'll need to figure out where the permission denied is coming from;
perhaps the problem is that your symlink requires resolution to a
directory that is not readable bu the
>>> 2014-02-19 14:11:58.636+0000: 7075: error : dnsmasqCapsRefreshInternal:747 :
failed to run '/usr/sbin/dnsmasq --version': : Success
>>> error: Failed to start network hetzner-subnet-v4
>>> error: failed to run '/usr/sbin/dnsmasq --version': : Success
These remaining error messages look awkward; we probably have issues in
the code that overwrites the useful message when it shouldn't. But
that's just cosmetic, compared to getting the real root cause of why
libvirtd can't execute dnsmasq.
However, running dnsmasq manually worked. Since that was the moment the first
phonecalls started because users could not access their services on the VMs
i quickly reverted everything to its previous state to get everything up and
running again. I dont have any testing server, so i could not play around with
it anymore (better to say not right now. I might have to wait until everyone is
sleeping ;)). Does libvirt has any problems when accessing a softlink instead
of a binary? Poorly that possibility came into my mind after reverting back to
its original state!
Libvirt shouldn't have any problems following a symlink, unless the
symlink resolves to a path that doesn't have proper permissions. Can
you paste actual terminal transcripts proving that you can manually
execute /usr/sbin/dnsmasq --version?
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
11:43 a.m.
On Mi, 2014-02-19 at 09:26 -0700, Eric Blake wrote:
Libvirt shouldn't have any problems following a symlink, unless
the
symlink resolves to a path that doesn't have proper permissions. Can
you paste actual terminal transcripts proving that you can manually
execute /usr/sbin/dnsmasq --version?
Yes. Sorry. I forgot to include those calls proving that in manually worked.
user@hv:~$ ls -la /usr/sbin/dnsmasq
lrwxrwxrwx 1 root root 23 Feb 19 18:26 /usr/sbin/dnsmasq -> /usr/local/sbin/dnsmasq
user@hv:~$ ls -la /usr/local/sbin/dnsmasq
-rwxr-xr-x 1 root root 302941 Feb 19 18:25 /usr/local/sbin/dnsmasq
As unprivileged user:
user@hv:~$ /usr/sbin/dnsmasq --version
Dnsmasq version 2.68 Copyright (c) 2000-2013 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP
no-conntrack ipset auth
This software comes with ABSOLUTELY NO WARRANTY.
Dnsmasq is free software, and you are welcome to redistribute it
under the terms of the GNU General Public License, version 2 or 3.
As root:
root@hv ~ # /usr/sbin/dnsmasq --version
Dnsmasq version 2.68 Copyright (c) 2000-2013 Simon Kelley
Compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP
no-conntrack ipset auth
This software comes with ABSOLUTELY NO WARRANTY.
Dnsmasq is free software, and you are welcome to redistribute it
under the terms of the GNU General Public License, version 2 or 3.
I can do more test if the users aren't awake and cant complain about
non working services and VMs ;)
Kind regards,
Sebastian
3929
days inactive
3934
days old
6 comments
3 participants
participants (3)
-
Eric Blake -
h0rst -
Laine Stump