[libvirt-users] Recommended change for the networking page in wiki
Hello all, I was told on IRC that I should come here to discuss a recommended change on the networking page in the wiki. If you take a look at the "Basic Script" shown here: http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections It does two things: 1) Create a DNAT rule in the NAT table of IPTABLES 2) Create a FORWARD rule in the FILTER table of IPTABLES The FORWARD rule is set up as it ought to be, however, the DNAT rule has some unintended consequences. I set up a DNAT on port 80, and suddenly, I couldn't access out on port 80 anymore from my guest machine. However, if I changed the destination address from "anywhere" to the IP of the host machine, the problem resolved. So I change the script to as follows. (Changes are highlighted. For some reason the original script didn't work using /bin/sh, but it did with /bin/bash, so I changed that too). *#!/bin/bash* # used some from advanced script to have multiple ports: use an equal number of guest and host ports Guest_name=xxxxxxx Guest_ipaddr=xxx.xxx.xxx.xx *Host_ipaddr=xxx.xxx.xxx.xx* Host_port=( '80' '443' ) Guest_port=( '80' '443' ) length=$(( ${#Host_port[@]} - 1 )) if [ "${1}" = "${Guest_name}" ]; then if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then for i in `seq 0 $length`; do iptables -t nat -D PREROUTING *-d ${Host_ipaddr}* -p tcp --dport ${Host_port[$i]} -j DNAT --to ${Guest_ipaddr}:${Guest_port[$i]} iptables -D FORWARD -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW -m tcp --dport ${Guest_port[$i]} -j ACCEPT done fi if [ "${2}" = "start" -o "${2}" = "reconnect" ]; then for i in `seq 0 $length`; do iptables -t nat -A PREROUTING *-d ${Host_ipaddr}* -p tcp --dport ${Host_port[$i]} -j DNAT --to ${Guest_ipaddr}:${Guest_port[$i]} iptables -I FORWARD 4 -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW -m tcp --dport ${Guest_port[$i]} -j ACCEPT done fi fi Lastly, I should note that I am using Ubuntu 14.04, both for the host and guest. I'm also curious as to why this is considered a hack method. It states in the wiki that "This method is a hack", but it doesn't express why. Many VM Servers have similar features. I know Virtual Box does, I use the same feature there. It may not be how I would set up a production server, but doesn't make it a hack. Thanks, BJ
On 06/08/2015 03:24 PM, BJ wrote:
Hello all,
I was told on IRC that I should come here to discuss a recommended change on the networking page in the wiki.
If you take a look at the "Basic Script" shown here: http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections
It does two things: 1) Create a DNAT rule in the NAT table of IPTABLES 2) Create a FORWARD rule in the FILTER table of IPTABLES
The FORWARD rule is set up as it ought to be, however, the DNAT rule has some unintended consequences. I set up a DNAT on port 80, and suddenly, I couldn't access out on port 80 anymore from my guest machine.
I'm not seeing the same results here. outbound port 80 is still allowed from my host once the rule is in place. Of course incoming port 80 isn't being redirected to the guest either, so I think I have bigger problems. (this is likely due to the fact that I haven't tried the script again since 2013 or so, and a lot has likely changed with the iptables rules added to a Fedora system since then)
However, if I changed the destination address from "anywhere" to the IP of the host machine, the problem resolved. So I change the script to as follows. (Changes are highlighted. For some reason the original script didn't work using /bin/sh, but it did with /bin/bash, so I changed that too).
I don't know for sure, but my guess is that this line: length=$(( ${#Host_port[@]} - 1 )) which was added by vgerris in order to support forwarding of multiple ports, could be what's causing the incompatibility (that wasn't in the original, simpler version of the script, written by me.)
*#!/bin/bash* # used some from advanced script to have multiple ports: use an equal number of guest and host ports
Guest_name=xxxxxxx Guest_ipaddr=xxx.xxx.xxx.xx *Host_ipaddr=xxx.xxx.xxx.xx* Host_port=( '80' '443' ) Guest_port=( '80' '443' ) length=$(( ${#Host_port[@]} - 1 )) if [ "${1}" = "${Guest_name}" ]; then if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then for i in `seq 0 $length`; do iptables -t nat -D PREROUTING *-d ${Host_ipaddr}* -p tcp --dport ${Host_port[$i]} -j DNAT --to ${Guest_ipaddr}:${Guest_port[$i]} iptables -D FORWARD -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW -m tcp --dport ${Guest_port[$i]} -j ACCEPT done fi if [ "${2}" = "start" -o "${2}" = "reconnect" ]; then for i in `seq 0 $length`; do iptables -t nat -A PREROUTING *-d ${Host_ipaddr}* -p tcp --dport ${Host_port[$i]} -j DNAT --to ${Guest_ipaddr}:${Guest_port[$i]} iptables -I FORWARD 4 -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW -m tcp --dport ${Guest_port[$i]} -j ACCEPT done fi fi
Lastly, I should note that I am using Ubuntu 14.04, both for the host and guest.
I'm also curious as to why this is considered a hack method. It states in the wiki that "This method is a hack", but it doesn't express why.
I consider it a hack because: 1) It requires the IP address of the guest to be known before the guest is started, so either you need to guess the guest's IP (if the guest is getting its IP address via dhcp) or configured the guest IP address in multiple places. 2) It requires using a hook script, which "taints" *all* guests on this machine, rendering them "unsupported" in the eyes of some (in practice, once you see what is causing the tainting it's not such a big deal, but it does mean that an external script gets control with elevated privileges). 3) The "configuration" is just the contents of the shell script, so it can't be supported by any higher level management application - you will always need to directly modify this single shell script file. 4) If you change the configuration for a guest while that guest is running, any forwarding rules that were a part of the old config but not in the new config will be orphaned in your iptables nat table until *all* rules are flushed. 5) If anything else messes with the iptables rules, these port forwarding rules are broken. Especially on older versions of libvirt (0.9.12 and older, which is around the time that entry was written), just restarting libvirtd would break the port forwarding. 6) As both of us have experienced here, interaction with the iptables rules of the underlying system can easily result in it not working at all. The proper way to support this would be XML added to the domain configuration, something like: <interface type='network'> <source network='default'/> <model type='virtio'/ <mac address='52:54:00:11:22:33'/> <redirect protocol='tcp'> <public address='1.2.3.4' port='2200'/> <private port='22'/> </redirect> ... </interface> (or something like that). libvirt would then auto-determine the IP address of the interface and add the necessary iptables rules (or perhaps an rinetd config line item, which may be less prone to sabotage by distro-specific default iptables setups). The inverse would be automatically done when the domain was shutdown.
Many VM Servers have similar features. I know Virtual Box does, I use the same feature there. It may not be how I would set up a production server, but doesn't make it a hack.
It's not the port forwarding itself that I consider to be a hack, it's the method that's used by that script to accomplish it. (And since I was the original author of the script and the entry in the wiki, I think I am allowed to criticize it :-)
Laine, you make a fair point about the hack. You are right, until this is natively supported by libvirt, I suppose it is a hack. I guess I've just grown very accustomed to hacks in linux. :-) That said, it seems like I've found the right person to look at the alterations I've made to the script. If you think I should update the wiki page to show the changes I've made to the script, I'll go ahead and do it. If you'd rather maintain it, I'm happy to let you do so. The change does seem logical to me. The original DNAT rule that it set up on my ubuntu machine basically said: "route port 80 traffic from anywhere to anywhere to the guest machine", which with that in place prevented me from using apt-get on the guest machine. After I made the change, everything seemed to work fine (port 80 traffic to the machine as well as apt-get). Of course, I'm only testing this on Ubuntu, and that's probably my biggest blind spot here. I have no idea if other distros might be slightly different. Thanks, BJ On Mon, Jun 8, 2015 at 2:43 PM, Laine Stump <laine@laine.org> wrote:
On 06/08/2015 03:24 PM, BJ wrote:
Hello all,
I was told on IRC that I should come here to discuss a recommended change on the networking page in the wiki.
If you take a look at the "Basic Script" shown here: http://wiki.libvirt.org/page/Networking#Forwarding_Incoming_Connections
It does two things: 1) Create a DNAT rule in the NAT table of IPTABLES 2) Create a FORWARD rule in the FILTER table of IPTABLES
The FORWARD rule is set up as it ought to be, however, the DNAT rule has some unintended consequences. I set up a DNAT on port 80, and suddenly, I couldn't access out on port 80 anymore from my guest machine.
I'm not seeing the same results here. outbound port 80 is still allowed from my host once the rule is in place. Of course incoming port 80 isn't being redirected to the guest either, so I think I have bigger problems. (this is likely due to the fact that I haven't tried the script again since 2013 or so, and a lot has likely changed with the iptables rules added to a Fedora system since then)
However, if I changed the destination address from "anywhere" to the IP of the host machine, the problem resolved. So I change the script to as follows. (Changes are highlighted. For some reason the original script didn't work using /bin/sh, but it did with /bin/bash, so I changed that too).
I don't know for sure, but my guess is that this line:
length=$(( ${#Host_port[@]} - 1 ))
which was added by vgerris in order to support forwarding of multiple ports, could be what's causing the incompatibility (that wasn't in the original, simpler version of the script, written by me.)
*#!/bin/bash* # used some from advanced script to have multiple ports: use an equal number of guest and host ports
Guest_name=xxxxxxx Guest_ipaddr=xxx.xxx.xxx.xx *Host_ipaddr=xxx.xxx.xxx.xx* Host_port=( '80' '443' ) Guest_port=( '80' '443' ) length=$(( ${#Host_port[@]} - 1 )) if [ "${1}" = "${Guest_name}" ]; then if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then for i in `seq 0 $length`; do iptables -t nat -D PREROUTING *-d ${Host_ipaddr}* -p tcp --dport ${Host_port[$i]} -j DNAT --to ${Guest_ipaddr}:${Guest_port[$i]} iptables -D FORWARD -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW -m tcp --dport ${Guest_port[$i]} -j ACCEPT done fi if [ "${2}" = "start" -o "${2}" = "reconnect" ]; then for i in `seq 0 $length`; do iptables -t nat -A PREROUTING *-d ${Host_ipaddr}* -p tcp --dport ${Host_port[$i]} -j DNAT --to ${Guest_ipaddr}:${Guest_port[$i]} iptables -I FORWARD 4 -d ${Guest_ipaddr}/32 -p tcp -m state --state NEW -m tcp --dport ${Guest_port[$i]} -j ACCEPT done fi fi
Lastly, I should note that I am using Ubuntu 14.04, both for the host and guest.
I'm also curious as to why this is considered a hack method. It states in the wiki that "This method is a hack", but it doesn't express why.
I consider it a hack because:
1) It requires the IP address of the guest to be known before the guest is started, so either you need to guess the guest's IP (if the guest is getting its IP address via dhcp) or configured the guest IP address in multiple places.
2) It requires using a hook script, which "taints" *all* guests on this machine, rendering them "unsupported" in the eyes of some (in practice, once you see what is causing the tainting it's not such a big deal, but it does mean that an external script gets control with elevated privileges).
3) The "configuration" is just the contents of the shell script, so it can't be supported by any higher level management application - you will always need to directly modify this single shell script file.
4) If you change the configuration for a guest while that guest is running, any forwarding rules that were a part of the old config but not in the new config will be orphaned in your iptables nat table until *all* rules are flushed.
5) If anything else messes with the iptables rules, these port forwarding rules are broken. Especially on older versions of libvirt (0.9.12 and older, which is around the time that entry was written), just restarting libvirtd would break the port forwarding.
6) As both of us have experienced here, interaction with the iptables rules of the underlying system can easily result in it not working at all.
The proper way to support this would be XML added to the domain configuration, something like:
<interface type='network'> <source network='default'/> <model type='virtio'/ <mac address='52:54:00:11:22:33'/> <redirect protocol='tcp'> <public address='1.2.3.4' port='2200'/> <private port='22'/> </redirect> ... </interface>
(or something like that). libvirt would then auto-determine the IP address of the interface and add the necessary iptables rules (or perhaps an rinetd config line item, which may be less prone to sabotage by distro-specific default iptables setups). The inverse would be automatically done when the domain was shutdown.
Many VM Servers have similar features. I know Virtual Box does, I use the same feature there. It may not be how I would set up a production server, but doesn't make it a hack.
It's not the port forwarding itself that I consider to be a hack, it's the method that's used by that script to accomplish it. (And since I was the original author of the script and the entry in the wiki, I think I am allowed to criticize it :-)
On 06/08/2015 02:43 PM, Laine Stump wrote:
However, if I changed the destination address from "anywhere" to the IP of the host machine, the problem resolved. So I change the script to as follows. (Changes are highlighted. For some reason the original script didn't work using /bin/sh, but it did with /bin/bash, so I changed that too).
I don't know for sure, but my guess is that this line:
length=$(( ${#Host_port[@]} - 1 ))
Correct - that line is a bashism, and is not portable when /bin/sh is dash.
which was added by vgerris in order to support forwarding of multiple ports, could be what's causing the incompatibility (that wasn't in the original, simpler version of the script, written by me.)
*#!/bin/bash* # used some from advanced script to have multiple ports: use an equal number of guest and host ports
Guest_name=xxxxxxx Guest_ipaddr=xxx.xxx.xxx.xx *Host_ipaddr=xxx.xxx.xxx.xx* Host_port=( '80' '443' ) Guest_port=( '80' '443' )
In fact, these two lines are also bashisms. All the more reason to require bash.
length=$(( ${#Host_port[@]} - 1 )) if [ "${1}" = "${Guest_name}" ]; then if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then
test ... -o ... (also spelled [ ... -o ... ]) is not portable, even on bash. There are some expressions that are completely ambiguous on how to be parsed, when -o is in the mix. It is better to spell it: [ ... ] || [ ... ] (that is, use the shell's || instead of test's -o to do the conjunction).
for i in `seq 0 $length`; do
seq is not portable outside of GNU/Linux systems.
Lastly, I should note that I am using Ubuntu 14.04, both for the host and guest.
Yep, that's a system that uses dash for /bin/sh.
I'm also curious as to why this is considered a hack method. It states in the wiki that "This method is a hack", but it doesn't express why.
I consider it a hack because:
1) It requires the IP address of the guest to be known before the guest is started, so either you need to guess the guest's IP (if the guest is getting its IP address via dhcp) or configured the guest IP address in multiple places.
Although recent work has been made to get libvirt to add an API that queries the guest for its IP address, once the guest is running. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
So should I go ahead and make the changes to the script that I suggested? Thanks, BJ On Mon, Jun 8, 2015 at 3:21 PM, Eric Blake <eblake@redhat.com> wrote:
On 06/08/2015 02:43 PM, Laine Stump wrote:
However, if I changed the destination address from "anywhere" to the IP of the host machine, the problem resolved. So I change the script to as follows. (Changes are highlighted. For some reason the original script didn't work using /bin/sh, but it did with /bin/bash, so I changed that too).
I don't know for sure, but my guess is that this line:
length=$(( ${#Host_port[@]} - 1 ))
Correct - that line is a bashism, and is not portable when /bin/sh is dash.
which was added by vgerris in order to support forwarding of multiple ports, could be what's causing the incompatibility (that wasn't in the original, simpler version of the script, written by me.)
*#!/bin/bash* # used some from advanced script to have multiple ports: use an equal number of guest and host ports
Guest_name=xxxxxxx Guest_ipaddr=xxx.xxx.xxx.xx *Host_ipaddr=xxx.xxx.xxx.xx* Host_port=( '80' '443' ) Guest_port=( '80' '443' )
In fact, these two lines are also bashisms. All the more reason to require bash.
length=$(( ${#Host_port[@]} - 1 )) if [ "${1}" = "${Guest_name}" ]; then if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then
test ... -o ... (also spelled [ ... -o ... ]) is not portable, even on bash. There are some expressions that are completely ambiguous on how to be parsed, when -o is in the mix. It is better to spell it: [ ... ] || [ ... ] (that is, use the shell's || instead of test's -o to do the conjunction).
for i in `seq 0 $length`; do
seq is not portable outside of GNU/Linux systems.
Lastly, I should note that I am using Ubuntu 14.04, both for the host and guest.
Yep, that's a system that uses dash for /bin/sh.
I'm also curious as to why this is considered a hack method. It states in the wiki that "This method is a hack", but it doesn't express why.
I consider it a hack because:
1) It requires the IP address of the guest to be known before the guest is started, so either you need to guess the guest's IP (if the guest is getting its IP address via dhcp) or configured the guest IP address in multiple places.
Although recent work has been made to get libvirt to add an API that queries the guest for its IP address, once the guest is running.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 06/12/2015 11:29 AM, BJ wrote:
So should I go ahead and make the changes to the script that I suggested?
I wanted to verify that it worked okay with your changed, but got a bit sidetracked by the fact that the redirection no longer works at all on my Fedora system as of F22. Since I don't have time to figure out what the problem is there, and you say that it works for you with this addition, I guess go ahead and make the change. You may want to look into the following though: 1) using the "network" hook instead of the "qemu" hook (the network hook didn't exist at the time this hook was written) 2) installing rinetd and having the hook script add/remove a line in rinetd.conf and signal it. This would eliminate all the potential points of failure related to doing it with iptables DNAT rules. (I've been meaning to try that out since someone suggested it a month or two ago, but it is *way* down on my list). Thanks for taking the time/effort to follow up on this!
Thanks, BJ
On Mon, Jun 8, 2015 at 3:21 PM, Eric Blake <eblake@redhat.com <mailto:eblake@redhat.com>> wrote:
On 06/08/2015 02:43 PM, Laine Stump wrote: >> However, if I changed the destination address from "anywhere" to the >> IP of the host machine, the problem resolved. So I change the script >> to as follows. (Changes are highlighted. For some reason the original >> script didn't work using /bin/sh, but it did with /bin/bash, so I >> changed that too). > > I don't know for sure, but my guess is that this line: > > length=$(( ${#Host_port[@]} - 1 ))
Correct - that line is a bashism, and is not portable when /bin/sh is dash.
> > which was added by vgerris in order to support forwarding of multiple > ports, could be what's causing the incompatibility (that wasn't in the > original, simpler version of the script, written by me.) > >> >> *#!/bin/bash* >> # used some from advanced script to have multiple ports: use an equal >> number of guest and host ports >> >> Guest_name=xxxxxxx >> Guest_ipaddr=xxx.xxx.xxx.xx >> *Host_ipaddr=xxx.xxx.xxx.xx* >> Host_port=( '80' '443' ) >> Guest_port=( '80' '443' )
In fact, these two lines are also bashisms. All the more reason to require bash.
>> length=$(( ${#Host_port[@]} - 1 )) >> if [ "${1}" = "${Guest_name}" ]; then >> if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then
test ... -o ... (also spelled [ ... -o ... ]) is not portable, even on bash. There are some expressions that are completely ambiguous on how to be parsed, when -o is in the mix. It is better to spell it: [ ... ] || [ ... ] (that is, use the shell's || instead of test's -o to do the conjunction).
>> for i in `seq 0 $length`; do
seq is not portable outside of GNU/Linux systems.
>> Lastly, I should note that I am using Ubuntu 14.04, both for the host >> and guest.
Yep, that's a system that uses dash for /bin/sh.
>> >> I'm also curious as to why this is considered a hack method. It states >> in the wiki that "This method is a hack", but it doesn't express why. > > I consider it a hack because: > > 1) It requires the IP address of the guest to be known before the guest > is started, so either you need to guess the guest's IP (if the guest is > getting its IP address via dhcp) or configured the guest IP address in > multiple places.
Although recent work has been made to get libvirt to add an API that queries the guest for its IP address, once the guest is running.
-- Eric Blake eblake redhat com +1-919-301-3266 <tel:%2B1-919-301-3266> Libvirt virtualization library http://libvirt.org
I'll go ahead and update it with my current script for now, but I'll do some research next week on those items you just mentioned, and see if I can make further improvements on the script. I'll let you know how that goes. Thanks, BJ On Fri, Jun 12, 2015 at 11:14 AM, Laine Stump <laine@laine.org> wrote:
On 06/12/2015 11:29 AM, BJ wrote:
So should I go ahead and make the changes to the script that I suggested?
I wanted to verify that it worked okay with your changed, but got a bit sidetracked by the fact that the redirection no longer works at all on my Fedora system as of F22. Since I don't have time to figure out what the problem is there, and you say that it works for you with this addition, I guess go ahead and make the change.
You may want to look into the following though:
1) using the "network" hook instead of the "qemu" hook (the network hook didn't exist at the time this hook was written)
2) installing rinetd and having the hook script add/remove a line in rinetd.conf and signal it. This would eliminate all the potential points of failure related to doing it with iptables DNAT rules. (I've been meaning to try that out since someone suggested it a month or two ago, but it is *way* down on my list).
Thanks for taking the time/effort to follow up on this!
Thanks, BJ
On Mon, Jun 8, 2015 at 3:21 PM, Eric Blake <eblake@redhat.com> wrote:
On 06/08/2015 02:43 PM, Laine Stump wrote:
However, if I changed the destination address from "anywhere" to the IP of the host machine, the problem resolved. So I change the script to as follows. (Changes are highlighted. For some reason the original script didn't work using /bin/sh, but it did with /bin/bash, so I changed that too).
I don't know for sure, but my guess is that this line:
length=$(( ${#Host_port[@]} - 1 ))
Correct - that line is a bashism, and is not portable when /bin/sh is dash.
which was added by vgerris in order to support forwarding of multiple ports, could be what's causing the incompatibility (that wasn't in the original, simpler version of the script, written by me.)
*#!/bin/bash* # used some from advanced script to have multiple ports: use an equal number of guest and host ports
Guest_name=xxxxxxx Guest_ipaddr=xxx.xxx.xxx.xx *Host_ipaddr=xxx.xxx.xxx.xx* Host_port=( '80' '443' ) Guest_port=( '80' '443' )
In fact, these two lines are also bashisms. All the more reason to require bash.
length=$(( ${#Host_port[@]} - 1 )) if [ "${1}" = "${Guest_name}" ]; then if [ "${2}" = "stopped" -o "${2}" = "reconnect" ]; then
test ... -o ... (also spelled [ ... -o ... ]) is not portable, even on bash. There are some expressions that are completely ambiguous on how to be parsed, when -o is in the mix. It is better to spell it: [ ... ] || [ ... ] (that is, use the shell's || instead of test's -o to do the conjunction).
for i in `seq 0 $length`; do
seq is not portable outside of GNU/Linux systems.
Lastly, I should note that I am using Ubuntu 14.04, both for the host and guest.
Yep, that's a system that uses dash for /bin/sh.
I'm also curious as to why this is considered a hack method. It states in the wiki that "This method is a hack", but it doesn't express why.
I consider it a hack because:
1) It requires the IP address of the guest to be known before the guest is started, so either you need to guess the guest's IP (if the guest is getting its IP address via dhcp) or configured the guest IP address in multiple places.
Although recent work has been made to get libvirt to add an API that queries the guest for its IP address, once the guest is running.
-- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (3)
-
BJ -
Eric Blake -
Laine Stump