[libvirt] IPV6 and routing
As I have mentioned in other messages, I am interested in having full support for IPv6 in libvirt. To me this includes having dhcp6 for IPV6 address assignment and using RA (radvd) to establish the default route. This is what I am using on my real LANs. Before starting into adding dhcp6 support to libvirt, I wanted to see just how it works with the current software. First of all, it appears that, when nat or routed are specified for IPv4, the IPv6 is routed. If it is an isolated/private network, then it can only work with other guests on that network. The iptables and ip6tables settings corresponded and were as expected. On the virtualization host, both IPv4 and IPv6 forwarding are enabled. While I can easily do stuff like ping6 and ssh -6 from virtual guests to the virtualization host, I have been unable to do anything with external hosts ... unless I add a static route for the virtual IPv6 network on the target host back to the virtualization host. This is the only way I have gotten anything to work. To complicate things, it seem that "everything" wants the IPv6 network to have prefix=64 or things do not work correctly. The real systems use fd00:dead:beef:17::/64 for their network. The virtual networks all use fd00:face:17:xx::/64 for their networks. The network traffic on the virtualization host is forwarded to the target host ... I can see the packets with wireshark on the target host. On the target host I tried specifying a static route for network fd00:face:17::/48 ... well, that really screwed things up, resulted in some "redirects" from the virtualization host saying the that it was sent a malformed packed ... it took a reboot to clean things up. OK, so leave the fd00:face:17:6::/64 static route on the target host but subnetwork this network on the virtualization host using networks like fd00:face:17:6:8::/80 and fd00:face:6:9::/80. This works if I manually configure IPv6 on the virtual guest. Since radvd is "upset" by a non-prefix=64 network, I was not surprised when the guest's automatic IPv6 address/network was not configured. OK, what am I missing? What don't I understand? If IPv6 is going to be useful in virtualization, then there must be some "easy" way to have other systems understand that the virtualization host is acting as a router for the virtual IPv6 networks it runs. While being able to go between the virtualization hosts and the virtual guests is very useful, I do not consider this sufficient. I have googled and found some stuff as well as reading more RFCs than I wanted to but I cannot find anything to address this issue. IIRC, I did find something under a libvirt document that indicates "routed" will be used for some kind of subnetworking. Does libvirt need an IPv6 "NAT" to make this work? Comments? Suggestions? Gene
On Sat, 6 Oct 2012, Gene Czarcinski wrote:
OK, what am I missing? What don't I understand?
If IPv6 is going to be useful in virtualization, then there must be some "easy" way to have other systems understand that the virtualization host is acting as a router for the virtual IPv6 networks it runs. While being able to go between the virtualization hosts and the virtual guests is very useful, I do not consider this sufficient.
We programatically, on a per VM basis, set up our ebtables and iptables rules at pmman.com (thus my 'ROADMAP' question earlier this week). Under RHEL 6's (and thus CentOS') KVM and libvirtd stock setup, there was a built-in filter as provided by libvirtd install -- as I recall: a 'clean-traffic' filter -- that we had to amend out, compared to prior xen setups under the earlier RHEL variant Have you dumped and examined the running rules affecting IPv6 traffic? -- Russ herrold
On 10/06/2012 05:29 PM, R P Herrold wrote:
On Sat, 6 Oct 2012, Gene Czarcinski wrote:
OK, what am I missing? What don't I understand?
If IPv6 is going to be useful in virtualization, then there must be some "easy" way to have other systems understand that the virtualization host is acting as a router for the virtual IPv6 networks it runs. While being able to go between the virtualization hosts and the virtual guests is very useful, I do not consider this sufficient.
We programatically, on a per VM basis, set up our ebtables and iptables rules at pmman.com (thus my 'ROADMAP' question earlier this week). Under RHEL 6's (and thus CentOS') KVM and libvirtd stock setup, there was a built-in filter as provided by libvirtd install -- as I recall: a 'clean-traffic' filter -- that we had to amend out, compared to prior xen setups under the earlier RHEL variant
Have you dumped and examined the running rules affecting IPv6 traffic?
-- Russ herrold
The ip6tables rules look fine. The problem is not that the packets are not sent to the target .... they are .. I ran wireshark on the target's NIC. The problem is getting the response back to the virtualization host. Shortly after I wrote my message I "discovered" something called "neighbor discovery proxy" and two attempts at implementing it: npd6 and ndppd. This is the RFC: http://tools.ietf.org/html/rfc4389 and here is a short description from npd6:
If you have a Linux gateway router terminating your ISP feed supporting IPv6, this may be just what you need. To summarise the problem it solves: your ISP has given you an /64 (or some other size) IPv6 prefix, with the last 64 bits (or whatever) entirely for your own use on a private-side of the network. The IPv6 addresses in use by your own devices may well not even be known to you – it’s possible that you use DHCP6 to statically pre-allocate them (yuck!) or more likely you are using /radvd/ on the gateway to advertise the ISP-supplied IPv6 prefix and let the devices themselves choose what they wish to tag on to that. It may be vaguely predictable (based upon the device’s Ethernet MAC address) or totally unpredictable (as per the Windows 7 box I looked at the other day!) ...
And to do this today you need to /statically pre-configure/ that full address into the Linux system. And if it changes, you need to change it. And if a new one appears, you need to ad it. And so on. Oh, and to add insult to injury, you cannot even display a list of which ones you have already configured in the system!!
And thus I offer npd6 as a solution: it runs on the gateway, and requires little configuration. You tell it your prefix and which is the ISP’s interface. There are a few optional knobs and levers. Then it runs and automatically responds to /any/ Neighbor Solicitation received from the ISP for a device with your prefix.
This "sounds" like it may be a solution and I have started some testing to see if and how they work. Gene
On 10/07/2012 06:28 AM, Gene Czarcinski wrote:
On 10/06/2012 05:29 PM, R P Herrold wrote:
On Sat, 6 Oct 2012, Gene Czarcinski wrote:
OK, what am I missing? What don't I understand?
If IPv6 is going to be useful in virtualization, then there must be some "easy" way to have other systems understand that the virtualization host is acting as a router for the virtual IPv6 networks it runs. While being able to go between the virtualization hosts and the virtual guests is very useful, I do not consider this sufficient.
We programatically, on a per VM basis, set up our ebtables and iptables rules at pmman.com (thus my 'ROADMAP' question earlier this week). Under RHEL 6's (and thus CentOS') KVM and libvirtd stock setup, there was a built-in filter as provided by libvirtd install -- as I recall: a 'clean-traffic' filter -- that we had to amend out, compared to prior xen setups under the earlier RHEL variant
Have you dumped and examined the running rules affecting IPv6 traffic?
-- Russ herrold
The ip6tables rules look fine. The problem is not that the packets are not sent to the target .... they are .. I ran wireshark on the target's NIC. The problem is getting the response back to the virtualization host.
Shortly after I wrote my message I "discovered" something called "neighbor discovery proxy" and two attempts at implementing it: npd6 and ndppd. This is the RFC: http://tools.ietf.org/html/rfc4389 and here is a short description from npd6:
If you have a Linux gateway router terminating your ISP feed supporting IPv6, this may be just what you need. To summarise the problem it solves: your ISP has given you an /64 (or some other size) IPv6 prefix, with the last 64 bits (or whatever) entirely for your own use on a private-side of the network. The IPv6 addresses in use by your own devices may well not even be known to you – it’s possible that you use DHCP6 to statically pre-allocate them (yuck!) or more likely you are using /radvd/ on the gateway to advertise the ISP-supplied IPv6 prefix and let the devices themselves choose what they wish to tag on to that. It may be vaguely predictable (based upon the device’s Ethernet MAC address) or totally unpredictable (as per the Windows 7 box I looked at the other day!) ...
And to do this today you need to /statically pre-configure/ that full address into the Linux system. And if it changes, you need to change it. And if a new one appears, you need to ad it. And so on. Oh, and to add insult to injury, you cannot even display a list of which ones you have already configured in the system!!
And thus I offer npd6 as a solution: it runs on the gateway, and requires little configuration. You tell it your prefix and which is the ISP’s interface. There are a few optional knobs and levers. Then it runs and automatically responds to /any/ Neighbor Solicitation received from the ISP for a device with your prefix.
This "sounds" like it may be a solution and I have started some testing to see if and how they work.
The more I look into this, the more I do not like the answer but it is what it is. First of all, ND-proxy is not the answer. Even if I could get it to work on one network, it would not "turn the corner" to another network. Unfortunately, radvd is not the answer either. Although I can configure radvd on the virtualization host, the router advertisements it does are ignored (by RFC requirement) by the system having the default route. As far as I can see, the only thing that a RA is good for is establishing the default route. As things currently exist, I have come to the conclusion that there are two answers that will work. 1. Manually configure the IPv6 address/network on the virtualization host. On the default route system, add a static route to the virtualization host. Things will then work. 2. It hurts, don't do it: use a bridge in the virtualization host for IPv6 virtual networks which need to access external systems. BTW, this is also needed for support of IPv4 routed networks. This can be made a little easier by using 172.16.<nn>.0/24 networks and adding a single static route for 172.16.0.0/16. But for IPv6, there is just a lot of stuff that only works for networks with a 64 prefix. When other prefixes (greater or lesser) are specified, things get "strange." By RFC definition, RA only works with prefix=64. There is a form of nat for IPv6 in the works ("prefix translation") in ip6tables. But, time will tell on how useful that will be. Gene
On 10/07/2012 12:10 PM, Gene Czarcinski wrote:
1. Manually configure the IPv6 address/network on the virtualization host. On the default route system, add a static route to the virtualization host. Things will then work.
2. It hurts, don't do it: use a bridge in the virtualization host for IPv6 virtual networks which need to access external systems.
BTW, this is also needed for support of IPv4 routed networks. This can be made a little easier by using 172.16.<nn>.0/24 networks and adding a single static route for 172.16.0.0/16. Giving this some additional thought, there is a small alternative which offers some advantages: use dhcp "nailed" host ip addresses. while this works for both IPv4 and IPv6, it is more important for IPv6 because all IPv6 virtual networksare "routed."
I have tested this using dnsmasq as the dhcpd6 and dns services provider and there are a number of options for relating a specific host/network-interface and an ip address. In addition to dnsmasq or dhcpd6, you will need to add a static route for that virtual network to the virtualization host on the system which is pointed to by the default route. There are some documents out there which imply to what I have concluded above but things are left still a little vague. Examples of what is out there are: http://berrange.com/posts/2009/12/13/routed-subnets-without-nat-for-libvirt-... http://blog.gadi.cc/routed-subnet-libvirt-kvm/ But, I do not see (or have not found yet) something under http://wiki.libvirt.org I am not sure where such information belongs but I do believe that it is needed. Examples of where it could be placed are: http://wiki.libvirt.org/page/VirtualNetworking#Routed_mode_2 http://wiki.libvirt.org/page/Networking http://libvirt.org/formatnetwork.html#examplesRoute Gene
On 10/06/2012 10:34 AM, Gene Czarcinski wrote:
As I have mentioned in other messages, I am interested in having full support for IPv6 in libvirt. To me this includes having dhcp6 for IPV6 address assignment and using RA (radvd) to establish the default route. This is what I am using on my real LANs.
Before starting into adding dhcp6 support to libvirt, I wanted to see just how it works with the current software. First of all, it appears that, when nat or routed are specified for IPv4, the IPv6 is routed.
At the time I added IPv6 support, there was no NAT for IPv6, and even now I don't know if it's clear that it will be accepted (truthfully I haven't been following it). So the choice was to either not allow IPv6 at all on NATed networks, or to just route the IPv6 in these cases. We chose the latter.
If it is an isolated/private network, then it can only work with other guests on that network. The iptables and ip6tables settings corresponded and were as expected. On the virtualization host, both IPv4 and IPv6 forwarding are enabled.
While I can easily do stuff like ping6 and ssh -6 from virtual guests to the virtualization host, I have been unable to do anything with external hosts ... unless I add a static route for the virtual IPv6 network on the target host back to the virtualization host.
.. or on the machine acting as the default route. Or if you advertise a route to the virtual network's subnet with a routing protocol.
This is the only way I have gotten anything to work. To complicate things, it seem that "everything" wants the IPv6 network to have prefix=64 or things do not work correctly.
Not totally. The one thing I'm aware of that requires a prefix=64 network is that IPv6 autoconf *by definition* will only work on networks with a 64 bit prefix. I'm not really sure why this is so (I might have read the reason a long time ago, but if so I've long forgotten what it was), but it is, and there's nothing to be done about it :-/
The real systems use fd00:dead:beef:17::/64 for their network. The virtual networks all use fd00:face:17:xx::/64 for their networks.
The network traffic on the virtualization host is forwarded to the target host ... I can see the packets with wireshark on the target host.
On the target host I tried specifying a static route for network fd00:face:17::/48 ... well, that really screwed things up, resulted in some "redirects" from the virtualization host saying the that it was sent a malformed packed ... it took a reboot to clean things up.
OK, so leave the fd00:face:17:6::/64 static route on the target host but subnetwork this network on the virtualization host using networks like fd00:face:17:6:8::/80 and fd00:face:6:9::/80. This works if I manually configure IPv6 on the virtual guest. Since radvd is "upset" by a non-prefix=64 network, I was not surprised when the guest's automatic IPv6 address/network was not configured.
OK, what am I missing? What don't I understand?
It really is unfortunate that autoconf only works for prefix=64, as I can easily imagine a case where the entire address space available to the admin would be prefix=64, so they would need to allow only prefix=(64+n) for each virtual network.
If IPv6 is going to be useful in virtualization, then there must be some "easy" way to have other systems understand that the virtualization host is acting as a router for the virtual IPv6 networks it runs.
That's what routing protocols do, and configuration of a routing protocol daemon to advertise out to the physical network is outside the scope of libvirt. This is nothing unique to IPv6 - people using IPv4 networks in <forward mode='route'/> mode have the same problem all the time. It's not even unique to virtualization - you would have the same problem if you created a physical subnet behind one of your hosts.
While being able to go between the virtualization hosts and the virtual guests is very useful, I do not consider this sufficient.
I have googled and found some stuff as well as reading more RFCs than I wanted to but I cannot find anything to address this issue.
IIRC, I did find something under a libvirt document that indicates "routed" will be used for some kind of subnetworking.
Sigh. That one comment has done more to make me feel old than anything else in a while :-/ Some history - The last time I needed to run a routing protocol was > 10 years ago, and back then I was running NetBSD and had just spent a bunch of time working on a router product based on BSD4.3; at that time, the standard program to use for RIP (the "routing information protocol" was called "routed" (i.e. the "route daemon"). There was another project called "gated" (gateway daemon) that implemented OSPF, EGP, and BGP, and that was about it. Cut to today (well, Sunday anyway - I got a bit distracted by work and didn't complete this reply in a timely fashion) and me seeing your comment about "routed" in a manner that implied "what is this?" I first naively ran "yum provides */routed" to see which package contained routed ("Surely it must still be the go-to daemon for routing information..."), and am told that no packages have it. So I search around and find no mention of it on Linux, then finally realize that it's under the BSD license, and there is no exact GPLed equivalent as there is for most standard system utilities. More searching and I see that the gated project is apparently completely defunct, that another project called "zebra" had taken up those reins, and it too is now defunct, but a fork of zebra called quagga is now around, and has a package in Fedora. So, the result of this trip: what seems to be the current in-use routing protocol daemon on Linux systems is 2 generations removed from what I last used. Now GET OFF MY LAWN!!!!!! :-) Anyway, I guess try quagga (unless someone else more in the know has a better suggestion). You'll need to figure out what routing protocol is in use on your network (if any) and go along with that. Or you can just add a static route on the default router for the network connected to your host's physical adapter.
Does libvirt need an IPv6 "NAT" to make this work?
One really nice thing about IPv6 is the elimination of NAT. One of the biggest headaches about IPv6 for some network admins is the lack of NAT. :-/ I personally would love to see NAT never exist in IPv6, but it appears that won't happen. If the proposed NAT for IPv6 has "landed", then we could certainly entertain patches to support it for <forward mode='nat'/>, but only if it is standards track and stable. (I seriously don't know how far along it is)
Comments? Suggestions?
In general, you may be helped by running a routing protocol daemon on your host (as long as your routing infrastructure is also running a routing protocol).
participants (3)
-
Gene Czarcinski -
Laine Stump -
R P Herrold