9:34 a.m.
As I have mentioned in other messages, I am interested in having full
support for IPv6 in libvirt. To me this includes having dhcp6 for IPV6
address assignment and using RA (radvd) to establish the default route.
This is what I am using on my real LANs.
Before starting into adding dhcp6 support to libvirt, I wanted to see
just how it works with the current software. First of all, it appears
that, when nat or routed are specified for IPv4, the IPv6 is routed. If
it is an isolated/private network, then it can only work with other
guests on that network. The iptables and ip6tables settings
corresponded and were as expected. On the virtualization host, both
IPv4 and IPv6 forwarding are enabled.
While I can easily do stuff like ping6 and ssh -6 from virtual guests to
the virtualization host, I have been unable to do anything with external
hosts ... unless I add a static route for the virtual IPv6 network on
the target host back to the virtualization host. This is the only way I
have gotten anything to work. To complicate things, it seem that
"everything" wants the IPv6 network to have prefix=64 or things do not
work correctly.
The real systems use fd00:dead:beef:17::/64 for their network. The
virtual networks all use fd00:face:17:xx::/64 for their networks.
The network traffic on the virtualization host is forwarded to the
target host ... I can see the packets with wireshark on the target host.
On the target host I tried specifying a static route for network
fd00:face:17::/48 ... well, that really screwed things up, resulted in
some "redirects" from the virtualization host saying the that it was
sent a malformed packed ... it took a reboot to clean things up.
OK, so leave the fd00:face:17:6::/64 static route on the target host but
subnetwork this network on the virtualization host using networks like
fd00:face:17:6:8::/80 and fd00:face:6:9::/80. This works if I manually
configure IPv6 on the virtual guest. Since radvd is "upset" by a
non-prefix=64 network, I was not surprised when the guest's automatic
IPv6 address/network was not configured.
OK, what am I missing? What don't I understand?
If IPv6 is going to be useful in virtualization, then there must be some
"easy" way to have other systems understand that the virtualization host
is acting as a router for the virtual IPv6 networks it runs. While
being able to go between the virtualization hosts and the virtual guests
is very useful, I do not consider this sufficient.
I have googled and found some stuff as well as reading more RFCs than I
wanted to but I cannot find anything to address this issue.
IIRC, I did find something under a libvirt document that indicates
"routed" will be used for some kind of subnetworking.
Does libvirt need an IPv6 "NAT" to make this work?
Comments? Suggestions?
Gene
4:29 p.m.
On Sat, 6 Oct 2012, Gene Czarcinski wrote:
OK, what am I missing? What don't I understand?
If IPv6 is going to be useful in virtualization, then there
must be some "easy" way to have other systems understand
that the virtualization host is acting as a router for the
virtual IPv6 networks it runs. While being able to go
between the virtualization hosts and the virtual guests is
very useful, I do not consider this sufficient.
We programatically, on a per VM basis, set up our ebtables and
iptables rules at pmman.com (thus my 'ROADMAP' question
earlier this week). Under RHEL 6's (and thus CentOS') KVM
and libvirtd stock setup, there was a built-in filter as
provided by libvirtd install -- as I recall: a 'clean-traffic'
filter -- that we had to amend out, compared to prior xen
setups under the earlier RHEL variant
Have you dumped and examined the running rules affecting IPv6
traffic?
-- Russ herrold
5:28 a.m.
On 10/06/2012 05:29 PM, R P Herrold wrote:
On Sat, 6 Oct 2012, Gene Czarcinski wrote:
> OK, what am I missing? What don't I understand?
>
> If IPv6 is going to be useful in virtualization, then there must be
> some "easy" way to have other systems understand that the
> virtualization host is acting as a router for the virtual IPv6
> networks it runs. While being able to go between the virtualization
> hosts and the virtual guests is very useful, I do not consider this
> sufficient.
We programatically, on a per VM basis, set up our ebtables and
iptables rules at pmman.com (thus my 'ROADMAP' question earlier this
week). Under RHEL 6's (and thus CentOS') KVM and libvirtd stock setup,
there was a built-in filter as provided by libvirtd install -- as I
recall: a 'clean-traffic' filter -- that we had to amend out, compared
to prior xen setups under the earlier RHEL variant
Have you dumped and examined the running rules affecting IPv6 traffic?
-- Russ herrold
The ip6tables rules look fine. The problem is not that the packets are
not sent to the target .... they are .. I ran wireshark on the target's
NIC. The problem is getting the response back to the virtualization host.
Shortly after I wrote my message I "discovered" something called
"neighbor discovery proxy" and two attempts at implementing it: npd6 and
ndppd. This is the RFC: http://tools.ietf.org/html/rfc4389
and here is a short description from npd6:
If you have a Linux gateway router terminating your ISP feed
supporting IPv6, this may be just what you need. To summarise the
problem it solves: your ISP has given you an /64 (or some other size)
IPv6 prefix, with the last 64 bits (or whatever) entirely for your own
use on a private-side of the network. The IPv6 addresses in use by
your own devices may well not even be known to you – it’s possible
that you use DHCP6 to statically pre-allocate them (yuck!) or more
likely you are using /radvd/ on the gateway to advertise the
ISP-supplied IPv6 prefix and let the devices themselves choose what
they wish to tag on to that. It may be vaguely predictable (based upon
the device’s Ethernet MAC address) or totally unpredictable (as per
the Windows 7 box I looked at the other day!)
...
And to do this today you need to /statically pre-configure/ that full
address into the Linux system. And if it changes, you need to change
it. And if a new one appears, you need to ad it. And so on. Oh, and to
add insult to injury, you cannot even display a list of which ones you
have already configured in the system!!
And thus I offer npd6 as a solution: it runs on the gateway, and
requires little configuration. You tell it your prefix and which is
the ISP’s interface. There are a few optional knobs and levers. Then
it runs and automatically responds to /any/ Neighbor Solicitation
received from the ISP for a device with your prefix.
This "sounds" like it may be a solution and I have started some testing
to see if and how they work.
Gene
11:10 a.m.
On 10/07/2012 06:28 AM, Gene Czarcinski wrote:
On 10/06/2012 05:29 PM, R P Herrold wrote:
> On Sat, 6 Oct 2012, Gene Czarcinski wrote:
>
>> OK, what am I missing? What don't I understand?
>>
>> If IPv6 is going to be useful in virtualization, then there must be
>> some "easy" way to have other systems understand that the
>> virtualization host is acting as a router for the virtual IPv6
>> networks it runs. While being able to go between the virtualization
>> hosts and the virtual guests is very useful, I do not consider this
>> sufficient.
>
> We programatically, on a per VM basis, set up our ebtables and
> iptables rules at pmman.com (thus my 'ROADMAP' question earlier this
> week). Under RHEL 6's (and thus CentOS') KVM and libvirtd stock
> setup, there was a built-in filter as provided by libvirtd install --
> as I recall: a 'clean-traffic' filter -- that we had to amend out,
> compared to prior xen setups under the earlier RHEL variant
>
> Have you dumped and examined the running rules affecting IPv6 traffic?
>
> -- Russ herrold
>
>
The ip6tables rules look fine. The problem is not that the packets are
not sent to the target .... they are .. I ran wireshark on the
target's NIC. The problem is getting the response back to the
virtualization host.
Shortly after I wrote my message I "discovered" something called
"neighbor discovery proxy" and two attempts at implementing it: npd6
and ndppd. This is the RFC: http://tools.ietf.org/html/rfc4389
and here is a short description from npd6:
> If you have a Linux gateway router terminating your ISP feed
> supporting IPv6, this may be just what you need. To summarise the
> problem it solves: your ISP has given you an /64 (or some other size)
> IPv6 prefix, with the last 64 bits (or whatever) entirely for your
> own use on a private-side of the network. The IPv6 addresses in use
> by your own devices may well not even be known to you – it’s possible
> that you use DHCP6 to statically pre-allocate them (yuck!) or more
> likely you are using /radvd/ on the gateway to advertise the
> ISP-supplied IPv6 prefix and let the devices themselves choose what
> they wish to tag on to that. It may be vaguely predictable (based
> upon the device’s Ethernet MAC address) or totally unpredictable (as
> per the Windows 7 box I looked at the other day!)
...
>
> And to do this today you need to /statically pre-configure/ that full
> address into the Linux system. And if it changes, you need to change
> it. And if a new one appears, you need to ad it. And so on. Oh, and
> to add insult to injury, you cannot even display a list of which ones
> you have already configured in the system!!
>
> And thus I offer npd6 as a solution: it runs on the gateway, and
> requires little configuration. You tell it your prefix and which is
> the ISP’s interface. There are a few optional knobs and levers. Then
> it runs and automatically responds to /any/ Neighbor Solicitation
> received from the ISP for a device with your prefix.
>
This "sounds" like it may be a solution and I have started some
testing to see if and how they work.
The more I look into this, the more I do not like the answer but it is
what it is.
First of all, ND-proxy is not the answer. Even if I could get it to
work on one network, it would not "turn the corner" to another network.
Unfortunately, radvd is not the answer either. Although I can configure
radvd on the virtualization host, the router advertisements it does are
ignored (by RFC requirement) by the system having the default route. As
far as I can see, the only thing that a RA is good for is establishing
the default route.
As things currently exist, I have come to the conclusion that there are
two answers that will work.
1. Manually configure the IPv6 address/network on the virtualization
host. On the default route system, add a static route to the
virtualization host. Things will then work.
2. It hurts, don't do it: use a bridge in the virtualization host for
IPv6 virtual networks which need to access external systems.
BTW, this is also needed for support of IPv4 routed networks. This can
be made a little easier by using 172.16.<nn>.0/24 networks and adding a
single static route for 172.16.0.0/16.
But for IPv6, there is just a lot of stuff that only works for networks
with a 64 prefix. When other prefixes (greater or lesser) are
specified, things get "strange." By RFC definition, RA only works with
prefix=64.
There is a form of nat for IPv6 in the works ("prefix translation") in
ip6tables. But, time will tell on how useful that will be.
Gene
8:31 a.m.
On 10/07/2012 12:10 PM, Gene Czarcinski wrote:
1. Manually configure the IPv6 address/network on the virtualization
host. On the default route system, add a static route to the
virtualization host. Things will then work.
2. It hurts, don't do it: use a bridge in the virtualization host for
IPv6 virtual networks which need to access external systems.
BTW, this is also needed for support of IPv4 routed networks. This can
be made a little easier by using 172.16.<nn>.0/24 networks and adding
a single static route for 172.16.0.0/16.
Giving this some additional thought,
there is a small alternative which
offers some advantages: use dhcp "nailed" host ip addresses. while this
works for both IPv4 and IPv6, it is more important for IPv6 because all
IPv6 virtual networksare "routed."
I have tested this using dnsmasq as the dhcpd6 and dns services provider
and there are a number of options for relating a specific
host/network-interface and an ip address. In addition to dnsmasq or
dhcpd6, you will need to add a static route for that virtual network to
the virtualization host on the system which is pointed to by the
default route.
There are some documents out there which imply to what I have concluded
above but things are left still a little vague. Examples of what is out
there are:
http://berrange.com/posts/2009/12/13/routed-subnets-without-nat-for-libvi...
http://blog.gadi.cc/routed-subnet-libvirt-kvm/
But, I do not see (or have not found yet) something under
http://wiki.libvirt.org
I am not sure where such information belongs but I do believe that it is
needed. Examples of where it could be placed are:
http://wiki.libvirt.org/page/VirtualNetworking#Routed_mode_2
http://wiki.libvirt.org/page/Networking
http://libvirt.org/formatnetwork.html#examplesRoute
Gene
1:35 p.m.
On 10/06/2012 10:34 AM, Gene Czarcinski wrote:
As I have mentioned in other messages, I am interested in having
full
support for IPv6 in libvirt. To me this includes having dhcp6 for
IPV6 address assignment and using RA (radvd) to establish the default
route. This is what I am using on my real LANs.
Before starting into adding dhcp6 support to libvirt, I wanted to see
just how it works with the current software. First of all, it appears
that, when nat or routed are specified for IPv4, the IPv6 is routed.
At the time I added IPv6 support, there was no NAT for IPv6, and even
now I don't know if it's clear that it will be accepted (truthfully I
haven't been following it). So the choice was to either not allow IPv6
at all on NATed networks, or to just route the IPv6 in these cases. We
chose the latter.
If it is an isolated/private network, then it can only work with
other guests on that network. The iptables and ip6tables settings
corresponded and were as expected. On the virtualization host, both
IPv4 and IPv6 forwarding are enabled.
While I can easily do stuff like ping6 and ssh -6 from virtual guests
to the virtualization host, I have been unable to do anything with
external hosts ... unless I add a static route for the virtual IPv6
network on the target host back to the virtualization host.
.. or on the machine acting as the default route. Or if you advertise a
route to the virtual network's subnet with a routing protocol.
This is the only way I have gotten anything to work. To complicate
things, it seem that "everything" wants the IPv6 network to have
prefix=64 or things do not work correctly.
Not totally. The one thing I'm aware of that requires a prefix=64
network is that IPv6 autoconf *by definition* will only work on networks
with a 64 bit prefix. I'm not really sure why this is so (I might have
read the reason a long time ago, but if so I've long forgotten what it
was), but it is, and there's nothing to be done about it :-/
The real systems use fd00:dead:beef:17::/64 for their network. The
virtual networks all use fd00:face:17:xx::/64 for their networks.
The network traffic on the virtualization host is forwarded to the
target host ... I can see the packets with wireshark on the target host.
On the target host I tried specifying a static route for network
fd00:face:17::/48 ... well, that really screwed things up, resulted in
some "redirects" from the virtualization host saying the that it was
sent a malformed packed ... it took a reboot to clean things up.
OK, so leave the fd00:face:17:6::/64 static route on the target host
but subnetwork this network on the virtualization host using networks
like fd00:face:17:6:8::/80 and fd00:face:6:9::/80. This works if I
manually configure IPv6 on the virtual guest. Since radvd is "upset"
by a non-prefix=64 network, I was not surprised when the guest's
automatic IPv6 address/network was not configured.
OK, what am I missing? What don't I understand?
It really is unfortunate that autoconf only works for prefix=64, as I
can easily imagine a case where the entire address space available to
the admin would be prefix=64, so they would need to allow only
prefix=(64+n) for each virtual network.
If IPv6 is going to be useful in virtualization, then there must be
some "easy" way to have other systems understand that the
virtualization host is acting as a router for the virtual IPv6
networks it runs.
That's what routing protocols do, and configuration of a routing
protocol daemon to advertise out to the physical network is outside the
scope of libvirt. This is nothing unique to IPv6 - people using IPv4
networks in <forward mode='route'/> mode have the same problem all the
time. It's not even unique to virtualization - you would have the same
problem if you created a physical subnet behind one of your hosts.
While being able to go between the virtualization hosts and the
virtual guests is very useful, I do not consider this sufficient.
I have googled and found some stuff as well as reading more RFCs than
I wanted to but I cannot find anything to address this issue.
IIRC, I did find something under a libvirt document that indicates
"routed" will be used for some kind of subnetworking.
Sigh. That one comment has done more to make me feel old than anything
else in a while :-/ Some history - The last time I needed to run a
routing protocol was > 10 years ago, and back then I was running NetBSD
and had just spent a bunch of time working on a router product based on
BSD4.3; at that time, the standard program to use for RIP (the "routing
information protocol" was called "routed" (i.e. the "route
daemon").
There was another project called "gated" (gateway daemon) that
implemented OSPF, EGP, and BGP, and that was about it.
Cut to today (well, Sunday anyway - I got a bit distracted by work and
didn't complete this reply in a timely fashion) and me seeing your
comment about "routed" in a manner that implied "what is this?"
I first naively ran "yum provides */routed" to see which package
contained routed ("Surely it must still be the go-to daemon for routing
information..."), and am told that no packages have it. So I search
around and find no mention of it on Linux, then finally realize that
it's under the BSD license, and there is no exact GPLed equivalent as
there is for most standard system utilities. More searching and I see
that the gated project is apparently completely defunct, that another
project called "zebra" had taken up those reins, and it too is now
defunct, but a fork of zebra called quagga is now around, and has a
package in Fedora.
So, the result of this trip: what seems to be the current in-use routing
protocol daemon on Linux systems is 2 generations removed from what I
last used.
Now GET OFF MY LAWN!!!!!! :-)
Anyway, I guess try quagga (unless someone else more in the know has a
better suggestion). You'll need to figure out what routing protocol is
in use on your network (if any) and go along with that.
Or you can just add a static route on the default router for the network
connected to your host's physical adapter.
Does libvirt need an IPv6 "NAT" to make this work?
One really nice thing about IPv6 is the elimination of NAT. One of the
biggest headaches about IPv6 for some network admins is the lack of NAT.
:-/ I personally would love to see NAT never exist in IPv6, but it
appears that won't happen. If the proposed NAT for IPv6 has "landed",
then we could certainly entertain patches to support it for <forward
mode='nat'/>, but only if it is standards track and stable. (I seriously
don't know how far along it is)
Comments? Suggestions?
In general, you may be helped by running a routing protocol daemon on
your host (as long as your routing infrastructure is also running a
routing protocol).
4428
days inactive
4431
days old
5 comments
3 participants
participants (3)
-
Gene Czarcinski -
Laine Stump -
R P Herrold