[Libvir] [PATCH] Add a default network
Hey, So, we want to install a default network which guests can connect to. This can be seen as e.g. a replacement for xenbr0 as the default bridge for xen guests. A few things concern me about the patch, see below for my thinking. However, I'm happy to punt on all of these issues for now and go ahead with the patch. 1) Ordering - we want the default network to have a predictable bridge name, so instead of relying on it being the first network and ending up auto-allocated vnet0, we use virbr0. The alternative is to actually have an ordering scheme, e.g. #define LIBVIRT_AUTOSTART_PRIORITY_DISABLE -1 #define LIBVIRT_AUTOSTART_PRIORITY_FIRST 0 #define LIBVIRT_AUTOSTART_PRIORITY_LAST 99 #define LIBVIRT_AUTOSTART_PRIORITY_DEFAULT 50 int virNetworkSetAutostart(virDomainPtr domain, int priority); autostart/00-default.xml -> ../default.xml autostart/99-MyNetwork.xml -> ../MyNetwork.xml Maybe that's useful, but probably moreso for guests than networks. The "virbr0" solution is fine, IMHO. 2) IP address choice - I've randomly chosen 192.168.122.1/24 as the IP address for the network, and this could happen to clash with an existing network. Since we masquerade outgoing traffic, I think the only problem this would cause in practice would be where the host machine is already on a 192.168.122/24 subnet and it could find itself connecting to a guest rather than another machine on the physical subnet. It's a default configuration choice which admins can change, and I don't see it causing huge problems in practice, so I'm not overly concerned. If it did turn out to be a problem, I'd probably add something like: <ip type="auto" /> But I'm not anxious to go down that route right now. 3) UUID generation - there's no UUID specified, so the default network will have a different UUID across reboots. We could trivially save the UUID at first boot, but we'd really need to put it in /var then. IMHO, it's something we can fix later if it turns out to be important. Cheers, Mark.
On Mon, Feb 26, 2007 at 04:09:58PM +0000, Mark McLoughlin wrote:
Hey, So, we want to install a default network which guests can connect to. This can be seen as e.g. a replacement for xenbr0 as the default bridge for xen guests.
A few things concern me about the patch, see below for my thinking. However, I'm happy to punt on all of these issues for now and go ahead with the patch.
1) Ordering - we want the default network to have a predictable bridge name, so instead of relying on it being the first network and ending up auto-allocated vnet0, we use virbr0.
The alternative is to actually have an ordering scheme, e.g.
#define LIBVIRT_AUTOSTART_PRIORITY_DISABLE -1 #define LIBVIRT_AUTOSTART_PRIORITY_FIRST 0 #define LIBVIRT_AUTOSTART_PRIORITY_LAST 99 #define LIBVIRT_AUTOSTART_PRIORITY_DEFAULT 50
int virNetworkSetAutostart(virDomainPtr domain, int priority);
autostart/00-default.xml -> ../default.xml autostart/99-MyNetwork.xml -> ../MyNetwork.xml
I think its fine to let the ordering be done based on network name. eg if the user wants an explicit ordering, then they can name their network '00default', rather than us munging the names.
Maybe that's useful, but probably moreso for guests than networks. The "virbr0" solution is fine, IMHO.
Hmm, in guests more explicit dependancy info 'web1' requires 'db1', except that this is pretty much an impossible problem to solve. eg, you can start 'db1' first, but how do you know how long it takes for db1 to boot up & start accepting connections. So I think ordering is prety much an impossible problem to solve in a generally useful case.
2) IP address choice - I've randomly chosen 192.168.122.1/24 as the IP address for the network, and this could happen to clash with an existing network. Since we masquerade outgoing traffic, I think the only problem this would cause in practice would be where the host machine is already on a 192.168.122/24 subnet and it could find itself connecting to a guest rather than another machine on the physical subnet.
Picking a range like that is fine, if its easy for the admin to change it in a config file - just need to make sure if the admin changes it (or say wants to disable it entirely), it doesn't get reverted / renable during an RPM upgrade. The former can be dealt with by marking %config, but not sure how we'd prevent deletion of the default network being reverted upon update.
If it did turn out to be a problem, I'd probably add something like:
<ip type="auto" />
But I'm not anxious to go down that route right now.
Thats really just moving the problem elsewhere - we'd just need another config option somewhere else to spec what the 'auto' range mapped to.
3) UUID generation - there's no UUID specified, so the default network will have a different UUID across reboots.
We could trivially save the UUID at first boot, but we'd really need to put it in /var then.
Generate it in %post and write it into the config file ? That's the approach DBus uses for its auto-generated UUID Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Mon, Feb 26, 2007 at 04:09:58PM +0000, Mark McLoughlin wrote:
So, we want to install a default network which guests can connect to. This can be seen as e.g. a replacement for xenbr0 as the default bridge for xen guests.
2) IP address choice - I've randomly chosen 192.168.122.1/24 as the IP address for the network, and this could happen to clash with an existing network.
Oh, the default network needs to provide IPv6 support out of the box too - whatever that entails ? Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Fri, 2007-03-02 at 17:15 +0000, Daniel P. Berrange wrote:
On Mon, Feb 26, 2007 at 04:09:58PM +0000, Mark McLoughlin wrote:
So, we want to install a default network which guests can connect to. This can be seen as e.g. a replacement for xenbr0 as the default bridge for xen guests.
2) IP address choice - I've randomly chosen 192.168.122.1/24 as the IP address for the network, and this could happen to clash with an existing network.
Oh, the default network needs to provide IPv6 support out of the box too - whatever that entails ?
Right, whatever that entails :-) So, we had two primary motivations for creating virtual networks: 1) Offline support - i.e. support inter-domain communication even when offline 2) Network switching - e.g. switching your laptop between different wireless networks However, with IPv6, the combination of link-local addresses, address auto-configuration and network renumbering should largely eliminate these problems. The conclusion then is that you mostly do want bridging with IPv6 - i.e. you want to bridge all guests onto your physical network whereby they will auto-configure using router advertisements on the physical link. One could imagine us allowing IPv6 virtual networks, where Dom0 acts as a proper IPv6 router advertising a delegated prefix to guest domains, but I'm not sure why that would be useful to people. The question, though, is how to make IPv6 available to guests which are connected to a virtual network out of a need for e.g. offline support. You still want NAT etc. for IPv4, but what to do about IPv6? The analogy, I think, is what would happen if your DSL provider statically allocated an IPv6 prefix to you while still also dynamically allocating an IPv4 address to you. You want to NAT IPv4 traffic using the IPv4 address, but you want your IPv6 traffic to be bridged to the IPv6 over PPP link in order to e.g. get router advertisements from the ISP end. That leads to the rather ugly conclusion that we should bridge IPv6 traffic from the virtual network to the physical interface, while still forwarding IPv4 traffic using NAT. In order to do that, we're going to need a) "shared physical interface" configuration i.e. a per-interface flag that determines whether other interfaces are allowed to bridge to it, b) an ipv6-only loopback interface for connecting the virtual network's bridge to the shared physical interface's bridge and c) a way for libvirtd to automatically switch the the virtual network's bridge to the currently active physical interface. That's all pretty far out stuff. In the mean time, I think all we can do is add an "ipv6 enabled" flag which would just ensure that the virtual networks interface in Dom0 has a link-local address. Any thoughts on all this are most welcome. Cheers, Mark.
On Tue, Mar 06, 2007 at 09:37:46AM +0000, Mark McLoughlin wrote:
On Fri, 2007-03-02 at 17:15 +0000, Daniel P. Berrange wrote:
On Mon, Feb 26, 2007 at 04:09:58PM +0000, Mark McLoughlin wrote:
So, we want to install a default network which guests can connect to. This can be seen as e.g. a replacement for xenbr0 as the default bridge for xen guests.
2) IP address choice - I've randomly chosen 192.168.122.1/24 as the IP address for the network, and this could happen to clash with an existing network.
Oh, the default network needs to provide IPv6 support out of the box too - whatever that entails ?
Right, whatever that entails :-)
So, we had two primary motivations for creating virtual networks:
1) Offline support - i.e. support inter-domain communication even when offline
2) Network switching - e.g. switching your laptop between different wireless networks
3) Isolated networks - ie not letting (some subset of) your VMs be exposed to scary wildwest of the internet :-) The latter isn't really relevant to the 'default network' use case though, but is still a use case we need to think of for interesting admin defined network topologies.
However, with IPv6, the combination of link-local addresses, address auto-configuration and network renumbering should largely eliminate these problems.
The conclusion then is that you mostly do want bridging with IPv6 - i.e. you want to bridge all guests onto your physical network whereby they will auto-configure using router advertisements on the physical link.
I must admit to not understaning IPv6 all that much. Do link-local addreses let the guest communicate with outside world, or is only enablling the VM-to-VM and VM-to-Host communications ?
One could imagine us allowing IPv6 virtual networks, where Dom0 acts as a proper IPv6 router advertising a delegated prefix to guest domains, but I'm not sure why that would be useful to people.
To let you isolate VMs from the wider world. Not relevant for a 'default network' use case though.
The question, though, is how to make IPv6 available to guests which are connected to a virtual network out of a need for e.g. offline support. You still want NAT etc. for IPv4, but what to do about IPv6?
The analogy, I think, is what would happen if your DSL provider statically allocated an IPv6 prefix to you while still also dynamically allocating an IPv4 address to you. You want to NAT IPv4 traffic using the IPv4 address, but you want your IPv6 traffic to be bridged to the IPv6 over PPP link in order to e.g. get router advertisements from the ISP end.
I don;t know of any DSL providers or DSL routers which do IPv6, but I'd expect that all my machines on my LAN magically get an IPv6 address and that they can access the outside world. I'd still expect incoming traffic to be restricted by the DSL router firewalling as per IPv4 incoming.
That leads to the rather ugly conclusion that we should bridge IPv6 traffic from the virtual network to the physical interface, while still forwarding IPv4 traffic using NAT.
Yeah, that seems to be the way it'd work.
In order to do that, we're going to need a) "shared physical interface" configuration i.e. a per-interface flag that determines whether other interfaces are allowed to bridge to it, b) an ipv6-only loopback interface for connecting the virtual network's bridge to the shared physical interface's bridge and c) a way for libvirtd to automatically switch the the virtual network's bridge to the currently active physical interface.
That's all pretty far out stuff. In the mean time, I think all we can do is add an "ipv6 enabled" flag which would just ensure that the virtual networks interface in Dom0 has a link-local address.
What does the link-local address let a VM do ? VM-Host and VM-VM comms, but not VM-Internet right ? Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Wed, 2007-03-07 at 18:15 +0000, Daniel P. Berrange wrote:
On Tue, Mar 06, 2007 at 09:37:46AM +0000, Mark McLoughlin wrote:
However, with IPv6, the combination of link-local addresses, address auto-configuration and network renumbering should largely eliminate these problems.
The conclusion then is that you mostly do want bridging with IPv6 - i.e. you want to bridge all guests onto your physical network whereby they will auto-configure using router advertisements on the physical link.
I must admit to not understaning IPv6 all that much.
My understanding is fairly limited, newly acquired and all pretty theoretical, so beware :-)
Do link-local addreses let the guest communicate with outside world, or is only enablling the VM-to-VM and VM-to-Host communications ?
link-local addresses are only valid on the local link, so e.g. a router won't forward such packets. So, my point is that link-local addresses gives you offline support, since domains can reach one another. How useful in practice that is, I don't know. You don't go typing in IPv6 addresses, so I guess it's only really useful if you can look up the guest's address in DNS or mDNS even when offline.
The question, though, is how to make IPv6 available to guests which are connected to a virtual network out of a need for e.g. offline support. You still want NAT etc. for IPv4, but what to do about IPv6?
The analogy, I think, is what would happen if your DSL provider statically allocated an IPv6 prefix to you while still also dynamically allocating an IPv4 address to you. You want to NAT IPv4 traffic using the IPv4 address, but you want your IPv6 traffic to be bridged to the IPv6 over PPP link in order to e.g. get router advertisements from the ISP end.
I don;t know of any DSL providers or DSL routers which do IPv6, but I'd expect that all my machines on my LAN magically get an IPv6 address and that they can access the outside world. I'd still expect incoming traffic to be restricted by the DSL router firewalling as per IPv4 incoming.
It's not clear to me how e.g. netgear would implement that in their routers. The obvious, but lame way to do it would be for your machines to only have link-local addresses and outgoing traffic gets NATed. That would suck, and you can't even do NAT with IPv6 apparently. Another way you could imagine would be for the your router to act as an IPv6 router for a delegated prefix, but I'm not sure how the ISP would communicate what that prefix should be to the router. Same with our situation, I'm not sure how a Dom0 acting as an IPv6 router would figure out what prefix has been delegated to it for its guests. The final way is that the DSL router just blindly bridges all IPv6 traffic onto the PPP link and, so, machines on your network are talking directly to an IPv6 router on the ISP's side which knows what prefix you have been delegated. Looking at rfc2472, IPv6 over PPP, I think that's the way it's supposed to work. That's fine, except it's really funky when you consider that your DSL router is still NAT-ing your IPv4 traffic. So, it's acting as a IP level router for your IPv4 traffic and a link-level bridge for your IPv6 traffic. That's doable, but it sounds bizarre to me. I guess it's no more bizarre than the fact that iptables works on a linux bridge, though. Oh, yeah - the firewall issue. Your firewall on a DSL router falls naturally out of the fact that it's doing NAT, but it'd need to actual IP filtering as it's bridging your IPv6 traffic for you to have the same firewall rules for IPv6. Uggh. Cheers, Mark.
On Fri, Mar 09, 2007 at 01:52:31PM +0000, Mark McLoughlin wrote:
On Wed, 2007-03-07 at 18:15 +0000, Daniel P. Berrange wrote:
Do link-local addreses let the guest communicate with outside world, or is only enablling the VM-to-VM and VM-to-Host communications ?
link-local addresses are only valid on the local link, so e.g. a router won't forward such packets.
So, my point is that link-local addresses gives you offline support, since domains can reach one another.
How useful in practice that is, I don't know. You don't go typing in IPv6 addresses, so I guess it's only really useful if you can look up the guest's address in DNS or mDNS even when offline.
Well even if you don't have formal DNS names for each guest, it would at least let funky zero-conf Avahi enabled apps do their magic discovery, so worthwhile from that POV.
The question, though, is how to make IPv6 available to guests which are connected to a virtual network out of a need for e.g. offline support. You still want NAT etc. for IPv4, but what to do about IPv6?
The analogy, I think, is what would happen if your DSL provider statically allocated an IPv6 prefix to you while still also dynamically allocating an IPv4 address to you. You want to NAT IPv4 traffic using the IPv4 address, but you want your IPv6 traffic to be bridged to the IPv6 over PPP link in order to e.g. get router advertisements from the ISP end.
I don;t know of any DSL providers or DSL routers which do IPv6, but I'd expect that all my machines on my LAN magically get an IPv6 address and that they can access the outside world. I'd still expect incoming traffic to be restricted by the DSL router firewalling as per IPv4 incoming.
It's not clear to me how e.g. netgear would implement that in their routers.
The obvious, but lame way to do it would be for your machines to only have link-local addresses and outgoing traffic gets NATed. That would suck, and you can't even do NAT with IPv6 apparently.
Yeah, sounds like this is rather frowned up in IPv6 world
Another way you could imagine would be for the your router to act as an IPv6 router for a delegated prefix, but I'm not sure how the ISP would communicate what that prefix should be to the router. Same with our situation, I'm not sure how a Dom0 acting as an IPv6 router would figure out what prefix has been delegated to it for its guests.
Yeah I was just reading this doc http://arstechnica.com/articles/paedia/IPv6.ars/2 And the "Stateless autoconfiguration" diagram seems to be exactly what I think we'd want. Every guest has a MAC addr so that deals with the lower 64-bits of the adress, but how do we choose the upper 64-bits to form our 'router advertisment'... Perhaps that's the bit that we stick in the libvirt XML as the configuration parameter <network> <name>default</name> <bridge name="virbr0" /> <ipv6 advprefix="2001:db8:31:0:0:0:0:1"/> </network>
Oh, yeah - the firewall issue. Your firewall on a DSL router falls naturally out of the fact that it's doing NAT, but it'd need to actual IP filtering as it's bridging your IPv6 traffic for you to have the same firewall rules for IPv6. Uggh.
Having to duplicate the firewall rules is not entirely surprising, so I figure we can deal with that. Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Daniel P. Berrange wrote:
And the "Stateless autoconfiguration" diagram seems to be exactly what I think we'd want. Every guest has a MAC addr so that deals with the lower 64-bits of the adress, but how do we choose the upper 64-bits to form our 'router advertisment'... Perhaps that's the bit that we stick in the libvirt XML as the configuration parameter
IPv6 has a way that hosts can choose a (semi-)random upper 64 bits as well. Unfortunately as far as I can see it's deprecated :-( Anyhow, you can read more about it here: http://www.faqs.org/rfcs/rfc3513.html (section 2.5.6) http://www.faqs.org/rfcs/rfc3879.html You might also consider using link-local addresses (which are not deprecated), since the guests are supposed to be directly connected to each other. I'm not sure of the implications of that though ...? Rich. -- Emerging Technologies, Red Hat http://et.redhat.com/~rjones/ 64 Baker Street, London, W1U 7DF Mobile: +44 7866 314 421 "[Negative numbers] darken the very whole doctrines of the equations and make dark of the things which are in their nature excessively obvious and simple" (Francis Maseres FRS, mathematician, 1759)
On Fri, 2007-03-09 at 14:57 +0000, Daniel P. Berrange wrote:
On Fri, Mar 09, 2007 at 01:52:31PM +0000, Mark McLoughlin wrote:
Another way you could imagine would be for the your router to act as an IPv6 router for a delegated prefix, but I'm not sure how the ISP would communicate what that prefix should be to the router. Same with our situation, I'm not sure how a Dom0 acting as an IPv6 router would figure out what prefix has been delegated to it for its guests.
Yeah I was just reading this doc
http://arstechnica.com/articles/paedia/IPv6.ars/2
And the "Stateless autoconfiguration" diagram seems to be exactly what I think we'd want. Every guest has a MAC addr so that deals with the lower 64-bits of the adress, but how do we choose the upper 64-bits to form our 'router advertisment'...
Yep, that's how it works but ...
Perhaps that's the bit that we stick in the libvirt XML as the configuration parameter
<network> <name>default</name> <bridge name="virbr0" /> <ipv6 advprefix="2001:db8:31:0:0:0:0:1"/> </network>
Where does the advprefix (or what I was calling a delegated prefix) come from in either the netgear DSL router or default virtual network case? In the latter case, I'd expect some sort way for a router to advertise delegated prefixes to other routers either via router advertisement or DHCPv6, but I don't think that's the way things are expected to work. In the former case, I'd expect the same, or perhaps I'd expect the prefix to be given as part of the IPv6 over PPP extensions, but no. So, it seems to me that IPv6 hosts are expected to be able to connect to an IPv6 router which has been manually configured with a delegated prefix. Which implies bridging in between the host and the router, even if NAT is being used by IPv4.
Oh, yeah - the firewall issue. Your firewall on a DSL router falls naturally out of the fact that it's doing NAT, but it'd need to actual IP filtering as it's bridging your IPv6 traffic for you to have the same firewall rules for IPv6. Uggh.
Having to duplicate the firewall rules is not entirely surprising, so I figure we can deal with that.
No, the problem is that you have to configure those firewall rules differently for IPv4 and IPv6. For the former, the default of "reject all incoming traffic" falls out of the fact you're using NAT, but you'd actually need an IP filter to have that same rule for IPv6. Okay, it's not a big problem, just really weird. Cheers, Mark.
participants (3)
-
Daniel P. Berrange -
Mark McLoughlin -
Richard W.M. Jones