Re: [libvirt] [PATCH] v2:Support for adding a static route to a bridge

On 03/15/2013 02:10 PM, Gene Czarcinski wrote: (First I want to make sure I correctly understand what you're wanting to do, so I'm going to try and explain it in my own words...) this patch is to automatically setup a route to a virtual network whose bridge device has no IP address assigned, and is therefore reachable only via one of the guests, is this correct? In other words, let's say that I have the following two networks defined (using IPv4 and all static IPs for brevity, but the entire discussion is equally applicable to IPv6): <network> <name>netVisibleToHost</name> <bridge name='virbr1'/> <forward mode='route'/> <ip address='192.168.200.1' prefix='24'/> </network> <network> <name>netHiddenFromHost</name> <bridge name='virbr2'/> </network> and you have a guestDirect that has two interfaces: <interface type='network'> <!-- 192.168.200.2/24 --> <source network='netVisibleToHost'/> </interface> <interface type='network'> <!-- 192.168.201.1/24 --> <source network='netHiddenFromHost'/> </interface> and another guestIndirect that has only one interface: <interface type='network'> <!-- 192.168.201.2/24 --> <source network='netHiddenFromHost'/> </interface> Additionally, the default route on guestDirect points to 192.168.200.1, and the default route on guestIndirect points to 192.168.201.1. (Presumably you don't want to simply add an IP address to netHiddenFromHost because (while it would solve your routing problems) it would violate some security policy you've built into your network topology - namely that all traffic to and from netHiddenFromHost *must* go through guestDirect.) Traffic outbound from guestIndirect would have no problem reaching its destination - it would go across netHiddenFromHost to guestDirect (192.168.201.1), which would know to forward it to the host (192.168.200.1) via netVisibleToHost, and the host presumably knows how to get to anywhere. The problem comes when trying to route the *response* destined for 192.168.201.2 (guestIndirect) - the outside world may know that those packets have to be sent to the host, but the host doesn't have a route for 192.168.201.0/24 - only guestDirect does. So, the solution is to add a route on the *host* that points traffic destined for 192.168.201.0/24 to guestDirect, a.k.a. 192.168.200.2. Since there's no place in /etc/sysconfig/network-scripts/route-* to put static routes with destinations that are only reachable through a transient interface such as the bridge devices created by libvirt, the obvious solution is to cause libvirt to add such a route, and the way you propose to do that is to add an <ip> element in the network named "netUnreachable". Am I understanding the issue so far? Assuming that I am, then as far as I can see, the correct place to configure this route isn't in the setup for netHiddenFromHost, but rather in netVisibleToHost - this is more in line with the way static routes are configured in the standard host network config (route-* files), and eliminates the problem that would occur if netHiddenFromHost was started before netVisibleToHost existed (the route would be added pointing at the wrong interface, if at all) So, what I'm proposing is that, to automatically setup the route in the above example, netHiddenFromHost would remain unchanged, and netVisibleToHost would look something like this: <network> <name>netVisibleToHost</name> <bridge name='virbr1'/> <forward mode='route'/> <ip address='192.168.200.1' prefix='24'/> <route address='192.168.201.0' prefix='24' gateway='192.168.200.2'/> </network> The route element could also have a family attribute just as <ip> does (although it's fairly simple to figure out the family just by trying to parse address and/or gateway). You might instead want to make <route> a subelement of <ip>, then validate that the gateway address is directly reachable from the given ip address (i.e. that it's on the same subnet). By putting the configuration here, you could be assured that the interface that will be used for the route will always exist at the time the route is added. Also it is conceptually more similar to the way that the routes in /etc/sysconfig/route-ethX all have gateways that are directly reachable via "ethX". *OR* (following is what I think is a bad idea, but maybe someone can tweak and salvage it :-) Here is an alternate proposal that has the advantage of tying the existence of the static route to the existence of not just the bridge, but of even the guest interface will be the gateway: instead of putting the route in the configuration of the netVisibleToHost network, put it directly in the configuration of the guest interface itself. That way when the guest is started the route will be added, and when the guest is shutdown, the route (which will anyway now be useless) will be removed. That could be added to the <interface> definition something like this: <interface type='network'> <!-- 192.168.200.2/24 --> <source network='netVisibleToHost'/> <route address='192.168.201.0' prefix='24' gateway='192.168.200.2'/> </interface> If this was done correctly, you could even hotplug a guest interface that would then be immediately used for a route, as well as updating an existing guest interface to add a route. The route added would use the information in the <route> element plus the bridge device the guest's tap device was connected to. I do have several reservations about this idea, though 1) Up until now there has been no "pollution" of the guest config with IP-related info, it has all been about hardware. (well, except for filterref parameters...) 2) Even worse than that is that the guest interface config *can't* know the IP address that the guest will use for the interface, yet we would now be hard-coding it into this new <route> element in the guest interface config. Ugh. (Now what would be *really* slick is if we could have routes in <interface> that learned their IP address from nwfilter's IP address snooping! :-) 3) If the interface happens to be connected to a bridge that has no IP address on the host (e.g. the bridge of netHiddenFromHost in the example above), the route will be a failure. 4) Likewise, if the interface is connected with macvtap, again the route would be a failure. So, in the end, the idea of adding <route> to <interface> is probably a no-go. (One upside - if the interface is a hostdev, we could simply add the route with no "dev X" appended, and it *would* work, as long as the host had an interface on the same subnet). It seems to me that the "dev" given here is incorrect. If I'm not mistaken, the route you're adding with your current code will use the device name of the bridge on "netHiddenFromHost" in this command, while what you *really* want is the bridge used for "netVisibleToHost" (which isn't even known/available at the time you start netHiddenFromHost - another reason to do it in one of the ways I'm suggesting).