[libvirt] Update on host interface configuration
Hi, I thought I'd write an update of what I've been doing with the host interface configuration previously discussed[1] After talking with Dan Williams, who is working in NetworkManager, it is clear that the network interface functionality for libvirt is also useful for NM and would help them with handling system-wide interface configuration. To accomodate that, I will implement network interface configuration in a separate library (inspiringly called 'netcf', until somebody thinks of something catchier) and add public API to libvirt that will use netcf to do the actual work. One thorny issue is how libvirt's public API and netcf's public API relate to each other, especially since both libvirt and netcf use XML descriptions to describe network interfaces (netcf does that for exactly the same reasons that libvirt does it, since it will have to deal with similar stability issues) For the time being, I am assuming that the two XML descriptions are identical, with no guarantee that that will always be the case. If and when we run into a case where libvirt's and netcf's needs diverge, we'll break that identity. On the implementation side, that means that, for now, netcf will do all the validation and pass errors back to libvirt. In more concrete terms, I put up a git repo[2] for netcf. There's hardly any code in there, the things I'd particularly appreciate review/feedback on are * the RelaxNG specification of the interface XML (in xml/interface.rng) together with examples of such interface specifications (in tests/interface/) * the (very rough) sketch of the netcf public API, which should map almost 1-1 to the public libvirt API in src/netcf.h I know that the XML schema doesn't cover everything we've discussed in [1], but I think it's a small enough subset to tackle relatively quickly and large enough to be useful. So, one of the main things to think about when looking at the schema is places where the schema would make it hard to make desirable enhancements in the near future (like Marius' point about default routes/gateways) One important point about the model of network config in the schema is that it is centered around what NM calls a 'connection' (though it's called 'interface' in the schema): a grouping of the components that make up the interface the user really cares about (a bridge or a bond) together with the components subordinate to it (e.g. ethernet interfaces enslaved to the bridge or bond) It's those groups that netcf deals with, i.e. if eth0 is enslaved to br0, netcf won't list eth0 as an interface, but only br0, with eth0 as a subinterface. For the implementation, netcf will have a separate backend for each network configuration regime (initscripts for Fedora/RHEL, /etc/network/interfaces for Debian, ...) - once the initscripts backend is done, I'd be very grateful for contributions for the other backends. To sketch out the initscripts implementation, I wrote two XSL stylesheets, xml/initscripts-get.xml that goes from interface XML -> initscripts files (via Augeas) and xml/initscripts-put.xml to go the other way for querying; I'll use those for the core of the initscripts backend. Whether other backands can be done in a similar manner is TBD. David [1] https://www.redhat.com/archives/libvir-list/2009-January/msg00350.html [2] http://git.et.redhat.com/?p=netcf.git;a=tree
Hi David, Nice work ... A thought as I update this Fedora feature page[1], given that the goal of the feature could be described as fixing things so that the "shared physical device" section of libvirt's Networking wiki page[2] isn't so nasty and 'orrible. Aside from just writing out the interface configuration, will we also handle the case where NetworkManager is disabled/unavailable and manually bring up the interface with initscripts and update iptables? Also, will NetworkManager automatically notice the configuration changing on disk and apply the configuration? Will we be able to detect what configurations NetworkManager can handle? Cheers, Mark. [1] - https://fedoraproject.org/wiki/Features/Shared_Network_Interface [2] - http://wiki.libvirt.org/page/Networking
On Fri, 2009-02-13 at 09:12 +0000, Mark McLoughlin wrote:
A thought as I update this Fedora feature page[1]
Thanks a ton for doing that - nice overview.
Aside from just writing out the interface configuration, will we also handle the case where NetworkManager is disabled/unavailable and manually bring up the interface with initscripts and update iptables?
That side of the integration is still a little murky to me ... ideally, we wouldn't have to worry about whether NM is running or not, and just bring interfaces up independently, with NM noticing any changes - Dan W: does that sounds reasonable ? Also, to make a shared phys interface useful, we'd defintiely need to add the right --physdev-is-bridged rule, contradicting what I said elsewhere about modifying iptables ;)
Also, will NetworkManager automatically notice the configuration changing on disk and apply the configuration?
Yes, NM either needs to be told what files/dirs to look at so that it can set up its own inotify watches, or (preferrably) we'll need to let it register callbacks and do the inotify biz ourselves.
Will we be able to detect what configurations NetworkManager can handle?
I am hoping we (i.e., libvirt) don't have to; though netcf will have to be taught how to deal with the interfaces that NM cares about and libvirt likely doesn't all that much (wireless) David
On Fri, 2009-02-13 at 09:12 +0000, Mark McLoughlin wrote:
Hi David,
Nice work ...
A thought as I update this Fedora feature page[1], given that the goal of the feature could be described as fixing things so that the "shared physical device" section of libvirt's Networking wiki page[2] isn't so nasty and 'orrible.
Aside from just writing out the interface configuration, will we also handle the case where NetworkManager is disabled/unavailable and manually bring up the interface with initscripts and update iptables?
Also, will NetworkManager automatically notice the configuration changing on disk and apply the configuration? Will we be able to detect what configurations NetworkManager can handle?
NM currently doesn't re-configure the connection on-the-fly when the backing store (ie, ifcfg or gconf) changes, but the changes will get applied the next time the connection is activated. The user (or some other program) can certainly re-activate the connection after writing out the bits to get the new settings applied. nm-system-settings would notice the changes via inotify, and read in the new settings. Then, when the ActivateConnection method was called for that specific connection, NM would re-activate it and apply the new settings. Dan
On Thu, 2009-02-26 at 12:40 -0500, Dan Williams wrote:
On Fri, 2009-02-13 at 09:12 +0000, Mark McLoughlin wrote:
Hi David,
Nice work ...
A thought as I update this Fedora feature page[1], given that the goal of the feature could be described as fixing things so that the "shared physical device" section of libvirt's Networking wiki page[2] isn't so nasty and 'orrible.
Aside from just writing out the interface configuration, will we also handle the case where NetworkManager is disabled/unavailable and manually bring up the interface with initscripts and update iptables?
Also, will NetworkManager automatically notice the configuration changing on disk and apply the configuration? Will we be able to detect what configurations NetworkManager can handle?
NM currently doesn't re-configure the connection on-the-fly when the backing store (ie, ifcfg or gconf) changes, but the changes will get applied the next time the connection is activated. The user (or some other program) can certainly re-activate the connection after writing out the bits to get the new settings applied.
nm-system-settings would notice the changes via inotify, and read in the new settings. Then, when the ActivateConnection method was called for that specific connection, NM would re-activate it and apply the new settings.
Yes, that agrees with how I've been thinking about this: changing the config and bringing an interface/connection up/down are different operations. David
I know I'm arriving very late to this discussion and should have read it all before posting, but... The project I'm working on wanted to be able to setup network configuration without making persistant changes to configuration files. This would allow testing if everything works well, and reboot without fear if things got broken. The greatest fear when configuring network remotely, is being locked out of your host. Do you plan something like this feature in netcf? Dan.
On Tue, 2009-02-24 at 18:24 +0200, Dan Kenigsberg wrote:
I know I'm arriving very late to this discussion and should have read it all before posting, but...
The project I'm working on wanted to be able to setup network configuration without making persistant changes to configuration files. This would allow testing if everything works well, and reboot without fear if things got broken. The greatest fear when configuring network remotely, is being locked out of your host.
Do you plan something like this feature in netcf?
No, not for netcf or libvirt. The host interface mgmt uses the distribution's native network scripts, which implies that all config changes are changes to those scripts. The fundamental difficulty with providing some sort of rollback functionality at the libvirt level is that libvirt has no idea whether the new config is working or not, and doesn't know how to test it. So any (syntactically valid) configuration of a network interface is as good as any other from libvirt's POV. That means that this type of functionality really belongs in a layer above libvirt - the API (will) make it pretty easy though to implement that: if you know that the current config for an interface is good/working, just do something like goodConfig = virInterfaceGetXmlDesc(..); virInterfaceDefine(newConfig); virInterfaceRestart(ifaceName); if (! newConfigIsWorking()) { virInterfaceDefine(goodConfig); virInterfaceRestart(ifaceName); } David
On Tue, Feb 24, 2009 at 11:06:02PM +0000, David Lutterkort wrote:
On Tue, 2009-02-24 at 18:24 +0200, Dan Kenigsberg wrote:
I know I'm arriving very late to this discussion and should have read it all before posting, but...
The project I'm working on wanted to be able to setup network configuration without making persistant changes to configuration files. This would allow testing if everything works well, and reboot without fear if things got broken. The greatest fear when configuring network remotely, is being locked out of your host.
Do you plan something like this feature in netcf?
No, not for netcf or libvirt. The host interface mgmt uses the distribution's native network scripts, which implies that all config changes are changes to those scripts.
The fundamental difficulty with providing some sort of rollback functionality at the libvirt level is that libvirt has no idea whether the new config is working or not, and doesn't know how to test it. So any (syntactically valid) configuration of a network interface is as good as any other from libvirt's POV.
That means that this type of functionality really belongs in a layer above libvirt - the API (will) make it pretty easy though to implement that: if you know that the current config for an interface is good/working, just do something like
goodConfig = virInterfaceGetXmlDesc(..); virInterfaceDefine(newConfig); virInterfaceRestart(ifaceName);
The real issue is, that in this stage, management may have lost connectivity to the node.
if (! newConfigIsWorking()) { virInterfaceDefine(goodConfig); virInterfaceRestart(ifaceName); }
How about having netcf/libvirt be aware of what is a goodConfig (with additional API verb), and back it up somewhere? Any change to to network configuration would have immediate effect, but unless explicitly set as "good", reverted on next boot.
On Fri, 2009-02-27 at 10:07 +0200, Dan Kenigsberg wrote:
The real issue is, that in this stage, management may have lost connectivity to the node.
What exactly is the scenario here ? Are you trying to manage any old node as long as it has libvirt installed ? As soon as you need any sort of management-specific code on the node, you might as well implement the logic below in your management code.
if (! newConfigIsWorking()) { virInterfaceDefine(goodConfig); virInterfaceRestart(ifaceName); }
How about having netcf/libvirt be aware of what is a goodConfig (with additional API verb), and back it up somewhere? Any change to to network configuration would have immediate effect, but unless explicitly set as "good", reverted on next boot.
While what you suggest will work in theory (with the minor caveat that reverting to a "good" config can only be triggered by a restart of libvirtd, not a reboot) it is a poor man's workaround for management systems that do not provide proper failsafe functionality. In particular, if you are part of the management system, you have a much better idea (on the node) what it means to have a working network configuration, and can recover from misconfigured network much more gracefully than with the big hammer of a system reboot. David
On Fri, Feb 27, 2009 at 02:59:23PM -0800, David Lutterkort wrote:
On Fri, 2009-02-27 at 10:07 +0200, Dan Kenigsberg wrote:
The real issue is, that in this stage, management may have lost connectivity to the node.
What exactly is the scenario here ? Are you trying to manage any old node as long as it has libvirt installed ? As soon as you need any sort of management-specific code on the node, you might as well implement the logic below in your management code.
The scenario is of a central management of multiple nodes, where libvirt is called through remote:///-style connection. If you have an agent on each node and don't use libvirt across the network, there's indeed no problem.
if (! newConfigIsWorking()) { virInterfaceDefine(goodConfig); virInterfaceRestart(ifaceName); }
How about having netcf/libvirt be aware of what is a goodConfig (with additional API verb), and back it up somewhere? Any change to to network configuration would have immediate effect, but unless explicitly set as "good", reverted on next boot.
While what you suggest will work in theory (with the minor caveat that reverting to a "good" config can only be triggered by a restart of libvirtd, not a reboot) it is a poor man's workaround for management systems that do not provide proper failsafe functionality.
In particular, if you are part of the management system, you have a much better idea (on the node) what it means to have a working network configuration, and can recover from misconfigured network much more gracefully than with the big hammer of a system reboot.
I think I'll have the big hammer in my toolkit, just in case..
From: libvir-list-bounces@redhat.com [mailto:libvir-list- bounces@redhat.com] On Behalf Of David Lutterkort ... In particular, if you are part of the management system, you have a much better idea (on the node) what it means to have a working network configuration, and can recover from misconfigured network much more gracefully than with the big hammer of a system reboot. [IH] indeed, but this implies the management system has to co-locate another management daemon on the node, since central management cannot rollback the node from an invalid network configuration. It will also be simpler for the other daemon to be the one accepting the network change command from the central management, using libvirt locally (or be notified by libvirt on BeforeNetworkChange/AfterNetworkChange to be able to backup conig, check for connectivity, and potentially only call the rollback locally, so there won't be a need to implement the remote management twice. another issue is that the change has to be treated as atomic for all interfaces together, not one by one.
participants (5)
-
Dan Kenigsberg -
Dan Williams -
David Lutterkort -
Itamar Heim -
Mark McLoughlin