[libvirt] RFC: virInterface change transaction API
I've been asked to implement what some people have termed as a "transaction-oriented" API for host interface configuration (ie virInterface*()). The basic intent is to allow rollback to a known-good config if anything goes wrong when changing around the host network config with virInterface*() functions. The most straightforward way to achieve this is that prior to calling virInterfaceDefine/virInterfaceUndefine, the current state of the host's network configuration (ie the /etc/sysconfig/network-scripts/ifcfg-* files in the case of Fedora and RHEL) would be saved off somewhere, and kept around until we're sure the new config is good; once we know that, we can just eliminate the backup. If, however, the user of virInterface*() explicitly requests, we could copy the files back; alternately if the system is rebooted without these known-good files being erased, we would assume that something went wrong and restore the original config. As with all other virInterface functions, the details of all this will be handled by netcf (and below), but since libvirt is the main consumer of netcf, I figure this is the appropriate place to discuss how it gets done, so please let me know any opinions on any piece of this. I plan to start the implementation "soon", as I want to be finished before the end of May. I see 3 layers to this: 1) libvirt At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf: virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags); For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback). 2) netcf The netcf api will have these same three APIs, just named slightly differently: ncf_change_start(struct netcf *ncf, unsigned int flags); There are two possibilities for this. Either: A) call the initscript described below to save all config files that might possibly be changed (snapshot_config) or B) set a flag in *ncf indicating that all future calls to netcf that would end up modifying a particular config file should save off that file *if it hasn't already been saved*. (A) is simpler, but relies on the initscript having exact/complete matching knowledge of what files netcf may change. Should we worry about that and deal with the complexities of (B), or is (A) good enough for now? ncf_change_rollback(struct netcf *ncf, unsigned int flags); Again, two possbilities: A) a) save the config of all current interfaces (in memory) b) call the initscript below to restore the config to its original state. c) compare the new config to the old, and: * bring down any interfaces that no longer exist (PROBLEM: once an interface has no config files, you can no longer operate on it with "ifdown") * bounce any interfaces that have changed * bring up any interfaces that have been re-added or B) a) ifdown all interfaces b) call initscript to restore previous config (rollback_config) c) ifup all interfaces. (A) is much simpler, but may lead to unnecessary difficulties when we bounce interfaces that didn't really need it. So, the same question oas for ncf_change_start() - is the more exact operation worth the extra complexity? ncf_change_commit(struct netcf *ncf, unsigned int flags); The simplest function - this will just call the initscript to erase the backup (commit_config). 3) initscript This initscript will at first live in (be installed by) netcf (called /etc/init.d/networking-config?), but hopefully it will eventually be accepted by the initscripts package (which includes the networking-related initscripts), as it is of general use. (Dan Kenigsberg already already took a stab at this script last year, but received no reply from the initscripts maintainers, implying they may not be too keen on the idea right now - it might take some convincing ;-) https://fedorahosted.org/pipermail/initscripts-devel/2010-February/000025.ht... It will have three commands, one of which will be called automatically by "start" (the command called automatically at boot time): snapshot_config This will save a copy of (what the script believes are - is this problematic?) all network-config related files. It may or may not be called by netcf (see the notes in ncf_start_change() above. If this function finds that a snapshot has already been taken, it should fail. rollback_config (automatically called from "start" at boottime) This will move back (from the saved copies) all files that were changed/removed since snapshot, *and delete any files that have been added*. Note that this command doesn't need to worry about ifup/ifdown, because it will be called prior to any other networking startup (part of the reason that netcf will need to deal with that). I notice that Dan K's version saves the modified files to a "rollback-${date}" directory. Does this seem like a good idea? It's nice to not lose anything, but there is no provision for eliminating old versions, so it could grow without bound. commit_config This will just remove all the files in the save directory. So, the two problems I have right now: 1) Do we accept the inexact method of just saving all files that match a list of patterns during *start(), then in *rollback() erasing all files matching that pattern and copying the old file back? Or do we need to keep track of what files have been changed/removed and added, and copy back / delete only those files during rollback? (A version control system would keep track of this rather nicely, but that's too complex for something that's intended to be a failsafe (and that we would also like to eventually be in the base OS install). Dan B. at one point suggested using patchfiles if I wanted the save info to keep exact track of which files would need to be replaced/deleted on rollback, but on further thought this turns out to not be workable, since we would need to run diff (to create the patchfile) after all changes had been made, and any outside changes to any of the files would leave the patchfile un-appliable, thus causing our "failsafe" to fail :-( ). Therefore, we will need to rely on the list of globs to tell us what files need to be deleted, or keep our own list in a separate file.) 2) Is it going to be okay to ifdown all interfaces prior to the rollback, and ifup all interfaces afterwards? Or must we compare the new config to the original, and ifdown only those interfaces that had been previously added/changed, then ifup only those interfaces that had been previously removed/changed? 3) If anyone has ideas on making the initscript more palatable to the initscripts people, please speak up! :-) (one comment from an initscripts person was that 1) for the general case it would be difficult to draw the line on what parts of network connectivity should be included in this rollback functionality, and 2) at some point this becomes a general system config problem, and would really be better addressed by a general system wide config management system. These are both concerns that need well qualified answers. (I tend to think that this is intended as a failsafe to prevent unreachable systems, so it should be as simple as possible, and thus shouldn't be burdened with the complexity of a full system config management system (which could also co-exist at a higher level), but better answers are welcome.)
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote:
I've been asked to implement what some people have termed as a "transaction-oriented" API for host interface configuration (ie virInterface*()). The basic intent is to allow rollback to a known-good config if anything goes wrong when changing around the host network config with virInterface*() functions.
The most straightforward way to achieve this is that prior to calling virInterfaceDefine/virInterfaceUndefine, the current state of the host's network configuration (ie the /etc/sysconfig/network-scripts/ifcfg-* files in the case of Fedora and RHEL) would be saved off somewhere, and kept around until we're sure the new config is good; once we know that, we can just eliminate the backup. If, however, the user of virInterface*() explicitly requests, we could copy the files back; alternately if the system is rebooted without these known-good files being erased, we would assume that something went wrong and restore the original config.
As with all other virInterface functions, the details of all this will be handled by netcf (and below), but since libvirt is the main consumer of netcf, I figure this is the appropriate place to discuss how it gets done, so please let me know any opinions on any piece of this. I plan to start the implementation "soon", as I want to be finished before the end of May.
I like the idea, and think that virtInterface* users will benefit from it. Few comments are inline.
I see 3 layers to this:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback).
When such a feature is added, we should make it dependent on FLAG_AUTO_ROLLBACK passed to ChangeStart. Higher levels on the management stack may want full controll over when rollback happens.
2) netcf
The netcf api will have these same three APIs, just named slightly differently:
ncf_change_start(struct netcf *ncf, unsigned int flags);
There are two possibilities for this. Either:
A) call the initscript described below to save all config files that might possibly be changed (snapshot_config)
or
B) set a flag in *ncf indicating that all future calls to netcf that would end up modifying a particular config file should save off that file *if it hasn't already been saved*.
(A) is simpler, but relies on the initscript having exact/complete matching knowledge of what files netcf may change. Should we worry about that and deal with the complexities of (B), or is (A) good enough for now?
I'm fine with gradual changes, so (A) is good enough for me, particularly as long as netcf is shipping the initscripts (3).
ncf_change_rollback(struct netcf *ncf, unsigned int flags);
Again, two possbilities:
A) a) save the config of all current interfaces (in memory) b) call the initscript below to restore the config to its original state. c) compare the new config to the old, and: * bring down any interfaces that no longer exist (PROBLEM: once an interface has no config files, you can no longer operate on it with "ifdown") * bounce any interfaces that have changed * bring up any interfaces that have been re-added or
B) a) ifdown all interfaces b) call initscript to restore previous config (rollback_config) c) ifup all interfaces.
(A) is much simpler, but may lead to unnecessary difficulties when we bounce interfaces that didn't really need it. So, the same question oas for ncf_change_start() - is the more exact operation worth the extra complexity?
(A) is simpler? Or did you mean (B)? I'm slightly worried about (B) causing disconnection of completely unrelated interfaces; this may break concurrent applications, or even the user of netcf.
ncf_change_commit(struct netcf *ncf, unsigned int flags);
The simplest function - this will just call the initscript to erase the backup (commit_config).
3) initscript
This initscript will at first live in (be installed by) netcf (called /etc/init.d/networking-config?), but hopefully it will eventually be accepted by the initscripts package (which includes the networking-related initscripts), as it is of general use. (Dan Kenigsberg already already took a stab at this script last year, but received no reply from the initscripts maintainers, implying they may not be too keen on the idea right now - it might take some convincing ;-)
https://fedorahosted.org/pipermail/initscripts-devel/2010-February/000025.ht...
It will have three commands, one of which will be called automatically by "start" (the command called automatically at boot time):
snapshot_config
This will save a copy of (what the script believes are - is this problematic?) all network-config related files. It may or may not be called by netcf (see the notes in ncf_start_change() above.
If this function finds that a snapshot has already been taken, it should fail.
rollback_config (automatically called from "start" at boottime)
This will move back (from the saved copies) all files that were changed/removed since snapshot, *and delete any files that have been added*.
Note that this command doesn't need to worry about ifup/ifdown, because it will be called prior to any other networking startup (part of the reason that netcf will need to deal with that).
I notice that Dan K's version saves the modified files to a "rollback-${date}" directory. Does this seem like a good idea? It's nice to not lose anything, but there is no provision for eliminating old versions, so it could grow without bound.
I sleep better at night when there are backups... Obviously, I should not have kept them beyond a certain limit (last 20). And I'd understand if you think that it is the business of a backup system, or conf management system, to take theses backups.
commit_config
This will just remove all the files in the save directory.
So, the two problems I have right now:
1) Do we accept the inexact method of just saving all files that match a list of patterns during *start(), then in *rollback() erasing all files matching that pattern and copying the old file back? Or do we need to keep track of what files have been changed/removed and added, and copy back / delete only those files during rollback?
(A version control system would keep track of this rather nicely, but that's too complex for something that's intended to be a failsafe (and that we would also like to eventually be in the base OS install). Dan B. at one point suggested using patchfiles if I wanted the save info to keep exact track of which files would need to be replaced/deleted on rollback, but on further thought this turns out to not be workable, since we would need to run diff (to create the patchfile) after all changes had been made, and any outside changes to any of the files would leave the patchfile un-appliable, thus causing our "failsafe" to fail :-( ). Therefore, we will need to rely on the list of globs to tell us what files need to be deleted, or keep our own list in a separate file.)
2) Is it going to be okay to ifdown all interfaces prior to the rollback, and ifup all interfaces afterwards? Or must we compare the new config to the original, and ifdown only those interfaces that had been previously added/changed, then ifup only those interfaces that had been previously removed/changed?
3) If anyone has ideas on making the initscript more palatable to the initscripts people, please speak up! :-) (one comment from an initscripts person was that 1) for the general case it would be difficult to draw the line on what parts of network connectivity should be included in this rollback functionality, and 2) at some point this becomes a general system config problem, and would really be better addressed by a general system wide config management system. These are both concerns that need well qualified answers. (I tend to think that this is intended as a failsafe to prevent unreachable systems, so it should be as simple as possible, and thus shouldn't be burdened with the complexity of a full system config management system (which could also co-exist at a higher level), but better answers are welcome.)
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On 04/14/2011 09:03 AM, Dan Kenigsberg wrote:
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback). When such a feature is added, we should make it dependent on FLAG_AUTO_ROLLBACK passed to ChangeStart. Higher levels on the management stack may want full controll over when rollback happens.
So are you saying that when they don't add FLAG_AUTO_ROLLBACK that the originals should be available for rollback via virInterfaceChangeRollback(), but that if the system is rebooted, they will be lost? Or do you mean that the originals will still be available for rollback, but the initscript will notice a flag in the save directory and skip the rollback at boot time (leaving the originals there for later manual rollback)? (I think you mean the latter, but want to make sure) Should there also be a "-f" flag to force saving new originals, even if something is already there? Or is this too dangerous, and we should require the user to call virInterfaceChangeCommit() to erase the old originals?
2) netcf
The netcf api will have these same three APIs, just named slightly differently:
ncf_change_start(struct netcf *ncf, unsigned int flags);
There are two possibilities for this. Either:
A) call the initscript described below to save all config files that might possibly be changed (snapshot_config)
or
B) set a flag in *ncf indicating that all future calls to netcf that would end up modifying a particular config file should save off that file *if it hasn't already been saved*.
(A) is simpler, but relies on the initscript having exact/complete matching knowledge of what files netcf may change. Should we worry about that and deal with the complexities of (B), or is (A) good enough for now? I'm fine with gradual changes, so (A) is good enough for me, particularly as long as netcf is shipping the initscripts (3).
Okay. I will use the "copy/erase/restore everything matching a glob" approach for now.
ncf_change_rollback(struct netcf *ncf, unsigned int flags);
Again, two possbilities:
A) a) save the config of all current interfaces (in memory) b) call the initscript below to restore the config to its original state. c) compare the new config to the old, and: * bring down any interfaces that no longer exist (PROBLEM: once an interface has no config files, you can no longer operate on it with "ifdown") * bounce any interfaces that have changed * bring up any interfaces that have been re-added or
B) a) ifdown all interfaces b) call initscript to restore previous config (rollback_config) c) ifup all interfaces.
(A) is much simpler, but may lead to unnecessary difficulties when we bounce interfaces that didn't really need it. So, the same question oas for ncf_change_start() - is the more exact operation worth the extra complexity?
(A) is simpler? Or did you mean (B)? I'm slightly worried about (B) causing disconnection of completely unrelated interfaces; this may break concurrent applications, or even the user of netcf.
Sorry, I meant to swap (A) and (B) before I sent so that I was consistently putting simpler things first and more complicated things later. It sounds like in addition to saving the files, I need to save a list of interfaces during ncf_change_start; during ncf_change_rollback, I can: 1) get a list of interfaces as they are after the changes about to be rolled back. 2) use a diff of the two lists to ifdown/undefine the interfaces that will be "un-added" 3) use a "uniq -d" of the lists to get a list of all the interfaces that have just been changed (or not touched at all), and save the current XML for all of those interfaces 4) restore the files 5) Get a new list of interfaces, and for each: 5a) if it's not in the list from (3), ifup the interface 5b) if it is in the list from (3), compare the restored XML to modified ifdown/ifup the interface*only* if the two XMLs differ. Does that sound workable, or will there still be potential for problems (eg, if somebody bridged a host interface, then rolled back, what would happen?)
ncf_change_commit(struct netcf *ncf, unsigned int flags);
The simplest function - this will just call the initscript to erase the backup (commit_config).
3) initscript
This initscript will at first live in (be installed by) netcf (called /etc/init.d/networking-config?), but hopefully it will eventually be accepted by the initscripts package (which includes the networking-related initscripts), as it is of general use. (Dan Kenigsberg already already took a stab at this script last year, but received no reply from the initscripts maintainers, implying they may not be too keen on the idea right now - it might take some convincing ;-)
https://fedorahosted.org/pipermail/initscripts-devel/2010-February/000025.ht...
It will have three commands, one of which will be called automatically by "start" (the command called automatically at boot time):
snapshot_config
This will save a copy of (what the script believes are - is this problematic?) all network-config related files. It may or may not be called by netcf (see the notes in ncf_start_change() above.
If this function finds that a snapshot has already been taken, it should fail.
rollback_config (automatically called from "start" at boottime)
This will move back (from the saved copies) all files that were changed/removed since snapshot, *and delete any files that have been added*.
Note that this command doesn't need to worry about ifup/ifdown, because it will be called prior to any other networking startup (part of the reason that netcf will need to deal with that).
I notice that Dan K's version saves the modified files to a "rollback-${date}" directory. Does this seem like a good idea? It's nice to not lose anything, but there is no provision for eliminating old versions, so it could grow without bound.
I sleep better at night when there are backups... Obviously, I should not have kept them beyond a certain limit (last 20). And I'd understand if you think that it is the business of a backup system, or conf management system, to take theses backups.
I'm agnostic. Anyone else have an opinion? Does there need to be a method in the API to get to these backups? Or are they just there for manual intervention in the case of a catastrophe?
commit_config
This will just remove all the files in the save directory.
So, the two problems I have right now:
I guess the answers to the following questions are: 1) it's okay to take the easy road. 2) We have to do it the hard way.
1) Do we accept the inexact method of just saving all files that match a list of patterns during *start(), then in *rollback() erasing all files matching that pattern and copying the old file back? Or do we need to keep track of what files have been changed/removed and added, and copy back / delete only those files during rollback?
(A version control system would keep track of this rather nicely, but that's too complex for something that's intended to be a failsafe (and that we would also like to eventually be in the base OS install). Dan B. at one point suggested using patchfiles if I wanted the save info to keep exact track of which files would need to be replaced/deleted on rollback, but on further thought this turns out to not be workable, since we would need to run diff (to create the patchfile) after all changes had been made, and any outside changes to any of the files would leave the patchfile un-appliable, thus causing our "failsafe" to fail :-( ). Therefore, we will need to rely on the list of globs to tell us what files need to be deleted, or keep our own list in a separate file.)
2) Is it going to be okay to ifdown all interfaces prior to the rollback, and ifup all interfaces afterwards? Or must we compare the new config to the original, and ifdown only those interfaces that had been previously added/changed, then ifup only those interfaces that had been previously removed/changed?
3) If anyone has ideas on making the initscript more palatable to the initscripts people, please speak up! :-) (one comment from an initscripts person was that 1) for the general case it would be difficult to draw the line on what parts of network connectivity should be included in this rollback functionality, and 2) at some point this becomes a general system config problem, and would really be better addressed by a general system wide config management system. These are both concerns that need well qualified answers. (I tend to think that this is intended as a failsafe to prevent unreachable systems, so it should be as simple as possible, and thus shouldn't be burdened with the complexity of a full system config management system (which could also co-exist at a higher level), but better answers are welcome.)
-- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Mon, Apr 18, 2011 at 12:25:21PM -0400, Laine Stump wrote:
On 04/14/2011 09:03 AM, Dan Kenigsberg wrote:
3) initscript
This initscript will at first live in (be installed by) netcf (called /etc/init.d/networking-config?), but hopefully it will eventually be accepted by the initscripts package (which includes the networking-related initscripts), as it is of general use. (Dan Kenigsberg already already took a stab at this script last year, but received no reply from the initscripts maintainers, implying they may not be too keen on the idea right now - it might take some convincing ;-)
https://fedorahosted.org/pipermail/initscripts-devel/2010-February/000025.ht...
It will have three commands, one of which will be called automatically by "start" (the command called automatically at boot time):
snapshot_config
This will save a copy of (what the script believes are - is this problematic?) all network-config related files. It may or may not be called by netcf (see the notes in ncf_start_change() above.
If this function finds that a snapshot has already been taken, it should fail.
rollback_config (automatically called from "start" at boottime)
This will move back (from the saved copies) all files that were changed/removed since snapshot, *and delete any files that have been added*.
Note that this command doesn't need to worry about ifup/ifdown, because it will be called prior to any other networking startup (part of the reason that netcf will need to deal with that).
I notice that Dan K's version saves the modified files to a "rollback-${date}" directory. Does this seem like a good idea? It's nice to not lose anything, but there is no provision for eliminating old versions, so it could grow without bound. I sleep better at night when there are backups... Obviously, I should not have kept them beyond a certain limit (last 20). And I'd understand if you think that it is the business of a backup system, or conf management system, to take theses backups.
I'm agnostic. Anyone else have an opinion? Does there need to be a method in the API to get to these backups? Or are they just there for manual intervention in the case of a catastrophe?
I think we're getting into overkill myself. Best to concentrate of getting the basic functionality present & working. If someone decides we need to extend this work to handle historical snapshots of the interface config we can deal with it later. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Thu, Apr 14, 2011 at 04:03:03PM +0300, Dan Kenigsberg wrote:
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote:
I've been asked to implement what some people have termed as a "transaction-oriented" API for host interface configuration (ie virInterface*()). The basic intent is to allow rollback to a known-good config if anything goes wrong when changing around the host network config with virInterface*() functions.
The most straightforward way to achieve this is that prior to calling virInterfaceDefine/virInterfaceUndefine, the current state of the host's network configuration (ie the /etc/sysconfig/network-scripts/ifcfg-* files in the case of Fedora and RHEL) would be saved off somewhere, and kept around until we're sure the new config is good; once we know that, we can just eliminate the backup. If, however, the user of virInterface*() explicitly requests, we could copy the files back; alternately if the system is rebooted without these known-good files being erased, we would assume that something went wrong and restore the original config.
As with all other virInterface functions, the details of all this will be handled by netcf (and below), but since libvirt is the main consumer of netcf, I figure this is the appropriate place to discuss how it gets done, so please let me know any opinions on any piece of this. I plan to start the implementation "soon", as I want to be finished before the end of May.
I like the idea, and think that virtInterface* users will benefit from it. Few comments are inline.
I see 3 layers to this:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback).
When such a feature is added, we should make it dependent on FLAG_AUTO_ROLLBACK passed to ChangeStart. Higher levels on the management stack may want full controll over when rollback happens.
I don't think a AUTO_ROLLBACK flag is sufficient. You'd almost certainly want to pass some info such as hostname/port number to test, and perhaps a test timeout in milliseconds, and perhaps a retry count. I suggest a separate API that is invoked just after starting the TXN, to provide auto-rollback data. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Apr 18, 2011 at 05:53:05PM +0100, Daniel P. Berrange wrote:
On Thu, Apr 14, 2011 at 04:03:03PM +0300, Dan Kenigsberg wrote:
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote:
I've been asked to implement what some people have termed as a "transaction-oriented" API for host interface configuration (ie virInterface*()). The basic intent is to allow rollback to a known-good config if anything goes wrong when changing around the host network config with virInterface*() functions.
The most straightforward way to achieve this is that prior to calling virInterfaceDefine/virInterfaceUndefine, the current state of the host's network configuration (ie the /etc/sysconfig/network-scripts/ifcfg-* files in the case of Fedora and RHEL) would be saved off somewhere, and kept around until we're sure the new config is good; once we know that, we can just eliminate the backup. If, however, the user of virInterface*() explicitly requests, we could copy the files back; alternately if the system is rebooted without these known-good files being erased, we would assume that something went wrong and restore the original config.
As with all other virInterface functions, the details of all this will be handled by netcf (and below), but since libvirt is the main consumer of netcf, I figure this is the appropriate place to discuss how it gets done, so please let me know any opinions on any piece of this. I plan to start the implementation "soon", as I want to be finished before the end of May.
I like the idea, and think that virtInterface* users will benefit from it. Few comments are inline.
I see 3 layers to this:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback).
When such a feature is added, we should make it dependent on FLAG_AUTO_ROLLBACK passed to ChangeStart. Higher levels on the management stack may want full controll over when rollback happens.
I don't think a AUTO_ROLLBACK flag is sufficient. You'd almost certainly want to pass some info such as hostname/port number to test, and perhaps a test timeout in milliseconds, and perhaps a retry count.
Yes, that's why I wanted complete control over auto rollback, when it is inroduced.
I suggest a separate API that is invoked just after starting the TXN, to provide auto-rollback data.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 04/19/2011 10:59 AM, Dan Kenigsberg wrote:
On Mon, Apr 18, 2011 at 05:53:05PM +0100, Daniel P. Berrange wrote:
On Thu, Apr 14, 2011 at 04:03:03PM +0300, Dan Kenigsberg wrote:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback). When such a feature is added, we should make it dependent on FLAG_AUTO_ROLLBACK
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote: passed to ChangeStart. Higher levels on the management stack may want full controll over when rollback happens. I don't think a AUTO_ROLLBACK flag is sufficient. You'd almost certainly want to pass some info such as hostname/port number to test, and perhaps a test timeout in milliseconds, and perhaps a retry count. Yes, that's why I wanted complete control over auto rollback, when it is inroduced.
What about the rollback that takes place at boot time if the changes haven't been committed? Should that *always* happen, or should it be controllable by flags as well. I had intended that at least that automatic rollback would be present in the first implementation...
On Tue, Apr 19, 2011 at 03:35:10PM -0400, Laine Stump wrote:
On 04/19/2011 10:59 AM, Dan Kenigsberg wrote:
On Mon, Apr 18, 2011 at 05:53:05PM +0100, Daniel P. Berrange wrote:
On Thu, Apr 14, 2011 at 04:03:03PM +0300, Dan Kenigsberg wrote:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback). When such a feature is added, we should make it dependent on FLAG_AUTO_ROLLBACK
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote: passed to ChangeStart. Higher levels on the management stack may want full controll over when rollback happens. I don't think a AUTO_ROLLBACK flag is sufficient. You'd almost certainly want to pass some info such as hostname/port number to test, and perhaps a test timeout in milliseconds, and perhaps a retry count. Yes, that's why I wanted complete control over auto rollback, when it is inroduced.
What about the rollback that takes place at boot time if the changes haven't been committed? Should that *always* happen, or should it be controllable by flags as well. I had intended that at least that automatic rollback would be present in the first implementation...
Somewhat related, I think the definition of and test for a successful configuration should be left to the application developer. We could provide some prepackaged tests, e.g., ping some host, connect to some host on some port, but I think the definition of working connectivity can be subtle, and I don't think we should impose one definition, or even a set of definitions, on all applications. Dave
On Tue, Apr 19, 2011 at 03:35:10PM -0400, Laine Stump wrote:
On 04/19/2011 10:59 AM, Dan Kenigsberg wrote:
On Mon, Apr 18, 2011 at 05:53:05PM +0100, Daniel P. Berrange wrote:
On Thu, Apr 14, 2011 at 04:03:03PM +0300, Dan Kenigsberg wrote:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback). When such a feature is added, we should make it dependent on FLAG_AUTO_ROLLBACK
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote: passed to ChangeStart. Higher levels on the management stack may want full controll over when rollback happens. I don't think a AUTO_ROLLBACK flag is sufficient. You'd almost certainly want to pass some info such as hostname/port number to test, and perhaps a test timeout in milliseconds, and perhaps a retry count. Yes, that's why I wanted complete control over auto rollback, when it is inroduced.
What about the rollback that takes place at boot time if the changes haven't been committed? Should that *always* happen, or should it be controllable by flags as well. I had intended that at least that automatic rollback would be present in the first implementation...
Rollback on reboot should always happen. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Fri, Apr 08, 2011 at 03:31:05PM -0400, Laine Stump wrote:
I see 3 layers to this:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback).
It helps to outline the usage scenarios for these APIs. The first scenario is where the app is running locally, and thus able to use the rollback API: a) Local app virInterfaceChangeStart n * virInterface{Define,Undefine,Start,Destroy} if (...app determined check to see if everything works..) virInterfaceChangeCommit else virInterfaceChangeRollback b) Remote app with rollback-on-reboot virInterfaceChangeStart n * virInterface{Define,Undefine,Start,Destroy} if (...app determined check to see if everything works..) virInterfaceChangeCommit else invoke remote power switch/fence agent to reboot ...init process does equivalent of virInterfaceChangeRollback c) Remote app with libvirt based auto-rollback virInterfaceChangeStart virInterfaceAutoRollbackTest("somehostname", someportnumber, somemillisecs) n * virInterface{Define,Undefine,Start,Destroy} virInterfaceChangeCommit (if they messed up, virInterfaceChangeCommit would have failed due to network failure, and after somemillisecs auto rollback test will trigger virInterfaceChangeRollback The virInterfaceAutoRollbackTest API is something I assume we'd build into libvirtd in the next stage of work after the basic APIs are done.
2) netcf
The netcf api will have these same three APIs, just named slightly differently:
ncf_change_start(struct netcf *ncf, unsigned int flags);
There are two possibilities for this. Either:
A) call the initscript described below to save all config files that might possibly be changed (snapshot_config)
or
B) set a flag in *ncf indicating that all future calls to netcf that would end up modifying a particular config file should save off that file *if it hasn't already been saved*.
(A) is simpler, but relies on the initscript having exact/complete matching knowledge of what files netcf may change. Should we worry about that and deal with the complexities of (B), or is (A) good enough for now?
I don't think it matters a huge amount.
ncf_change_rollback(struct netcf *ncf, unsigned int flags);
Again, two possbilities:
A) a) save the config of all current interfaces (in memory) b) call the initscript below to restore the config to its original state. c) compare the new config to the old, and: * bring down any interfaces that no longer exist (PROBLEM: once an interface has no config files, you can no longer operate on it with "ifdown") * bounce any interfaces that have changed * bring up any interfaces that have been re-added
Yes, this doesn't really work, because you need to determine what has changed *before* restoring the config, so that you can have a chance of doing 'ifdown' while the config still actually exists.
B) a) ifdown all interfaces b) call initscript to restore previous config (rollback_config) c) ifup all interfaces.
(A) is much simpler, but may lead to unnecessary difficulties when we bounce interfaces that didn't really need it. So, the same question oas for ncf_change_start() - is the more exact operation worth the extra complexity?
To me A) seems more complex than B). The problem with B) is that if you have a multi-homed network, you may be taking down interfaces that where never changed.
ncf_change_commit(struct netcf *ncf, unsigned int flags);
The simplest function - this will just call the initscript to erase the backup (commit_config).
3) initscript
This initscript will at first live in (be installed by) netcf (called /etc/init.d/networking-config?), but hopefully it will eventually be accepted by the initscripts package (which includes the networking-related initscripts), as it is of general use. (Dan Kenigsberg already already took a stab at this script last year, but received no reply from the initscripts maintainers, implying they may not be too keen on the idea right now - it might take some convincing ;-)
https://fedorahosted.org/pipermail/initscripts-devel/2010-February/000025.ht...
It will have three commands, one of which will be called automatically by "start" (the command called automatically at boot time):
snapshot_config
This will save a copy of (what the script believes are - is this problematic?) all network-config related files. It may or may not be called by netcf (see the notes in ncf_start_change() above.
If this function finds that a snapshot has already been taken, it should fail.
rollback_config (automatically called from "start" at boottime)
This will move back (from the saved copies) all files that were changed/removed since snapshot, *and delete any files that have been added*.
Note that this command doesn't need to worry about ifup/ifdown, because it will be called prior to any other networking startup (part of the reason that netcf will need to deal with that).
I notice that Dan K's version saves the modified files to a "rollback-${date}" directory. Does this seem like a good idea? It's nice to not lose anything, but there is no provision for eliminating old versions, so it could grow without bound.
commit_config
This will just remove all the files in the save directory.
So, the two problems I have right now:
1) Do we accept the inexact method of just saving all files that match a list of patterns during *start(), then in *rollback() erasing all files matching that pattern and copying the old file back? Or do we need to keep track of what files have been changed/removed and added, and copy back / delete only those files during rollback?
(A version control system would keep track of this rather nicely, but that's too complex for something that's intended to be a failsafe (and that we would also like to eventually be in the base OS install). Dan B. at one point suggested using patchfiles if I wanted the save info to keep exact track of which files would need to be replaced/deleted on rollback, but on further thought this turns out to not be workable, since we would need to run diff (to create the patchfile) after all changes had been made, and any outside changes to any of the files would leave the patchfile un-appliable, thus causing our "failsafe" to fail :-( ). Therefore, we will need to rely on the list of globs to tell us what files need to be deleted, or keep our own list in a separate file.)
2) Is it going to be okay to ifdown all interfaces prior to the rollback, and ifup all interfaces afterwards? Or must we compare the new config to the original, and ifdown only those interfaces that had been previously added/changed, then ifup only those interfaces that had been previously removed/changed?
As long as that is hidden as a private impl detail, it is not critical. Long term I think you really only want to touch interfaces you actually changed. eg, if you messed up eth0 config (used for libvirtd access) but did not mess up eth1 (used for guest traffic), we don't really want to screw up all guest networking to repair eth0. (NB assuming eth1 and eth0 are separate LANs, so you couldn't just connect to libvirtd on eth1 and issue a virIntefaceChangeRollback call)
3) If anyone has ideas on making the initscript more palatable to the initscripts people, please speak up! :-) (one comment from an initscripts person was that 1) for the general case it would be difficult to draw the line on what parts of network connectivity should be included in this rollback functionality, and 2) at some point this becomes a general system config problem, and would really be better addressed by a general system wide config management system. These are both concerns that need well qualified answers. (I tend to think that this is intended as a failsafe to prevent unreachable systems, so it should be as simple as possible, and thus shouldn't be burdened with the complexity of a full system config management system (which could also co-exist at a higher level), but better answers are welcome.)
We have a well defined set of files which we need to cope with so I think the scope is pretty clear. Trying to solve the entire OS system config problem is just insanity. We want to be able to ensure connectivity in as simple a manner as possible. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Going back over this based on the comments I received... I will begin implementing it this week, starting from the bottom, by adapting the initscript that Dan Kenigsberg already wrote: https://fedorahosted.org/pipermail/initscripts-devel/2010-February/000025.ht... On 04/08/2011 03:31 PM, Laine Stump wrote:
I've been asked to implement what some people have termed as a "transaction-oriented" API for host interface configuration (ie virInterface*()). The basic intent is to allow rollback to a known-good config if anything goes wrong when changing around the host network config with virInterface*() functions.
The most straightforward way to achieve this is that prior to calling virInterfaceDefine/virInterfaceUndefine, the current state of the host's network configuration (ie the /etc/sysconfig/network-scripts/ifcfg-* files in the case of Fedora and RHEL) would be saved off somewhere, and kept around until we're sure the new config is good; once we know that, we can just eliminate the backup. If, however, the user of virInterface*() explicitly requests, we could copy the files back; alternately if the system is rebooted without these known-good files being erased, we would assume that something went wrong and restore the original config.
As with all other virInterface functions, the details of all this will be handled by netcf (and below), but since libvirt is the main consumer of netcf, I figure this is the appropriate place to discuss how it gets done, so please let me know any opinions on any piece of this. I plan to start the implementation "soon", as I want to be finished before the end of May.
I see 3 layers to this:
1) libvirt
At the libvirt layer, this feature just requires 3 new APIs, which are directly passed through to netcf:
virInterfaceChangeStart(virConnectPtr conn, unsigned int flags); virInterfaceChangeCommit(virConnectPtr conn, unsigned int flags); virInterfaceChangeRollback(virConnectPtr conn, unsigned int flags);
For the initial implementation, these will be simple passthroughs to similarly named netcf functions. (in the future, it would be useful for the server side of libvirt to determine if client<->server connectivity was lost due to the network changes, and automatically tell netcf to do a rollback).
I think the libvirt layer can stand as described.
2) netcf
The netcf api will have these same three APIs, just named slightly differently:
ncf_change_start(struct netcf *ncf, unsigned int flags);
There are two possibilities for this. Either:
A) call the initscript described below to save all config files that might possibly be changed (snapshot_config)
Until/unless a solid problem is discovered that prevents it, I will implement option A.
or
B) set a flag in *ncf indicating that all future calls to netcf that would end up modifying a particular config file should save off that file *if it hasn't already been saved*.
(A) is simpler, but relies on the initscript having exact/complete matching knowledge of what files netcf may change. Should we worry about that and deal with the complexities of (B), or is (A) good enough for now?
ncf_change_rollback(struct netcf *ncf, unsigned int flags);
Again, two possbilities:
A) a) save the config of all current interfaces (in memory) b) call the initscript below to restore the config to its original state. c) compare the new config to the old, and: * bring down any interfaces that no longer exist (PROBLEM: once an interface has no config files, you can no longer operate on it with "ifdown") * bounce any interfaces that have changed * bring up any interfaces that have been re-added or
B) a) ifdown all interfaces b) call initscript to restore previous config (rollback_config) c) ifup all interfaces.
(note that I had intended to switch A and B above, so that the following comment made sense). I will initially implement (B), since it is very simple and will get things going, but will plan to switch it to (A) prior to commit, since there seems to be well-founded concerns that (B) may cause avoidable problems in some situations.
(A) is much simpler, but may lead to unnecessary difficulties when we bounce interfaces that didn't really need it. So, the same question oas for ncf_change_start() - is the more exact operation worth the extra complexity?
ncf_change_commit(struct netcf *ncf, unsigned int flags);
The simplest function - this will just call the initscript to erase the backup (commit_config).
3) initscript
This initscript will at first live in (be installed by) netcf (called /etc/init.d/networking-config?), but hopefully it will eventually be accepted by the initscripts package (which includes the networking-related initscripts), as it is of general use. (Dan Kenigsberg already already took a stab at this script last year, but received no reply from the initscripts maintainers, implying they may not be too keen on the idea right now - it might take some convincing ;-)
https://fedorahosted.org/pipermail/initscripts-devel/2010-February/000025.ht...
It will have three commands, one of which will be called automatically by "start" (the command called automatically at boot time):
snapshot_config
This will save a copy of (what the script believes are - is this problematic?) all network-config related files. It may or may not be called by netcf (see the notes in ncf_start_change() above.
If this function finds that a snapshot has already been taken, it should fail.
rollback_config (automatically called from "start" at boottime)
This will move back (from the saved copies) all files that were changed/removed since snapshot, *and delete any files that have been added*.
Note that this command doesn't need to worry about ifup/ifdown, because it will be called prior to any other networking startup (part of the reason that netcf will need to deal with that).
I notice that Dan K's version saves the modified files to a "rollback-${date}" directory. Does this seem like a good idea? It's nice to not lose anything, but there is no provision for eliminating old versions, so it could grow without bound.
commit_config
This will just remove all the files in the save directory.
Dan's script saves old versions of config with no limit. I will modify this to keep only a few old versions, so that it won't grow without limit.
participants (4)
-
Dan Kenigsberg -
Daniel P. Berrange -
Dave Allan -
Laine Stump