[libvirt] staggered guest startup / startup order
Is there a way that you can set the guests startup order and stagger their startup at boot? Regards, Gerry
On Mon, May 25, 2009 at 11:51:07PM -0400, Gerry Reno wrote:
Is there a way that you can set the guests startup order and stagger their startup at boot?
Hum, no. We only have the autostart flag on the domain but nothing about ordering of starts on a wider basis. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
Quick intro since this is my first post to the list: My name is Michael Ansel, and I am third year Electrical and Computer Engineering / Computer Science double major at Duke University in Durham, North Carolina. I've done a little bit of work off and on over the past several years with virtualization on Linux, and my project/self-inflicted torture for the summer is to put together a stable, object-oriented wrapper for the libvirt ruby bindings so that VM management is extremely easy from within ruby (though... I haven't quite gotten that idea off the ground). Anyways, enough about me and onto the issue at hand: How hard would something like this (staggered boot order) be to add/integrate with an optional, external config file (or as a [true|false|1|2|3|..]. style directive in the main XML)? I'd be interested in putting something together if it is a feasible addition and someone can point me to the general area of the code that needs to be changed. Also, is there an indicator in libvirt for when a domain has finished booting, or would this boot staggering need to be time based? Thanks! Michael On Tue, May 26, 2009 at 5:50 AM, Daniel Veillard <veillard@redhat.com> wrote:
On Mon, May 25, 2009 at 11:51:07PM -0400, Gerry Reno wrote:
Is there a way that you can set the guests startup order and stagger their startup at boot?
Hum, no. We only have the autostart flag on the domain but nothing about ordering of starts on a wider basis.
Daniel
-- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
-- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Quoting Michael Ansel <libvir@anselcomputers.com>: [...]
How hard would something like this (staggered boot order) be to add/integrate with an optional, external config file (or as a
Disclaimer: I have no clue as to how difficult it would be to add or integrate. However, considering that the boot order is most likely dependent on a higher level application (the app hosted in the guest) and it's dependencies, does it really make sense to add that integration to libvirt/libvirtd? To truly get a staggered order, it should really know something about the application preferences etc. I'd argue that this level of capability is something that should be handled by, for instance a cluster resource manager?
[true|false|1|2|3|..]. style directive in the main XML)? I'd be interested in putting something together if it is a feasible addition and someone can point me to the general area of the code that needs to be changed. Also, is there an indicator in libvirt for when a domain has finished booting, or would this boot staggering need to be time
Time based seems way too simplistic since there, in a production environment could be a number of issues determining the time it takes for a specific guest to boot, including the load on the host system. Also, the fact that the guest is done booting in by no means an indication that the application its hosting is ready to be interacted with by one of the dependent guests (example: database recovery for some of the commercial database vendors occurs asynchronously to the completion of the boot process, yet the DB isn't ready for transactions until the recovery is complete, which could be long after the boot process is "complete" from an OS perspective.) // Thomas ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On Tue, May 26, 2009 at 10:11:52AM -0400, Thomas Sjolshagen wrote:
Quoting Michael Ansel <libvir@anselcomputers.com>:
[true|false|1|2|3|..]. style directive in the main XML)? I'd be interested in putting something together if it is a feasible addition and someone can point me to the general area of the code that needs to be changed. Also, is there an indicator in libvirt for when a domain has finished booting, or would this boot staggering need to be time
Time based seems way too simplistic since there, in a production environment could be a number of issues determining the time it takes for a specific guest to boot, including the load on the host system.
Also, the fact that the guest is done booting in by no means an indication that the application its hosting is ready to be interacted with by one of the dependent guests (example: database recovery for some of the commercial database vendors occurs asynchronously to the completion of the boot process, yet the DB isn't ready for transactions until the recovery is complete, which could be long after the boot process is "complete" from an OS perspective.)
Yeah, this is what makes ordered startup of guests a 'feature' that is doomed to fail. It is essentially impossible to determine 'finished booing' from the host OS, and time delays are inevitably going to fail on some percentage of boot attempts. The only reliable way is to have the applications which depend on resources from other guests automatically retry connections, or have something in the guest which probes to see if the external resource is ready before attempting to start the app in question. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
On Tue, May 26, 2009 at 10:11:52AM -0400, Thomas Sjolshagen wrote:
Quoting Michael Ansel <libvir@anselcomputers.com>:
[true|false|1|2|3|..]. style directive in the main XML)? I'd be interested in putting something together if it is a feasible addition and someone can point me to the general area of the code that needs to be changed. Also, is there an indicator in libvirt for when a domain has finished booting, or would this boot staggering need to be time
Time based seems way too simplistic since there, in a production environment could be a number of issues determining the time it takes for a specific guest to boot, including the load on the host system.
Also, the fact that the guest is done booting in by no means an indication that the application its hosting is ready to be interacted with by one of the dependent guests (example: database recovery for some of the commercial database vendors occurs asynchronously to the completion of the boot process, yet the DB isn't ready for transactions until the recovery is complete, which could be long after the boot process is "complete" from an OS perspective.)
Yeah, this is what makes ordered startup of guests a 'feature' that is doomed to fail. It is essentially impossible to determine 'finished booing' from the host OS, and time delays are inevitably going to fail on some percentage of boot attempts. The only reliable way is to have the applications which depend on resources from other guests automatically retry connections, or have something in the guest which probes to see if the external resource is ready before attempting to start the app in question.
Daniel
Just a simple time-based solution would be sufficient. Right now when multiple guests all try to boot at the same time it takes about 4 times as long for all of them to boot up as when you boot them individually. Apparently too much context switching. So if we could check off an autoboot flag and then say +120 seconds, that guest would start booting 120 seconds after the autoboot process started. Even this simple mechansim would help quite a bit. Regards, Gerry
In addition to Dan's reply, operating systems ought to indicate what stage they have reached in the boot process (eg. "BIOS up", "kernel up", "userspace up", "applications started"), perhaps by writing a status code into a register. Some PC BIOSes do this already, writing to IO port 0x80, see for example: http://lkml.indiana.edu/hypermail/linux/kernel/0203.1/0770.html If you are using a watchdog driver, feasibly you might use this to detect when userspace in the guest is active. Anyway, this is worth discussing with the relevant upstream projects, QEMU, IPMI, various watchdog hardware ... Rich. -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://et.redhat.com/~rjones/libguestfs/ See what it can do: http://et.redhat.com/~rjones/libguestfs/recipes.html
Daniel P. Berrange wrote:
On Tue, May 26, 2009 at 10:11:52AM -0400, Thomas Sjolshagen wrote:
Quoting Michael Ansel <libvir@anselcomputers.com>:
[true|false|1|2|3|..]. style directive in the main XML)? I'd be interested in putting something together if it is a feasible addition and someone can point me to the general area of the code that needs to be changed. Also, is there an indicator in libvirt for when a domain has finished booting, or would this boot staggering need to be time
Time based seems way too simplistic since there, in a production environment could be a number of issues determining the time it takes for a specific guest to boot, including the load on the host system.
Also, the fact that the guest is done booting in by no means an indication that the application its hosting is ready to be interacted with by one of the dependent guests (example: database recovery for some of the commercial database vendors occurs asynchronously to the completion of the boot process, yet the DB isn't ready for transactions until the recovery is complete, which could be long after the boot process is "complete" from an OS perspective.)
Yeah, this is what makes ordered startup of guests a 'feature' that is doomed to fail. It is essentially impossible to determine 'finished booing' from the host OS, and time delays are inevitably going to fail on some percentage of boot attempts. The only reliable way is to have the applications which depend on resources from other guests automatically retry connections, or have something in the guest which probes to see if the external resource is ready before attempting to start the app in question.
Daniel
Just a simple time-based solution would be sufficient. Right now when multiple guests all try to boot at the same time it takes about 4 times as long for all of them to boot up as when you boot them individually. Apparently too much context switching. So if we could check off an autoboot flag and then say +120 seconds, that guest would start booting 120 seconds after the autoboot process started. Even this simple mechansim would help quite a bit. Regards, Gerry
On Tue, May 26, 2009 at 12:42:47PM -0400, Gerry Reno wrote:
Just a simple time-based solution would be sufficient. Right now when multiple guests all try to boot at the same time it takes about 4 times as long for all of them to boot up as when you boot them individually. Apparently too much context switching. So if we could check off an autoboot flag and then say +120 seconds, that guest would start booting 120 seconds after the autoboot process started. Even this simple mechansim would help quite a bit.
Now this is a interesting problem :-) It doesn't so much suggest the provision of guest startup ordering, but rather the serialization of guest startup - only if there were no other viable solution elsewhere though. I'd be surprised if context switching were the problem & if it was, then I'd suggest beating up on the kernel guys to fix their schedular. More likely is competition for disk access, and the parallel guest startup causing storm of disk seeks, destroying overall I/O throughput. I would still wonder if this could be solved in the kernel I/O schedular either automatically, or with libvirt setting some tunable knob to tell it to try and batch up I/O for the VM to avoid so much seeking. Of course this is all conjecture without profiling / benchmarks to identify just where the problem is. Regards, Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
participants (6)
-
Daniel P. Berrange -
Daniel Veillard -
Gerry Reno -
Michael Ansel -
Richard W.M. Jones -
Thomas Sjolshagen