[Libvir] Tricky provisioning problem with inactive domains
Adding support for inactive domains was supposed to make everyone's life easier, but as luck would have it, its actually made one thing very much harder. In the virt-inst/virt-manager tools provisioning works something like this: In paravirt case: - Create a guest using an explicit kernel/initrd from the images/xen directory on the install CD - Write a config file to /etc/xen setup to boot using pygrub In fullvirt case: - Create a guest booting directly off a CDROM - Write a config file to /etc/xen setup to boot off the harddisk So in both these cases, the libvirt XML config for the very first boot of the guest is different, from the XML config for subsequent boots. With the new inactive domain support in libvirt & xend, we can't write out config files directly, instead there is the virDomainDefine() API, which calls to appropriate APIs in XenD. And this is where the problem arises: 1. If we call virDomainDefine() to write the long term config, then virDomainStart() will not be using the correct boot method for the initial install. 2. If we call virDomainDefine() to write the initial install config, then virDomainStart() will kick off the install correctly, but on subsequent boots we'll end up booting the installer again instead of the just installed OS. 3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name. 4. Conversely if we use virDomainDefine() to write the config, and then tried to create a one-off guest with virDomainCreate() the latter will fail due to duplicate names. So, thus far the only way out of the trap I can think of is: 1. Use virDomainCreate() to kick off the initial install 2. Poll virDomainLookupByXXX() to watch for this initial guest shutting down 3. Write out the persistent config using virDomainDefine() The big problem with this, is that if the user were to exit virt-manager sometime after the guest install starts, but before step 3, the config for the guest will never be written, even though it has successfully installed. There are two further ideas I've had - both requiring additional APIs in libvirt & probably XenD too. - Make it possible change the boot configuration of an existing guest. This would let us do: 1. Use virDomainDefine() to define a config file suitable for installing the guest, ie using explicit kernel/initrd 2. Use virDomainStart() to kick off the installer 3. Uew new API to change the guest config to remove explicit kernel & initrd config, and add a bootloader for pygrub. Or in HVM case, switch boot order to use harddisk instead of CDROM & detach the CDROM device. - Make it possible to start an existing inactive guest using an alternative one-off configuration. This would let us do: 1. Use virDomainDefine() to define a config file suitable for running the guest during normal operation. 2. Use virDomainStartConfig(xml) to start the guest with a special config suitable for installing the guest. Ultimately I think we do need the means to change arbitrary parts of a guest configuration, so I think the first option would be the preferred approach. The trouble is that I think implementing this would require using the new XenAPI, or adding a number of new methods to the legacy SXPR API which is not really very desirable. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
Daniel P. Berrange wrote:
Adding support for inactive domains was supposed to make everyone's life easier, but as luck would have it, its actually made one thing very much harder. In the virt-inst/virt-manager tools provisioning works something like this:
In paravirt case:
- Create a guest using an explicit kernel/initrd from the images/xen directory on the install CD - Write a config file to /etc/xen setup to boot using pygrub
In fullvirt case:
- Create a guest booting directly off a CDROM - Write a config file to /etc/xen setup to boot off the harddisk
So in both these cases, the libvirt XML config for the very first boot of the guest is different, from the XML config for subsequent boots. With the new inactive domain support in libvirt & xend, we can't write out config files directly, instead there is the virDomainDefine() API, which calls to appropriate APIs in XenD. And this is where the problem arises:
1. If we call virDomainDefine() to write the long term config, then virDomainStart() will not be using the correct boot method for the initial install. 2. If we call virDomainDefine() to write the initial install config, then virDomainStart() will kick off the install correctly, but on subsequent boots we'll end up booting the installer again instead of the just installed OS. 3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name. 4. Conversely if we use virDomainDefine() to write the config, and then tried to create a one-off guest with virDomainCreate() the latter will fail due to duplicate names.
So, thus far the only way out of the trap I can think of is:
1. Use virDomainCreate() to kick off the initial install 2. Poll virDomainLookupByXXX() to watch for this initial guest shutting down 3. Write out the persistent config using virDomainDefine()
The big problem with this, is that if the user were to exit virt-manager sometime after the guest install starts, but before step 3, the config for the guest will never be written, even though it has successfully installed.
There are two further ideas I've had - both requiring additional APIs in libvirt & probably XenD too.
- Make it possible change the boot configuration of an existing guest. This would let us do:
1. Use virDomainDefine() to define a config file suitable for installing the guest, ie using explicit kernel/initrd 2. Use virDomainStart() to kick off the installer 3. Uew new API to change the guest config to remove explicit kernel & initrd config, and add a bootloader for pygrub. Or in HVM case, switch boot order to use harddisk instead of CDROM & detach the CDROM device.
- Make it possible to start an existing inactive guest using an alternative one-off configuration. This would let us do:
1. Use virDomainDefine() to define a config file suitable for running the guest during normal operation. 2. Use virDomainStartConfig(xml) to start the guest with a special config suitable for installing the guest.
Ultimately I think we do need the means to change arbitrary parts of a guest configuration, so I think the first option would be the preferred approach. The trouble is that I think implementing this would require using the new XenAPI, or adding a number of new methods to the legacy SXPR API which is not really very desirable.
I rather like the idea of having both features available, actually. The ability to start a guest with an alternate one-off config (not just a different kernel and initrd, but perhaps different memory, disk, network configs?) strikes me as useful, especially in automated test harness applications. And as you say the ability to programmatically modify the config is also useful. --Hugh
On Mon, Jan 15, 2007 at 02:14:36PM +0000, Daniel P. Berrange wrote:
Adding support for inactive domains was supposed to make everyone's life easier, but as luck would have it, its actually made one thing very much harder. In the virt-inst/virt-manager tools provisioning works something like this:
In paravirt case:
- Create a guest using an explicit kernel/initrd from the images/xen directory on the install CD - Write a config file to /etc/xen setup to boot using pygrub
In fullvirt case:
- Create a guest booting directly off a CDROM - Write a config file to /etc/xen setup to boot off the harddisk
So in both these cases, the libvirt XML config for the very first boot of the guest is different, from the XML config for subsequent boots. With the new inactive domain support in libvirt & xend, we can't write out config files directly, instead there is the virDomainDefine() API, which calls to appropriate APIs in XenD. And this is where the problem arises:
1. If we call virDomainDefine() to write the long term config, then virDomainStart() will not be using the correct boot method for the initial install. 2. If we call virDomainDefine() to write the initial install config, then virDomainStart() will kick off the install correctly, but on subsequent boots we'll end up booting the installer again instead of the just installed OS. 3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name. 4. Conversely if we use virDomainDefine() to write the config, and then tried to create a one-off guest with virDomainCreate() the latter will fail due to duplicate names.
So, thus far the only way out of the trap I can think of is:
1. Use virDomainCreate() to kick off the initial install 2. Poll virDomainLookupByXXX() to watch for this initial guest shutting down 3. Write out the persistent config using virDomainDefine()
The big problem with this, is that if the user were to exit virt-manager sometime after the guest install starts, but before step 3, the config for the guest will never be written, even though it has successfully installed.
There are two further ideas I've had - both requiring additional APIs in libvirt & probably XenD too.
- Make it possible change the boot configuration of an existing guest. This would let us do:
1. Use virDomainDefine() to define a config file suitable for installing the guest, ie using explicit kernel/initrd 2. Use virDomainStart() to kick off the installer 3. Uew new API to change the guest config to remove explicit kernel & initrd config, and add a bootloader for pygrub. Or in HVM case, switch boot order to use harddisk instead of CDROM & detach the CDROM device.
- Make it possible to start an existing inactive guest using an alternative one-off configuration. This would let us do:
1. Use virDomainDefine() to define a config file suitable for running the guest during normal operation. 2. Use virDomainStartConfig(xml) to start the guest with a special config suitable for installing the guest.
Ultimately I think we do need the means to change arbitrary parts of a guest configuration, so I think the first option would be the preferred approach. The trouble is that I think implementing this would require using the new XenAPI, or adding a number of new methods to the legacy SXPR API which is not really very desirable.
Well the core of the problem is that there is 2 set of information, distinct in their value and meaning and we have only one placeholder for both. It seems to me that basically we are overloading the <os> block while we would need a separate <install> block containing the informations needed to install the domain. Why not plit them out, and keep both in the configuration ? It seems it would open the door to: - the user option of restarting the installation of a domain, which with virtualization is certainly a more common operation than reinstalling a bare system - allow to keep track of how and where a given domain was installed and with what data - ease provisioning of a similar domain based on the informations provided in that <install> block Basically we could create the domain configuration with both the <os> and <install> block, the install one for example missing an installation date, suggesting the user to run the installation process instead of the normal boot of the instance. I'm not even sure libvirt would have to be modified for this, it will by default ignore the <install> block, and the application on top of it could just modify the XML before calling Create() . Making libvirt install aware would force an API extenstion though, I'm not 100% sure that's really needed. Did I understood correctly the problem ? I think adding extra informations about the install data is in any case a good idea, especialy when one think in terms of provisionning. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Mon, Jan 15, 2007 at 09:46:37AM -0500, Daniel Veillard wrote:
On Mon, Jan 15, 2007 at 02:14:36PM +0000, Daniel P. Berrange wrote:
Adding support for inactive domains was supposed to make everyone's life easier, but as luck would have it, its actually made one thing very much harder. In the virt-inst/virt-manager tools provisioning works something like this:
In paravirt case:
- Create a guest using an explicit kernel/initrd from the images/xen directory on the install CD - Write a config file to /etc/xen setup to boot using pygrub
In fullvirt case:
- Create a guest booting directly off a CDROM - Write a config file to /etc/xen setup to boot off the harddisk
So in both these cases, the libvirt XML config for the very first boot of the guest is different, from the XML config for subsequent boots. With the new inactive domain support in libvirt & xend, we can't write out config files directly, instead there is the virDomainDefine() API, which calls to appropriate APIs in XenD. And this is where the problem arises:
1. If we call virDomainDefine() to write the long term config, then virDomainStart() will not be using the correct boot method for the initial install. 2. If we call virDomainDefine() to write the initial install config, then virDomainStart() will kick off the install correctly, but on subsequent boots we'll end up booting the installer again instead of the just installed OS. 3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name. 4. Conversely if we use virDomainDefine() to write the config, and then tried to create a one-off guest with virDomainCreate() the latter will fail due to duplicate names.
So, thus far the only way out of the trap I can think of is:
1. Use virDomainCreate() to kick off the initial install 2. Poll virDomainLookupByXXX() to watch for this initial guest shutting down 3. Write out the persistent config using virDomainDefine()
The big problem with this, is that if the user were to exit virt-manager sometime after the guest install starts, but before step 3, the config for the guest will never be written, even though it has successfully installed.
There are two further ideas I've had - both requiring additional APIs in libvirt & probably XenD too.
- Make it possible change the boot configuration of an existing guest. This would let us do:
1. Use virDomainDefine() to define a config file suitable for installing the guest, ie using explicit kernel/initrd 2. Use virDomainStart() to kick off the installer 3. Uew new API to change the guest config to remove explicit kernel & initrd config, and add a bootloader for pygrub. Or in HVM case, switch boot order to use harddisk instead of CDROM & detach the CDROM device.
- Make it possible to start an existing inactive guest using an alternative one-off configuration. This would let us do:
1. Use virDomainDefine() to define a config file suitable for running the guest during normal operation. 2. Use virDomainStartConfig(xml) to start the guest with a special config suitable for installing the guest.
Ultimately I think we do need the means to change arbitrary parts of a guest configuration, so I think the first option would be the preferred approach. The trouble is that I think implementing this would require using the new XenAPI, or adding a number of new methods to the legacy SXPR API which is not really very desirable.
Well the core of the problem is that there is 2 set of information, distinct in their value and meaning and we have only one placeholder for both. It seems to me that basically we are overloading the <os> block while we would need a separate <install> block containing the informations needed to install the domain. Why not plit them out, and keep both in the configuration ? It seems it would open the door to: - the user option of restarting the installation of a domain, which with virtualization is certainly a more common operation than reinstalling a bare system - allow to keep track of how and where a given domain was installed and with what data - ease provisioning of a similar domain based on the informations provided in that <install> block
Yes. The idea with <install> and <os> is ***very good***.
Basically we could create the domain configuration with both the <os> and <install> block, the install one for example missing an installation date, suggesting the user to run the installation process instead of the normal boot of the instance. I'm not even sure libvirt would have to be modified for this, it will by default ignore the <install> block, and the application on top of it could
I think libvirt would have to be modified for this -- at least a separate API which is able to loads data from the <install> block and returns it to libvirt based application. I'd like to use virt-install --config /etc/xen/myvm.xml when I need to re-install the domain.
just modify the XML before calling Create() . Making libvirt install aware would force an API extenstion though, I'm not 100% sure that's really needed.
It's cheap add there this API extenstion for <install> and it's without an impact to the old libvirt API. Our life will be more funny with this API ;-)
Did I understood correctly the problem ? I think adding extra informations about the install data is in any case a good idea, especialy when one think in terms of provisionning.
Yes. Karel -- Karel Zak <kzak@redhat.com>
On Mon, 2007-01-15 at 14:14 +0000, Daniel P. Berrange wrote:
3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name.
I think I'd expect to be able to call virDomainDefine() to change an active domain's config without affecting it until it restarts. Cheers, Mark.
On Mon, Jan 15, 2007 at 08:57:33PM +0000, Mark McLoughlin wrote:
On Mon, 2007-01-15 at 14:14 +0000, Daniel P. Berrange wrote:
3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name.
I think I'd expect to be able to call virDomainDefine() to change an active domain's config without affecting it until it restarts.
In fact a little good news. I've just re-checked Xen 3.0.4 and this does indeed work. You can start a guest, and then change its config at will using virDomainDefine. Its just my hand-crafted 3.0.3 inactive domain driver which doesn't let this work - which I can trivially fix. So this thread looks like it may in fact be a non-issue. Regards, Dan. -- |=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=| |=- Perl modules: http://search.cpan.org/~danberr/ -=| |=- Projects: http://freshmeat.net/~danielpb/ -=| |=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
On Mon, Jan 15, 2007 at 09:22:37PM +0000, Daniel P. Berrange wrote:
On Mon, Jan 15, 2007 at 08:57:33PM +0000, Mark McLoughlin wrote:
On Mon, 2007-01-15 at 14:14 +0000, Daniel P. Berrange wrote:
3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name.
I think I'd expect to be able to call virDomainDefine() to change an active domain's config without affecting it until it restarts.
In fact a little good news. I've just re-checked Xen 3.0.4 and this does indeed work. You can start a guest, and then change its config at will using virDomainDefine. Its just my hand-crafted 3.0.3 inactive domain driver which doesn't let this work - which I can trivially fix. So this thread looks like it may in fact be a non-issue.
Good :-) Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Mon, Jan 15, 2007 at 08:57:33PM +0000, Mark McLoughlin wrote:
On Mon, 2007-01-15 at 14:14 +0000, Daniel P. Berrange wrote:
3. We could just use virDomainCreate() to start installer, and try to use virDomainDefine() to write the long term config - the latter call will fail though because there will already be a running guest with that name.
I think I'd expect to be able to call virDomainDefine() to change an active domain's config without affecting it until it restarts.
I was apparrently completely off base in my previous answer, with a better understanding of the problem now I agree with Mark. This may require to push a change to Xen but really is the sanest behaviour API wise. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
participants (5)
-
Daniel P. Berrange -
Daniel Veillard -
Hugh Brock -
Karel Zak -
Mark McLoughlin