[libvirt] [RFC] libxenlight driver
I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release. My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver. There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can be used as a direct replacement for xm, allowing unconditional use of that subdriver. BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested. I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver. Regards, Jim [1] http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00436.html [2] http://lists.xensource.com/archives/html/xen-devel/2010-11/msg00344.html
On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release.
My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver.
There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can
Seems to call for a new driver actually,
be used as a direct replacement for xm, allowing unconditional use of that subdriver.
BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested.
I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver.
Regards, Jim
[1] http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00436.html [2] http://lists.xensource.com/archives/html/xen-devel/2010-11/msg00344.html
Hum, licencing would have to be sorted out. If this is only linked into libvirtd then using GPL only bits would be fine I guess but if directly linked into libvirt shared lib then we will have to be careful to select only LGPL libraries. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release.
My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver.
Something on the system must be stateful, continually monitoring guests & taking neccessary actions ? eg If XenD isn't used, then what is responsible for restarting guests which crash, or performing core dumps on crashed guests, etc, etc ? This would have a bearing on how best to design a libvirt driver
There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can be used as a direct replacement for xm, allowing unconditional use of that subdriver.
BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested.
I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver.
Due to the number of revisions of Xen userspace stack, and the need to talk to so many pieces to get an efficient driver, the current Xen unified driver is rather hairy. Particularly if XenD itself is deprecated as a control mechanism, then I'd go for a new standalone driver, that runs from libvirtd context and leverages the standard libvirt storage/network/inteface drivers for non-HV stuff (which I assume libxenlight doesn't cover). Regards, Daniel
Daniel P. Berrange wrote:
On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release.
My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver.
Something on the system must be stateful, continually monitoring guests & taking neccessary actions ? eg If XenD isn't used, then what is responsible for restarting guests which crash, or performing core dumps on crashed guests, etc, etc ?
Good questions. I have just started looking at the new toolstack, and frankly don't yet know how this is handled. Adding xen-devel for comment ...
This would have a bearing on how best to design a libvirt driver
There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can be used as a direct replacement for xm, allowing unconditional use of that subdriver.
BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested.
I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver.
Due to the number of revisions of Xen userspace stack, and the need to talk to so many pieces to get an efficient driver, the current Xen unified driver is rather hairy. Particularly if XenD itself is deprecated as a control mechanism, then I'd go for a new standalone driver, that runs from libvirtd context and leverages the standard libvirt storage/network/inteface drivers for non-HV stuff (which I assume libxenlight doesn't cover).
Correct. AFAICT, libxenlight does not cover any host storage or network management. A libxenlight driver will be a hypervsisor driver only. Regards, Jim
On Fri, 2011-01-21 at 15:48 +0000, Jim Fehlig wrote:
Daniel P. Berrange wrote:
On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release.
My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver.
Something on the system must be stateful, continually monitoring guests & taking neccessary actions ? eg If XenD isn't used, then what is responsible for restarting guests which crash, or performing core dumps on crashed guests, etc, etc ?
Good questions. I have just started looking at the new toolstack, and frankly don't yet know how this is handled. Adding xen-devel for comment ...
Yes and no, in xen, the hypervisor internal data structures combined with xenstore is the state (pretty much anyway). The purpose of libxenlight is to abstract away these details. However, when a domain is created with libxenlight one must wait for a few events to ensure the proper clean-up of the domain. The events are basically 'domain death' or 'device ejected'. These exist so the toolstack can handle rebooting, core dumping the domain, inserting the next CD image, or whatever. I am not sure how libvirt handles such events normally but you may either integrate with a poll/select based mainloop or have threads or processes with self-pipes handling these. So in summary, state is maintained by the xen system outside of any libvirt driver, but handling the kind of events you mention will require some special work to be done for domains created by libvirt.
This would have a bearing on how best to design a libvirt driver
There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can be used as a direct replacement for xm, allowing unconditional use of that subdriver.
BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested.
I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver.
Due to the number of revisions of Xen userspace stack, and the need to talk to so many pieces to get an efficient driver, the current Xen unified driver is rather hairy. Particularly if XenD itself is deprecated as a control mechanism, then I'd go for a new standalone driver, that runs from libvirtd context and leverages the standard libvirt storage/network/inteface drivers for non-HV stuff (which I assume libxenlight doesn't cover).
Correct. AFAICT, libxenlight does not cover any host storage or network management. A libxenlight driver will be a hypervsisor driver only.
Yes definitely. We handle attaching and detaching NIC's and block devices. Any other management is orthogonal to libxenlight (as it is for xend). BTW. I would also add my vote for a separate code-base for xend/traditional vs. libxenlight drivers as there is no similarity of interface. The xend HTTP RPC calls are unrelated to the libxl API. In any case, please keep xen-devel in the distribution list :) Gianni
On Fri, Jan 21, 2011 at 04:33:55PM +0000, Gianni Tedesco wrote:
On Fri, 2011-01-21 at 15:48 +0000, Jim Fehlig wrote:
Daniel P. Berrange wrote:
On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release.
My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver.
Something on the system must be stateful, continually monitoring guests & taking neccessary actions ? eg If XenD isn't used, then what is responsible for restarting guests which crash, or performing core dumps on crashed guests, etc, etc ?
Good questions. I have just started looking at the new toolstack, and frankly don't yet know how this is handled. Adding xen-devel for comment ...
Yes and no, in xen, the hypervisor internal data structures combined with xenstore is the state (pretty much anyway). The purpose of libxenlight is to abstract away these details.
However, when a domain is created with libxenlight one must wait for a few events to ensure the proper clean-up of the domain. The events are basically 'domain death' or 'device ejected'. These exist so the toolstack can handle rebooting, core dumping the domain, inserting the next CD image, or whatever. I am not sure how libvirt handles such events normally but you may either integrate with a poll/select based mainloop or have threads or processes with self-pipes handling these.
So in summary, state is maintained by the xen system outside of any libvirt driver, but handling the kind of events you mention will require some special work to be done for domains created by libvirt.
Ok, so basically the mgmt app that is using the libxenlight API is taking the place of XenD. Thus in the new scheme libvirtd would have to handle those kind of events. This is fine because we already do that kind of thing with our other HV drivers for QEMU/KVM, UserModeLinux and LXC.
This would have a bearing on how best to design a libvirt driver
There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can be used as a direct replacement for xm, allowing unconditional use of that subdriver.
BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested.
I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver.
Due to the number of revisions of Xen userspace stack, and the need to talk to so many pieces to get an efficient driver, the current Xen unified driver is rather hairy. Particularly if XenD itself is deprecated as a control mechanism, then I'd go for a new standalone driver, that runs from libvirtd context and leverages the standard libvirt storage/network/inteface drivers for non-HV stuff (which I assume libxenlight doesn't cover).
Correct. AFAICT, libxenlight does not cover any host storage or network management. A libxenlight driver will be a hypervsisor driver only.
Yes definitely. We handle attaching and detaching NIC's and block devices. Any other management is orthogonal to libxenlight (as it is for xend).
BTW. I would also add my vote for a separate code-base for xend/traditional vs. libxenlight drivers as there is no similarity of interface. The xend HTTP RPC calls are unrelated to the libxl API.
Agreed, now you clarified my question(s) above, I think a dedicated libxenlight driver in libvirtd is the only sensible option. On a related note, where does XenAPI fit into this picture ? Previously XenD would provide (at least some of) the XenAPI functionality in the open source stack, while I understand that the XenEnterprise had a separate impl of XenAPI. It sounds like XenAPI is likely only relevant for XenEnterprise connectivity in the future. Regards, Daniel
On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote:
On a related note, where does XenAPI fit into this picture ? Previously XenD would provide (at least some of) the XenAPI functionality in the open source stack, while I understand that the XenEnterprise had a separate impl of XenAPI. It sounds like XenAPI is likely only relevant for XenEnterprise connectivity in the future.
The xapi toolstack which implements XenAPI is now open source as well and is used in XCP (as well as XenServer). Ian.
Ian Campbell wrote:
On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote:
On a related note, where does XenAPI fit into this picture ? Previously XenD would provide (at least some of) the XenAPI functionality in the open source stack, while I understand that the XenEnterprise had a separate impl of XenAPI. It sounds like XenAPI is likely only relevant for XenEnterprise connectivity in the future.
The xapi toolstack which implements XenAPI is now open source as well and is used in XCP (as well as XenServer).
But there is no intent on putting this toolstack in the traditional Xen releases from xen.org correct? Jim
On Fri, 2011-01-21 at 17:02 +0000, Jim Fehlig wrote:
Ian Campbell wrote:
On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote:
On a related note, where does XenAPI fit into this picture ? Previously XenD would provide (at least some of) the XenAPI functionality in the open source stack, while I understand that the XenEnterprise had a separate impl of XenAPI. It sounds like XenAPI is likely only relevant for XenEnterprise connectivity in the future.
The xapi toolstack which implements XenAPI is now open source as well and is used in XCP (as well as XenServer).
But there is no intent on putting this toolstack in the traditional Xen releases from xen.org correct?
Although XCP and xapi are themselves xen.org projects they are maintained and released separately. The intention is that in the future xapi will use libxl too, but still as an external consumer not part of the main xen hg repository. The xcp and xapi folks hang out on the xen-api@lists.xensource.com list. Ian.
Jim Fehlig writes ("Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver"):
Ian Campbell wrote:
The xapi toolstack which implements XenAPI is now open source as well and is used in XCP (as well as XenServer).
But there is no intent on putting this toolstack in the traditional Xen releases from xen.org correct?
That's right, there isn't. Ian.
On Fri, 21 Jan 2011, Daniel P. Berrange wrote:
Ok, so basically the mgmt app that is using the libxenlight API is taking the place of XenD. Thus in the new scheme libvirtd would have to handle those kind of events.
Right. Libxenlight provides all the functions needed to do that and you can find a reference implementation in xl (tools/libxl/xl_cmdimpl.c).
This is fine because we already do that kind of thing with our other HV drivers for QEMU/KVM, UserModeLinux and LXC.
Good to hear.
On a related note, where does XenAPI fit into this picture ? Previously XenD would provide (at least some of) the XenAPI functionality in the open source stack, while I understand that the XenEnterprise had a separate impl of XenAPI. It sounds like XenAPI is likely only relevant for XenEnterprise connectivity in the future.
Nowadays XAPI (the XenServer toolstack) is Open Source and provides a full feature implementation of the API. I think that users that need XenAPI functionalities are probably best served by using XAPI rather than XenD.
On Fri, 2011-01-21 at 17:01 +0000, Stefano Stabellini wrote:
On Fri, 21 Jan 2011, Daniel P. Berrange wrote:
Ok, so basically the mgmt app that is using the libxenlight API is taking the place of XenD. Thus in the new scheme libvirtd would have to handle those kind of events.
Right. Libxenlight provides all the functions needed to do that and you can find a reference implementation in xl (tools/libxl/xl_cmdimpl.c).
This is fine because we already do that kind of thing with our other HV drivers for QEMU/KVM, UserModeLinux and LXC.
Good to hear.
+1
On a related note, where does XenAPI fit into this picture ? Previously XenD would provide (at least some of) the XenAPI functionality in the open source stack, while I understand that the XenEnterprise had a separate impl of XenAPI. It sounds like XenAPI is likely only relevant for XenEnterprise connectivity in the future.
Nowadays XAPI (the XenServer toolstack) is Open Source and provides a full feature implementation of the API. I think that users that need XenAPI functionalities are probably best served by using XAPI rather than XenD.
Yes agreed, in my personal opinion, XenAPI in xend (and perhaps xend itself) has a perilous future. Gianni
On Fri, 21 Jan 2011, Jim Fehlig wrote:
Daniel P. Berrange wrote:
On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote:
I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release.
My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver.
Something on the system must be stateful, continually monitoring guests & taking neccessary actions ? eg If XenD isn't used, then what is responsible for restarting guests which crash, or performing core dumps on crashed guests, etc, etc ?
Good questions. I have just started looking at the new toolstack, and frankly don't yet know how this is handled. Adding xen-devel for comment ...
xenstore is stateful and the hypervisor keeps a state of the system so there is no need for libxenlight to maintain yet another additional state of the system on top of them: it just relies on the informations already available through them. xl (the minimal C toolstack on top of libxenlight) spawns a daemon per domain to handle domain death events and cdrom events. I should also mention that one of the design goals of libxenlight is that no libxenlight users (toolstacks like libvirt) should need to use anything else but libxenlight to perform Xen operations. So you shouldn't need to call any xenctrl functions or deal with xenstore directly.
This would have a bearing on how best to design a libvirt driver
I don't know much about libvirt but I would write a brand new libxenlight driver.
There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can be used as a direct replacement for xm, allowing unconditional use of that subdriver.
BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested.
I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver.
Due to the number of revisions of Xen userspace stack, and the need to talk to so many pieces to get an efficient driver, the current Xen unified driver is rather hairy. Particularly if XenD itself is deprecated as a control mechanism, then I'd go for a new standalone driver, that runs from libvirtd context and leverages the standard libvirt storage/network/inteface drivers for non-HV stuff (which I assume libxenlight doesn't cover).
Correct. AFAICT, libxenlight does not cover any host storage or network management. A libxenlight driver will be a hypervsisor driver only.
Yep. Feel free to ask any questions you might have on libxenlight to xen-devel!
Stefano Stabellini writes ("Re: [Xen-devel] Re: [libvirt] [RFC] libxenlight driver"):
I should also mention that one of the design goals of libxenlight is that no libxenlight users (toolstacks like libvirt) should need to use anything else but libxenlight to perform Xen operations. So you shouldn't need to call any xenctrl functions or deal with xenstore directly.
This is quite important. We have tried to do this and we have succeeded for the "xl" command's requirements, at least. However since we currently have only one user of libxl, "xl" itself, it is very likely that you will find that there are deficiencies in the libxl API which make it difficult to use properly in libvirt. We would very much like to fix these problems so please report them all to us. We will work with you to make libxl have the API you need so that you can write libvirt properly. Inevitably this will mean that the libvirt adaptation will need the libxl from Xen 4.2, not the Xen 4.1 version which is currently in freeze. However in Xen 4.1 xend is still supported. Please also send your libvirt patches to xen-devel for review, etc.
I don't know much about libvirt but I would write a brand new libxenlight driver.
Yes, absolutely. Having taken a look at the existing code in libvirt for driving xenstore/xen/etc., none of it looks sensibly reuseable. Ian.
participants (8)
-
Daniel P. Berrange -
Daniel Veillard -
Gianni Tedesco -
Ian Campbell -
Ian Campbell -
Ian Jackson -
Jim Fehlig -
Stefano Stabellini