I'm looking into creating a driver for the new Xen xl/libxl toolstack (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 release. My first hurdle is deciding whether this should be a new driver or integrated with existing xen-unified driver. Initially I thought a new driver would be a better approach - a clean break from the old code, similar to the xenapi driver. libxenlight is also stateless (no managed domains), which seems like another good argument for a new driver. But libxenlight is really just another interface into the same hypervisor, so in that regard it should be a xen-unified subdriver. There are certainly benefits to the xen-unified subdriver approach, e.g. the existing hypervisor and xenstore subdrivers can be leveraged, the former providing all the capabilities code. But AFAIK, libxenlight and xend should not be used together, so I don't think we would want the xend subdriver activated if libxenlight is detected. Supposedly xl can

be used as a direct replacement for xm, allowing unconditional use of that subdriver. BTW, Ian Jackson responded [2] to some of my questions regarding compatibility between old and new toolstack if you are interested. I'd like to hear other's opinions on a new driver vs. a xen-unified subdriver. Regards, Jim [1] http://lists.xensource.com/archives/html/xen-devel/2009-11/msg00436.html [2] http://lists.xensource.com/archives/html/xen-devel/2010-11/msg00344.html

On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote: > I'm looking into creating a driver for the new Xen xl/libxl toolstack > (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 > release. > > My first hurdle is deciding whether this should be a new driver or > integrated with existing xen-unified driver. Initially I thought a new > driver would be a better approach - a clean break from the old code, > similar to the xenapi driver. libxenlight is also stateless (no managed > domains), which seems like another good argument for a new driver. But > libxenlight is really just another interface into the same hypervisor, > so in that regard it should be a xen-unified subdriver. > Something on the system must be stateful, continually monitoring guests & taking neccessary actions ? eg If XenD isn't used, then what is responsible for restarting guests which crash, or performing core dumps on crashed guests, etc, etc ?

This would have a bearing on how best to design a libvirt driver > There are certainly benefits to the xen-unified subdriver approach, e.g. > the existing hypervisor and xenstore subdrivers can be leveraged, the > former providing all the capabilities code. But AFAIK, libxenlight and > xend should not be used together, so I don't think we would want the > xend subdriver activated if libxenlight is detected. Supposedly xl can > be used as a direct replacement for xm, allowing unconditional use of > that subdriver. > > BTW, Ian Jackson responded [2] to some of my questions regarding > compatibility between old and new toolstack if you are interested. > > I'd like to hear other's opinions on a new driver vs. a xen-unified > subdriver. > Due to the number of revisions of Xen userspace stack, and the need to talk to so many pieces to get an efficient driver, the current Xen unified driver is rather hairy. Particularly if XenD itself is deprecated as a control mechanism, then I'd go for a new standalone driver, that runs from libvirtd context and leverages the standard libvirt storage/network/inteface drivers for non-HV stuff (which I assume libxenlight doesn't cover).

On Fri, 2011-01-21 at 15:48 +0000, Jim Fehlig wrote: > Daniel P. Berrange wrote: > > On Thu, Jan 20, 2011 at 04:49:25PM -0700, Jim Fehlig wrote: > > > >> I'm looking into creating a driver for the new Xen xl/libxl toolstack > >> (aka libxenlight [1]), set to become the default in upcoming Xen 4.1.0 > >> release. > >> > >> My first hurdle is deciding whether this should be a new driver or > >> integrated with existing xen-unified driver. Initially I thought a new > >> driver would be a better approach - a clean break from the old code, > >> similar to the xenapi driver. libxenlight is also stateless (no managed > >> domains), which seems like another good argument for a new driver. But > >> libxenlight is really just another interface into the same hypervisor, > >> so in that regard it should be a xen-unified subdriver. > >> > > > > Something on the system must be stateful, continually monitoring > > guests & taking neccessary actions ? eg If XenD isn't used, then > > what is responsible for restarting guests which crash, or performing > > core dumps on crashed guests, etc, etc ? > > > > Good questions. I have just started looking at the new toolstack, and > frankly don't yet know how this is handled. Adding xen-devel for > comment ... Yes and no, in xen, the hypervisor internal data structures combined with xenstore is the state (pretty much anyway). The purpose of libxenlight is to abstract away these details. However, when a domain is created with libxenlight one must wait for a few events to ensure the proper clean-up of the domain. The events are basically 'domain death' or 'device ejected'. These exist so the toolstack can handle rebooting, core dumping the domain, inserting the next CD image, or whatever. I am not sure how libvirt handles such events normally but you may either integrate with a poll/select based mainloop or have threads or processes with self-pipes handling these. So in summary, state is maintained by the xen system outside of any libvirt driver, but handling the kind of events you mention will require some special work to be done for domains created by libvirt.

> > This would have a bearing on how best to design a libvirt driver > > > > > >> There are certainly benefits to the xen-unified subdriver approach, e.g. > >> the existing hypervisor and xenstore subdrivers can be leveraged, the > >> former providing all the capabilities code. But AFAIK, libxenlight and > >> xend should not be used together, so I don't think we would want the > >> xend subdriver activated if libxenlight is detected. Supposedly xl can > >> be used as a direct replacement for xm, allowing unconditional use of > >> that subdriver. > >> > >> BTW, Ian Jackson responded [2] to some of my questions regarding > >> compatibility between old and new toolstack if you are interested. > >> > >> I'd like to hear other's opinions on a new driver vs. a xen-unified > >> subdriver. > >> > > > > Due to the number of revisions of Xen userspace stack, and the > > need to talk to so many pieces to get an efficient driver, the > > current Xen unified driver is rather hairy. Particularly if > > XenD itself is deprecated as a control mechanism, then I'd > > go for a new standalone driver, that runs from libvirtd context > > and leverages the standard libvirt storage/network/inteface > > drivers for non-HV stuff (which I assume libxenlight doesn't > > cover). > > > > Correct. AFAICT, libxenlight does not cover any host storage or network > management. A libxenlight driver will be a hypervsisor driver only. Yes definitely. We handle attaching and detaching NIC's and block devices. Any other management is orthogonal to libxenlight (as it is for xend). BTW. I would also add my vote for a separate code-base for xend/traditional vs. libxenlight drivers as there is no similarity of interface. The xend HTTP RPC calls are unrelated to the libxl API.

On Fri, 2011-01-21 at 16:44 +0000, Daniel P. Berrange wrote: > On a related note, where does XenAPI fit into this picture ? > Previously XenD would provide (at least some of) the XenAPI > functionality in the open source stack, while I understand > that the XenEnterprise had a separate impl of XenAPI. It > sounds like XenAPI is likely only relevant for XenEnterprise > connectivity in the future. > The xapi toolstack which implements XenAPI is now open source as well and is used in XCP (as well as XenServer).

Ian Campbell wrote: > The xapi toolstack which implements XenAPI is now open source as well > and is used in XCP (as well as XenServer). But there is no intent on putting this toolstack in the traditional Xen releases from xen.org correct?

On Fri, 21 Jan 2011, Daniel P. Berrange wrote: > Ok, so basically the mgmt app that is using the libxenlight API > is taking the place of XenD. Thus in the new scheme libvirtd > would have to handle those kind of events. Right. Libxenlight provides all the functions needed to do that and you can find a reference implementation in xl (tools/libxl/xl_cmdimpl.c). > This is fine because > we already do that kind of thing with our other HV drivers for > QEMU/KVM, UserModeLinux and LXC. Good to hear.

> On a related note, where does XenAPI fit into this picture ? > Previously XenD would provide (at least some of) the XenAPI > functionality in the open source stack, while I understand > that the XenEnterprise had a separate impl of XenAPI. It > sounds like XenAPI is likely only relevant for XenEnterprise > connectivity in the future. Nowadays XAPI (the XenServer toolstack) is Open Source and provides a full feature implementation of the API. I think that users that need XenAPI functionalities are probably best served by using XAPI rather than XenD.

I should also mention that one of the design goals of libxenlight is that no libxenlight users (toolstacks like libvirt) should need to use anything else but libxenlight to perform Xen operations. So you shouldn't need to call any xenctrl functions or deal with xenstore directly.

I don't know much about libvirt but I would write a brand new libxenlight driver.