[libvirt] [RFC] Events API
Hi - We have some folks looking into the implementation of events (just VM state transition events, for now) in libvirtd. I've been assuming that events will be XML strings, something like: <event type="domain-state-transition" timestamp="xxxxx"> <domain-id>22</domain-id> <old-state>nostate</old-state> <new-state>running</new-state> </event> where the contents of the <event> element are determined by the event type, and the attributes type and timestamp are required. Originally, I was thinking one would listen for events by registering a callback, something like: int virConnectAddEventHandler(virConnPtr conn, char **filter, virEventHandler handler, void *arg) where filter is either NULL, indicating interest in all events, or else a NULL-terminated vector of event type names, allowing filtering by event type. void handler(conn, const char *eventXML, arg) would be called for each matching event. I'm a little concerned that a vector of event type names isn't really adequate for specifying a filter. Does this need to be more general (XPathString exprs??) But my larger concern is that an asynchronous callback mechanism (as proposed above) assumes the presence of some thread / process from which to make the callbacks. This works fine in the libvirtd context, but not outside of it. For instance, we build a "client only" version of libvirt with ONLY the remote driver, which currently doesn't require pthreads at all. Introducing asynchronous callbacks into the API means pthreads is now required for this. I'm not sure how much requiring this extra thread matters. If it does, we could always define a synchronous delivery mechanism instead. For instance, we could have a virDeliverEvents(conn) call to make the callbacks for any outstanding events. Or we could just dispense with callbacks altogether, and return a readable (pipe) fd from which the client can read events. Comments?? Dave P.S. We'll also need an internal API for sending events (most likely, an extension of the existing virEvent stuff), but let's get the basic shape of the external interface agreed upon first.
On Thu, Sep 18, 2008 at 12:45:07PM -0400, David Lively wrote:
Hi -
We have some folks looking into the implementation of events (just VM state transition events, for now) in libvirtd. I've been assuming that events will be XML strings, something like:
<event type="domain-state-transition" timestamp="xxxxx"> <domain-id>22</domain-id> <old-state>nostate</old-state> <new-state>running</new-state> </event>
where the contents of the <event> element are determined by the event type, and the attributes type and timestamp are required.
Originally, I was thinking one would listen for events by registering a callback, something like: int virConnectAddEventHandler(virConnPtr conn, char **filter, virEventHandler handler, void *arg) where filter is either NULL, indicating interest in all events, or else a NULL-terminated vector of event type names, allowing filtering by event type. void handler(conn, const char *eventXML, arg) would be called for each matching event.
I think I'd prefer to have properly typed event callbacks for each type of event we're dealing with, and to register callbacks against the object being tracked. We also don't want to do transitions on all the states - eg transitions between nostate & running are happening many times a second - its not useful to track that IMHO. Against a virConnectPtr object I'd expect to be able to register to get an event upon - A new domain object coming into existance - A existing domain object going out of existance So, you could register a callback, call Rich's virConnectListAllDomains() once, and then rely on the callbacks from that point onwards to keep your list of domains up2date. So in case of listening for domains: enum { VIR_CONNECT_DOMAIN_EVENT_ADDED, VIR_CONNECT_DOMAIN_EVENT_REMOVED, }; typedef int (*virConnectDomainEventCallback)(virConnectPtr conn, virDomainPtr dom, int event, void *opaque); int virCOnnectDomainEventRegister(virConnectPtr conn, virConnectDomainEventCallback cb, void *opaque); There would eventually be equivalent API for virNetworkPtr objects and virStoragePoolPtr objects, to track addition & removal of them. Against a virDomainPtr object, I'd expect to be able to register to get an event upon the significant state transitions. If we exclude the transition between nonstate & running which just happens far to often to be practical to track, I don't see a need to filter the events further - just have the callback get all events against that domain object. enum { VIR_DOMAIN_EVENT_STARTED VIR_DOMAIN_EVENT_SUSPENDED VIR_DOMAIN_EVENT_RESUMED VIR_DOMAIN_EVENT_STOPPED VIR_DOMAIN_EVENT_SAVED, VIR_DOMAIN_EVENT_RESTORED, }; typedef int (*virDomainLifecycleEventCallback)(virDomainPtr dom, int event, void *opaque); virDomainLifecycleEventRegister(virDomainPtr dom, virDomainLifecycleEventCallback cb, void *opaque);
I'm a little concerned that a vector of event type names isn't really adequate for specifying a filter. Does this need to be more general (XPathString exprs??)
IMHO, XML / xpath is rather overkill for getting lifecycle events.
But my larger concern is that an asynchronous callback mechanism (as proposed above) assumes the presence of some thread / process from which to make the callbacks. ???This works fine in the libvirtd context, but not outside of it. For instance, we build a "client only" version of libvirt with ONLY the remote driver, which currently doesn't require pthreads at all. Introducing asynchronous callbacks into the API means pthreads is now required for this.
I wouldn't want to use threads for this - any application which is structured in such a way as to be able to make use of async events will have some kind of event loop implementation. We merely need to provoide a way to hook libvirt into that event loop. We already have the API defined for this - src/event.h, and have an demonstration impl that the daemon uses qemud/event.c. So we'd want to validate this src/event.h API contract by doing a proof-of concept impl with an external Glib event loop and if it proves sane, then make the event.h file part of the public API. Applications could either craft their event loop impl themselves, or we can provide some add-on pre-built helper libraries with common impls. eg, an optional libvirt-glib library which comes with a pre-built event impl for applications which use glib. Likewise a libvirt-qt helper. That'd cover pretty much all common GUI apps. This approach has been used very successfully by DBus to avoid tieing dbus to a specific event loop impl.
I'm not sure how much requiring this extra thread matters. If it does, we could always define a synchronous delivery mechanism instead. For instance, we could have a virDeliverEvents(conn) call to make the callbacks for any outstanding events. Or we could just dispense with callbacks altogether, and return a readable (pipe) fd from which the client can read events.
I don't like the idea of exposing an FD as I consider this to be internal impl details. Allowing apps to see it will constrain our abilty to change internals. So I prefer to have the applicaiton provide an event loop impl, which we can call into to register FDs or timeout as & when libvirt deems neccessary. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Daniel P. Berrange wrote:
Applications could either craft their event loop impl themselves, or we can provide some add-on pre-built helper libraries with common impls. eg, an optional libvirt-glib library which comes with a pre-built event impl for applications which use glib. Likewise a libvirt-qt helper. That'd cover pretty much all common GUI apps.
Sounds sane. avahi does the same btw, and I like that approach. cheers, Gerd
On Fri, Sep 19, 2008 at 01:14:14PM +0200, Gerd Hoffmann wrote:
Daniel P. Berrange wrote:
Applications could either craft their event loop impl themselves, or we can provide some add-on pre-built helper libraries with common impls. eg, an optional libvirt-glib library which comes with a pre-built event impl for applications which use glib. Likewise a libvirt-qt helper. That'd cover pretty much all common GUI apps.
Sounds sane. avahi does the same btw, and I like that approach.
Avahi's interface is a thin wrapper arounds dbus's interface :-) And indeed the libvirtd's event.c impl fits into Avahi's interface nicely which demonstrates we're consistent in our thinking. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Sep 19, 2008 at 10:54:39AM +0100, Daniel P. Berrange wrote:
Against a virConnectPtr object I'd expect to be able to register to get an event upon
- A new domain object coming into existance - A existing domain object going out of existance
So, you could register a callback, call Rich's virConnectListAllDomains() once, and then rely on the callbacks from that point onwards to keep your list of domains up2date. So in case of listening for domains:
Just a remark but unfortunately that scheme forces a race between the start of the event flow and the return of the list. The way used in the file monitoring API (FAM which I dislike but at least fixed that problem) is that when you register you get a flow of initial events allowing to setup your list of object. Certainly less efficient than single synchronous call but avoid the race. The user code is also simpler because you only use the events to maintain your state. Performance vs. accuracy , the balance is still open for long lived objects like domains though, but as virtualization gets integrated and efficient maybe it's better to play safe. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
On Mon, Sep 22, 2008 at 10:54:58PM +0200, Daniel Veillard wrote:
On Fri, Sep 19, 2008 at 10:54:39AM +0100, Daniel P. Berrange wrote:
Against a virConnectPtr object I'd expect to be able to register to get an event upon
- A new domain object coming into existance - A existing domain object going out of existance
So, you could register a callback, call Rich's virConnectListAllDomains() once, and then rely on the callbacks from that point onwards to keep your list of domains up2date. So in case of listening for domains:
Just a remark but unfortunately that scheme forces a race between the start of the event flow and the return of the list. The way used in the file monitoring API (FAM which I dislike but at least fixed that problem) is that when you register you get a flow of initial events allowing to setup your list of object. Certainly less efficient than single synchronous call but avoid the race. The user code is also simpler because you only use the events to maintain your state. Performance vs. accuracy , the balance is still open for long lived objects like domains though, but as virtualization gets integrated and efficient maybe it's better to play safe.
I'm not sure I understand what you mean by the race - if you call the virConnectListAllDomains() first, and then register for events, then there is a window where you may have missed some domains starting. If you register for events first and then call virConnectListAllDomains() then worst case you see domains that you already know about - you ought not to miss any events. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Tue, Sep 23, 2008 at 09:19:28AM +0100, Daniel P. Berrange wrote:
On Mon, Sep 22, 2008 at 10:54:58PM +0200, Daniel Veillard wrote:
On Fri, Sep 19, 2008 at 10:54:39AM +0100, Daniel P. Berrange wrote:
Against a virConnectPtr object I'd expect to be able to register to get an event upon
- A new domain object coming into existance - A existing domain object going out of existance
So, you could register a callback, call Rich's virConnectListAllDomains() once, and then rely on the callbacks from that point onwards to keep your list of domains up2date. So in case of listening for domains:
Just a remark but unfortunately that scheme forces a race between the start of the event flow and the return of the list. The way used in the file monitoring API (FAM which I dislike but at least fixed that problem) is that when you register you get a flow of initial events allowing to setup your list of object. Certainly less efficient than single synchronous call but avoid the race. The user code is also simpler because you only use the events to maintain your state. Performance vs. accuracy , the balance is still open for long lived objects like domains though, but as virtualization gets integrated and efficient maybe it's better to play safe.
I'm not sure I understand what you mean by the race - if you call the virConnectListAllDomains() first, and then register for events, then there is a window where you may have missed some domains starting. If you register for events first and then call virConnectListAllDomains() then worst case you see domains that you already know about - you ought not to miss any events.
The problem is that you cannot order the events w.r.t. the time where the list was actually made. If you get a foo domain stopped event just after the virConnectListAllDomains() returns with foo running in it, you can't tell if the domain was just stopped and restarted before the list or if the domain is really stopped. Maybe there is another event of foo being started pending but basically you have some uncertainty about the state and the event. You have 2 unsynchronized information flow and no causal dependancies garanteed between the two :-) Keeping a single flow is the only way I think to garantee you always have a coherent state representation on the client considering the kind of API we have. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
Daniel Veillard wrote:
On Tue, Sep 23, 2008 at 09:19:28AM +0100, Daniel P. Berrange wrote:
Against a virConnectPtr object I'd expect to be able to register to get an event upon
- A new domain object coming into existance - A existing domain object going out of existance
So, you could register a callback, call Rich's virConnectListAllDomains() once, and then rely on the callbacks from that point onwards to keep your list of domains up2date. So in case of listening for domains: Just a remark but unfortunately that scheme forces a race between
On Fri, Sep 19, 2008 at 10:54:39AM +0100, Daniel P. Berrange wrote: the start of the event flow and the return of the list. The way used in the file monitoring API (FAM which I dislike but at least fixed that problem) is that when you register you get a flow of initial events allowing to setup your list of object. Certainly less efficient than single synchronous call but avoid the race. The user code is also simpler because you only use the events to maintain your state. Performance vs. accuracy , the balance is still open for long lived objects like domains though, but as virtualization gets integrated and efficient maybe it's better to play safe. I'm not sure I understand what you mean by the race - if you call the virConnectListAllDomains() first, and then register for events, then
On Mon, Sep 22, 2008 at 10:54:58PM +0200, Daniel Veillard wrote: there is a window where you may have missed some domains starting. If you register for events first and then call virConnectListAllDomains() then worst case you see domains that you already know about - you ought not to miss any events.
The problem is that you cannot order the events w.r.t. the time where the list was actually made. If you get a foo domain stopped event just after the virConnectListAllDomains() returns with foo running in it, you can't tell if the domain was just stopped and restarted before the list or if the domain is really stopped. Maybe there is another event of foo being started pending but basically you have some uncertainty about the state and the event. You have 2 unsynchronized information flow and no causal dependancies garanteed between the two :-) Keeping a single flow is the only way I think to garantee you always have a coherent state representation on the client considering the kind of API we have.
But the event flow is ordered and complete right? The domain can't stop and restart without both events being generated. In your example, you would know that the domain list is stale with respect to the stopped domain. If the start message is pending, the 'right' decision is still to ignore the domain list entry for the stopped domain. The domain list can only supplement knowledge for domains that have not been seen in the event stream. Steve
Daniel
On Thu, Sep 18, 2008 at 12:45:07PM -0400, David Lively wrote:
I'm a little concerned that a vector of event type names isn't really adequate for specifying a filter. Does this need to be more general (XPathString exprs??)
I think I'm with Dan on this one. I would start small -- just domains coming & going (unless VirtualIron needs other events). Since there is no limit to the number of API calls we can have in libvirt, add an API call just for registering for these domain events. Instead of trying to overload untyped strings with complicated meanings.
But my larger concern is that an asynchronous callback mechanism (as proposed above) assumes the presence of some thread / process from which to make the callbacks. This works fine in the libvirtd context, but not outside of it. For instance, we build a "client only" version of libvirt with ONLY the remote driver, which currently doesn't require pthreads at all. Introducing asynchronous callbacks into the API means pthreads is now required for this.
I'm not quite sure I follow this -- you mean it introduces pthreads into libvirt or into the caller? As far as I can see, nothing about this would require threads in either. The remote protocol allows event messages to be passed back asynchronously, although the current remote driver wouldn't expect to receive them which might be a problem for backwards/forwards compatibility. (Therefore the remote client must tell the remote server that it can handle asynchronous events, and the remote client must be prepared to talk to a server which cannot understand asynchronous events -- there is an internal feature API you can use for this). Notwithstanding that, you would have to add a client call to poll for events or (better) to expose the file descriptor so that callers can use it in select(2)/poll(2). Actually there are a number of ways to do this -- IIRC dbus does something semi-elegant. Rich. -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones Read my OCaml programming blog: http://camltastic.blogspot.com/ Fedora now supports 68 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
On Fri, Sep 19, 2008 at 11:14:07AM +0100, Richard W.M. Jones wrote:
On Thu, Sep 18, 2008 at 12:45:07PM -0400, David Lively wrote:
I'm a little concerned that a vector of event type names isn't really adequate for specifying a filter. Does this need to be more general (XPathString exprs??)
I think I'm with Dan on this one. I would start small -- just domains coming & going (unless VirtualIron needs other events). Since there is no limit to the number of API calls we can have in libvirt, add an API call just for registering for these domain events. Instead of trying to overload untyped strings with complicated meanings.
But my larger concern is that an asynchronous callback mechanism (as proposed above) assumes the presence of some thread / process from which to make the callbacks. ???This works fine in the libvirtd context, but not outside of it. For instance, we build a "client only" version of libvirt with ONLY the remote driver, which currently doesn't require pthreads at all. Introducing asynchronous callbacks into the API means pthreads is now required for this.
I'm not quite sure I follow this -- you mean it introduces pthreads into libvirt or into the caller? As far as I can see, nothing about this would require threads in either.
The remote protocol allows event messages to be passed back asynchronously, although the current remote driver wouldn't expect to receive them which might be a problem for backwards/forwards compatibility. (Therefore the remote client must tell the remote server that it can handle asynchronous events, and the remote client must be prepared to talk to a server which cannot understand asynchronous events -- there is an internal feature API you can use for this). Notwithstanding that, you would have to add a client call to poll for events or (better) to expose the file descriptor so that callers can use it in select(2)/poll(2).
Making a client poll for events is evil because it prevents the CPU going into low power mode when nothing is happening. Exposing a FD severely limits your flexibility, because libvirt then has no control over when the FD is being monitored - you may well not want it being select'd on all the time - and/or want to change the events being monitored for at various times.
Actually there are a number of ways to do this -- IIRC dbus does something semi-elegant.
Yes, the DBus approach (which is what I followed for the src/events.h API) is the way I like best. It doesn't require libvirt to do tricks internally with threads, and doesn't force a particular event loop impl onto applications, and still allows libvirt full control over just when & how FD is addded to the event loop for monitoring. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Sep 19, 2008 at 11:22:24AM +0100, Daniel P. Berrange wrote:
Making a client poll for events is evil because it prevents the CPU going into low power mode when nothing is happening.
Didn't mean literally poll of course :-)
Exposing a FD severely limits your flexibility, because libvirt then has no control over when the FD is being monitored - you may well not want it being select'd on all the time - and/or want to change the events being monitored for at various times.
Yup, didn't get down to the bottom of your other email. Exposing an FD is also tricky on Windows too (where you have the joys of "WSAWaitForMultipleEvents"), so best to avoid it if possible. If someone (dbus? glib?) has solved this in a platform-independent way already then so much the better. Rich. -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones virt-df lists disk usage of guests without needing to install any software inside the virtual machine. Supports Linux and Windows. http://et.redhat.com/~rjones/virt-df/
On Fri, Sep 19, 2008 at 11:30:01AM +0100, Richard W.M. Jones wrote:
On Fri, Sep 19, 2008 at 11:22:24AM +0100, Daniel P. Berrange wrote:
Making a client poll for events is evil because it prevents the CPU going into low power mode when nothing is happening.
Didn't mean literally poll of course :-)
Exposing a FD severely limits your flexibility, because libvirt then has no control over when the FD is being monitored - you may well not want it being select'd on all the time - and/or want to change the events being monitored for at various times.
Yup, didn't get down to the bottom of your other email. Exposing an FD is also tricky on Windows too (where you have the joys of "WSAWaitForMultipleEvents"), so best to avoid it if possible. If someone (dbus? glib?) has solved this in a platform-independent way already then so much the better.
The glib event loop implementation on Windows is built using the WSAWaitForMultipleEvents API, so any libvirt-glib.so impl of the events.h API would 'just work' in this respect Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, 2008-09-19 at 11:14 +0100, Richard W.M. Jones wrote:
On Thu, Sep 18, 2008 at 12:45:07PM -0400, David Lively wrote:
I'm a little concerned that a vector of event type names isn't really adequate for specifying a filter. Does this need to be more general (XPathString exprs??)
I think I'm with Dan on this one. I would start small -- just domains coming & going (unless VirtualIron needs other events). Since there is no limit to the number of API calls we can have in libvirt, add an API call just for registering for these domain events. Instead of trying to overload untyped strings with complicated meanings.
Okay. I'm fine with a more strongly-typed event protocol. While it's more work to add new classes of events (as compared with extending event XML), that's probably a Good Thing :-)
But my larger concern is that an asynchronous callback mechanism (as proposed above) assumes the presence of some thread / process from which to make the callbacks. This works fine in the libvirtd context, but not outside of it. For instance, we build a "client only" version of libvirt with ONLY the remote driver, which currently doesn't require pthreads at all. Introducing asynchronous callbacks into the API means pthreads is now required for this.
I'm not quite sure I follow this -- you mean it introduces pthreads into libvirt or into the caller? As far as I can see, nothing about this would require threads in either.
I meant that if we expected the callbacks to "just happen", libvirt (at least, the libvirtd-less version) would need to spawn a thread to make the callbacks. But this is quite easily avoided by providing a hook that clients are responsible for calling for event delivery, as suggested. I had considered this as an alternative to the file-descriptor interface (but thought the fd interface was convenient with to use w/select() & poll()). After considering the problems with fds and power-saving and windows compatibility, I agree an event-delivery hook sounds like the best idea, perhaps: int virDeliverEvents(int timeout) where timeout is interpreted as in poll() (i.e., max millisecs to block, 0 means don't block, negative means block forever).
The remote protocol allows event messages to be passed back asynchronously, although the current remote driver wouldn't expect to receive them which might be a problem for backwards/forwards compatibility. (Therefore the remote client must tell the remote server that it can handle asynchronous events, and the remote client must be prepared to talk to a server which cannot understand asynchronous events -- there is an internal feature API you can use for this). Notwithstanding that, you would have to add a client call to poll for events or (better) to expose the file descriptor so that callers can use it in select(2)/poll(2).
Yeah, I expect the remote implementation will be the worst part! Thanks for the pointers. Thanks, Dave
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 David Lively schreef:
Yeah, I expect the remote implementation will be the worst part! Thanks for the pointers.
I already created a complete avahi messaging implementation around libvirt. And to be honest, it is trivial to do so, but who is going to fetch events from the different backends in a better way than polling every second? Stefan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEAREKAAYFAkjT+JwACgkQYH1+F2Rqwn2EUACgiAdnJRA/wkSywJIG6poo1BVZ iOAAnjyYuXYQ0Hou/J312ZF3+meHXK2B =pmSr -----END PGP SIGNATURE-----
On Fri, Sep 19, 2008 at 11:08:37AM -0400, David Lively wrote:
int virDeliverEvents(int timeout)
This isn't really much use because it doesn't integrate with common main loops. I suggest you look at glib to see how it handles main loops and how to integrate with it. Rich. -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones Read my OCaml programming blog: http://camltastic.blogspot.com/ Fedora now supports 68 OCaml packages (the OPEN alternative to F#) http://cocan.org/getting_started_with_ocaml_on_red_hat_and_fedora
Okay - I'm looking at glib docs. You guys are suggesting I create something that's easy to plug into a glib main loop, right? So ... something that's easy to "wrap" with a GSource via g_source_new()? (Stop me now if I'm on the wrong track!) So ... something that plugs into GSourceFuncs elements (in particular prepare/check/dispatch/finalize) easily?? int virEventsNeedDelivering(virConnectionPtr conn) int virEventsDispatch(virConnectionPtr conn) int virEventsCleanup(virConnectionPtr conn) and perhaps: int virEventsRunOnce(virConnectionPtr conn) to wrap them all for non-glib-ish implementations? Sorry if I'm being dense. I've looked at the qemud events interface, but I still feel like I'm missing something here. Dave On Mon, 2008-09-22 at 07:23 +0100, Richard W.M. Jones wrote:
On Fri, Sep 19, 2008 at 11:08:37AM -0400, David Lively wrote:
int virDeliverEvents(int timeout)
This isn't really much use because it doesn't integrate with common main loops. I suggest you look at glib to see how it handles main loops and how to integrate with it.
Rich.
On Mon, Sep 22, 2008 at 11:11:17AM -0400, David Lively wrote:
Okay - I'm looking at glib docs. You guys are suggesting I create something that's easy to plug into a glib main loop, right? So ... something that's easy to "wrap" with a GSource via g_source_new()? (Stop me now if I'm on the wrong track!) So ... something that plugs into GSourceFuncs elements (in particular prepare/check/dispatch/finalize) easily??
int virEventsNeedDelivering(virConnectionPtr conn) int virEventsDispatch(virConnectionPtr conn) int virEventsCleanup(virConnectionPtr conn)
and perhaps: int virEventsRunOnce(virConnectionPtr conn)
to wrap them all for non-glib-ish implementations?
No, this the wrong approach. This is defining an event loop impl - we don't want todo that. We need to define an API to let an application provide a set of callback for libvirt to talk to an existing event loop impl. ie a way for libvirt to register FD watches and timeouts, not a way for apps to manually process libvirt events in this way this example shows. The public API for this is along the lines of that currently defined in the src/events.h file. There are a set of functions libvirt needs in order to register actions for FDs, and/or timeouts. So the public API should consist of a way to register impls of these APIs typedef int (*virEventAddHandleFunc)(int, int, virEventHandleCallback, void *); typedef void (*virEventUpdateHandleFunc)(int, int); typedef int (*virEventRemoveHandleFunc)(int); typedef int (*virEventAddTimeoutFunc)(int, virEventTimeoutCallback, void *); typedef void (*virEventUpdateTimeoutFunc)(int, int); typedef int (*virEventRemoveTimeoutFunc)(int); void virEventRegisterImpl(virEventAddHandleFunc addHandle, virEventUpdateHandleFunc updateHandle, virEventRemoveHandleFunc removeHandle, virEventAddTimeoutFunc addTimeout, virEventUpdateTimeoutFunc updateTimeout, virEventRemoveTimeoutFunc removeTimeout); A separate libvirt-glib.so, would provide a API call virEventRegisterGLib() which calls virEventRegisterImpl() with a suitable implementation for glib. An application like virt-manager which uses glib and wants events would then calll virEventRegisterGLib(). If it had a custom event loop of its own, then it could call virEventRegisterImpl() directly with its special impl. It may be worth making our public API even more closely aligned with dbus - see dbus-connection.h and dbus-server.h - so people writing glue functions for it could just reuse what they've already written for dbus. typedef dbus_bool_t (* DBusAddWatchFunction) (DBusWatch *watch, void *data); typedef void (* DBusWatchToggledFunction) (DBusWatch *watch, void *data); typedef void (* DBusRemoveWatchFunction) (DBusWatch *watch, void *data); typedef dbus_bool_t (* DBusAddTimeoutFunction) (DBusTimeout *timeout, void *data); typedef void (* DBusTimeoutToggledFunction) (DBusTimeout *timeout, void *data); typedef void (* DBusRemoveTimeoutFunction) (DBusTimeout *timeout, void *data); dbus_bool_t dbus_server_set_watch_functions (DBusServer *server, DBusAddWatchFunction add_function, DBusRemoveWatchFunction remove_function, DBusWatchToggledFunction toggled_function, void *data, DBusFreeFunction free_data_function); dbus_bool_t dbus_server_set_timeout_functions (DBusServer *server, DBusAddTimeoutFunction add_function, DBusRemoveTimeoutFunction remove_function, DBusTimeoutToggledFunction toggled_function, void *data, DBusFreeFunction free_data_function); A 'watch' in DBus terminology is a file descriptor monitor. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
Okay, at long last, I see what you mean (I think). Apps using libvirt events must register an EventImpl via virRegisterEventImpl. Internally, suppose we implement events via an anonymous pipe. libvirt would call EventImpl.addHandle(pipe_read_fd, POLLIN ..., __virHandleEvents, conn) so the app's main loop would monitor fd (in whatever manner it chooses), then call __virHandleEvents(fd, conn) when it detected activity. __virHandleEvents would pull the event from the pipe and dispatch handlers as appropriate. We'd call eventImpl.removeHandle(pipe_read_fd) when the domain goes away. Am I finally on the right track?? If so, are you proposing that we simply make the existing src/events.h interface public? At this point, I don't see the need for anything but addHandle and removeHandle (but I can see how the others might be useful to libvirt eventually). Dave On Mon, 2008-09-22 at 16:22 +0100, Daniel P. Berrange wrote:
No, this the wrong approach. This is defining an event loop impl - we don't want todo that. We need to define an API to let an application provide a set of callback for libvirt to talk to an existing event loop impl. ie a way for libvirt to register FD watches and timeouts, not a way for apps to manually process libvirt events in this way this example shows.
The public API for this is along the lines of that currently defined in the src/events.h file.
There are a set of functions libvirt needs in order to register actions for FDs, and/or timeouts. So the public API should consist of a way to register impls of these APIs
typedef int (*virEventAddHandleFunc)(int, int, virEventHandleCallback, void *); typedef void (*virEventUpdateHandleFunc)(int, int); typedef int (*virEventRemoveHandleFunc)(int);
typedef int (*virEventAddTimeoutFunc)(int, virEventTimeoutCallback, void *); typedef void (*virEventUpdateTimeoutFunc)(int, int); typedef int (*virEventRemoveTimeoutFunc)(int);
void virEventRegisterImpl(virEventAddHandleFunc addHandle, virEventUpdateHandleFunc updateHandle, virEventRemoveHandleFunc removeHandle, virEventAddTimeoutFunc addTimeout, virEventUpdateTimeoutFunc updateTimeout, virEventRemoveTimeoutFunc removeTimeout);
A separate libvirt-glib.so, would provide a API call
virEventRegisterGLib()
which calls virEventRegisterImpl() with a suitable implementation for glib. An application like virt-manager which uses glib and wants events would then calll virEventRegisterGLib(). If it had a custom event loop of its own, then it could call virEventRegisterImpl() directly with its special impl.
It may be worth making our public API even more closely aligned with dbus - see dbus-connection.h and dbus-server.h - so people writing glue functions for it could just reuse what they've already written for dbus.
typedef dbus_bool_t (* DBusAddWatchFunction) (DBusWatch *watch, void *data); typedef void (* DBusWatchToggledFunction) (DBusWatch *watch, void *data); typedef void (* DBusRemoveWatchFunction) (DBusWatch *watch, void *data);
typedef dbus_bool_t (* DBusAddTimeoutFunction) (DBusTimeout *timeout, void *data); typedef void (* DBusTimeoutToggledFunction) (DBusTimeout *timeout, void *data); typedef void (* DBusRemoveTimeoutFunction) (DBusTimeout *timeout, void *data);
dbus_bool_t dbus_server_set_watch_functions (DBusServer *server, DBusAddWatchFunction add_function, DBusRemoveWatchFunction remove_function, DBusWatchToggledFunction toggled_function, void *data, DBusFreeFunction free_data_function); dbus_bool_t dbus_server_set_timeout_functions (DBusServer *server, DBusAddTimeoutFunction add_function, DBusRemoveTimeoutFunction remove_function, DBusTimeoutToggledFunction toggled_function, void *data, DBusFreeFunction free_data_function);
A 'watch' in DBus terminology is a file descriptor monitor.
Daniel
On Mon, Sep 22, 2008 at 03:15:25PM -0400, David Lively wrote:
Okay, at long last, I see what you mean (I think). Apps using libvirt events must register an EventImpl via virRegisterEventImpl. Internally, suppose we implement events via an anonymous pipe. libvirt would call
EventImpl.addHandle(pipe_read_fd, POLLIN ..., __virHandleEvents, conn)
so the app's main loop would monitor fd (in whatever manner it chooses), then call __virHandleEvents(fd, conn) when it detected activity. __virHandleEvents would pull the event from the pipe and dispatch handlers as appropriate. We'd call eventImpl.removeHandle(pipe_read_fd) when the domain goes away. Am I finally on the right track??
Well the answer will depend on the particular virtualization driver. For the remote driver (which includes QEMU + LXC + any others running inside the daemon) you already have a file descriptor that can be monitored - the UNIX/TCP socket to the daemon. For the Xen driver we would grab the xenstored UNIX socket file descriptor and register xenstore watches - the FD would become readable when the watch fires. So, yes, the remote driver would register its FD with EventImpl.addHandle. In the GLib<->libvirt event loop binding, this would in turn register the FD with the GLib event loop and a callback wrapper. The GLib<->libvirt event loop binding would detect the neccessary poll condition, invok the the callback associated with that FD, which would then invoke the callback actually regisstered by the remote driver virHandleEvents() is not something that would directly play a part - that's a specific part of the libvirtd daemon's event loop impl.
If so, are you proposing that we simply make the existing src/events.h interface public? At this point, I don't see the need for anything but addHandle and removeHandle (but I can see how the others might be useful to libvirt eventually).
The 'updateHandle' method was essentially an efficiency method - it allows us to change the poll events that a FD is being monitored for without having to register & unregister. This both helps efficiency of the event loop impl, and means the impl can guarentee this operation will never encounter OOM. If we split the virEventRegisterImpl() function into two, in the same way that dbus does - virEventRegisterWatchImpl, virEventRegisterTimeoutImpl then we could just make the watch impl public for now and worry about the timeout impl later if there turns out to be a need for it. Daniel -- |: Red Hat, Engineering, London -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|
On Fri, Sep 19, 2008 at 11:08:37AM -0400, David Lively wrote:
On Fri, 2008-09-19 at 11:14 +0100, Richard W.M. Jones wrote:
On Thu, Sep 18, 2008 at 12:45:07PM -0400, David Lively wrote:
I'm a little concerned that a vector of event type names isn't really adequate for specifying a filter. Does this need to be more general (XPathString exprs??)
I think I'm with Dan on this one. I would start small -- just domains coming & going (unless VirtualIron needs other events). Since there is no limit to the number of API calls we can have in libvirt, add an API call just for registering for these domain events. Instead of trying to overload untyped strings with complicated meanings.
Okay. I'm fine with a more strongly-typed event protocol. While it's more work to add new classes of events (as compared with extending event XML), that's probably a Good Thing :-)
it makes processing event simpler/cheaper, and if you want to scale that's not neglectible :-0
But my larger concern is that an asynchronous callback mechanism (as proposed above) assumes the presence of some thread / process from which to make the callbacks. This works fine in the libvirtd context, but not outside of it. For instance, we build a "client only" version of libvirt with ONLY the remote driver, which currently doesn't require pthreads at all. Introducing asynchronous callbacks into the API means pthreads is now required for this.
I'm not quite sure I follow this -- you mean it introduces pthreads into libvirt or into the caller? As far as I can see, nothing about this would require threads in either.
I meant that if we expected the callbacks to "just happen", libvirt (at least, the libvirtd-less version) would need to spawn a thread to make the callbacks.
But this is quite easily avoided by providing a hook that clients are responsible for calling for event delivery, as suggested. I had considered this as an alternative to the file-descriptor interface (but thought the fd interface was convenient with to use w/select() & poll()).
the fd registration is really not ideal, as Dan and Rich suggested the hook based solution allows cleaner code, and more portability.
After considering the problems with fds and power-saving and windows compatibility, I agree an event-delivery hook sounds like the best idea, perhaps:
int virDeliverEvents(int timeout)
where timeout is interpreted as in poll() (i.e., max millisecs to block, 0 means don't block, negative means block forever).
let's copy the kind of interface DBus provided, at least for consistency :-)
The remote protocol allows event messages to be passed back asynchronously, although the current remote driver wouldn't expect to receive them which might be a problem for backwards/forwards compatibility. (Therefore the remote client must tell the remote server that it can handle asynchronous events, and the remote client must be prepared to talk to a server which cannot understand asynchronous events -- there is an internal feature API you can use for this). Notwithstanding that, you would have to add a client call to poll for events or (better) to expose the file descriptor so that callers can use it in select(2)/poll(2).
Yeah, I expect the remote implementation will be the worst part! Thanks for the pointers.
One thing i would like to make sure for remote is that we can reuse the existing connection, as opening a second connection initiated by the server may give troubles with firewalls. Daniel -- Daniel Veillard | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ daniel@veillard.com | Rpmfind RPM search engine http://rpmfind.net/ http://veillard.com/ | virtualization library http://libvirt.org/
participants (7)
-
Daniel P. Berrange -
Daniel Veillard -
David Lively -
Gerd Hoffmann -
Richard W.M. Jones -
Stefan de Konink -
Steve Ofsthun