4:56 a.m.
I had some private discussions with a developer who wants to start using
libvirt, but requires asynch notifications of domain start/stop (and
other stuff, but this will do for the moment). Right now he's using
XenStore watches. The alternative of polling didn't look very appealing
for him.
A rough API for this might look like this:
typedef int virEventHandle;
/* Register a callback to catch domain start events. */
virEventHandle virEventDomainStart
(virConnectPtr,
void (*f) (virConnectPtr, int domid));
/* Register a callback to catch domain stop events. */
virEventHandle virEventDomainStop
(virConnectPtr,
void (*f) (virConnectPtr, int domid));
/* Unregister any type of callback. */
int virEventCancel (virConnectPtr, virEventHandle handle);
Implementation
--------------
In the Xen case, we can use XenStore watches[1]. Callbacks from
XenStore can be translated directly into event callbacks.
In the qemu case, the daemon knows when the qemu instances that it
manages come and go.
In the qemu/remote case, asynch notifications are nominally supported by
the protocol, but I haven't really looked at all the implications.
Seems like we'd need to use MSG_OOB to signal the client process, and
translate the SIGURG signal received into a callback. This is all
tricky and Unix-specific though so if you have any better ideas ... We
would also need to avoid mixing ordinary request/response with the
notifications, which OOB takes care of.
Rich.
[1] http://wiki.xensource.com/xenwiki/XenBusDriver
--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in
England and Wales under Company Registration No. 03798903
5:30 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Hi,
libvirt, but requires asynch notifications of domain start/stop (and
Notification. Not async notification.
Point is I want avoid waking up every second or so to poll for rare
events. There is no need for the events being truly async.
typedef int virEventHandle;
/* Register a callback to catch domain start events. */
virEventHandle virEventDomainStart (virConnectPtr, void (*f)
(virConnectPtr, int domid));
/* Register a callback to catch domain stop events. */ virEventHandle
virEventDomainStop (virConnectPtr, void (*f) (virConnectPtr, int
domid));
/* Unregister any type of callback. */ int virEventCancel
(virConnectPtr, virEventHandle handle);
Doesn't need to be a callback model.
In the Xen case, we can use XenStore watches[1]. Callbacks from
XenStore can be translated directly into event callbacks.
xenstore doesn't calls you back. xenstore gives you a file handle you
can select() on and a function you can call when data is available to
figure which watch did fire.
You can register a token together with the watch, which in turn you can
use to implement callbacks. But libxenstore itself doesn't callback you.
In the qemu/remote case, asynch notifications are nominally
supported
by the protocol, but I haven't really looked at all the implications.
Seems like we'd need to use MSG_OOB to signal the client process, and
translate the SIGURG signal received into a callback.
I'd like to veto that idea. Calling a callback from a signal handler is
just asking for trouble. It isn't safe to call libc functions from a
signal handler for example. Also libvirt hijacking the SIGURG signal is
not very nice.
We would also need to avoid mixing ordinary request/response with
the
notifications
Why? Just disallowing notifications while a request is in flight should
be enougth I think.
I'd like to have something which integrates nicely with the usual
select() based event loops, like the xenstore model. Sort of "here is a
file handle, and if data comes in call this function." The function in
turn could either call callbacks or return some event struct.
cheers,
Gerd
5:58 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
On Wed, Jul 18, 2007 at 12:30:03PM +0200, Gerd Hoffmann wrote:
Hi,
> libvirt, but requires asynch notifications of domain start/stop (and
Notification. Not async notification.
Point is I want avoid waking up every second or so to poll for rare
events. There is no need for the events being truly async.
> typedef int virEventHandle;
>
> /* Register a callback to catch domain start events. */
> virEventHandle virEventDomainStart (virConnectPtr, void (*f)
> (virConnectPtr, int domid));
>
> /* Register a callback to catch domain stop events. */ virEventHandle
> virEventDomainStop (virConnectPtr, void (*f) (virConnectPtr, int
> domid));
>
> /* Unregister any type of callback. */ int virEventCancel
> (virConnectPtr, virEventHandle handle);
Doesn't need to be a callback model.
In general events are a pain because the integration in the application
is never that clean or simple.
For the callback model to work you need the application to call the library
in some way (to then check the condition and do the callback), to me threading
in the library is not a good working model because then you callback from a
different thread and lot of confusion/races ensue.
> In the Xen case, we can use XenStore watches[1]. Callbacks from
> XenStore can be translated directly into event callbacks.
xenstore doesn't calls you back. xenstore gives you a file handle you
can select() on and a function you can call when data is available to
figure which watch did fire.
You can register a token together with the watch, which in turn you can
use to implement callbacks. But libxenstore itself doesn't callback you.
> In the qemu/remote case, asynch notifications are nominally supported
> by the protocol, but I haven't really looked at all the implications.
> Seems like we'd need to use MSG_OOB to signal the client process, and
> translate the SIGURG signal received into a callback.
I'd like to veto that idea. Calling a callback from a signal handler is
just asking for trouble. It isn't safe to call libc functions from a
signal handler for example. Also libvirt hijacking the SIGURG signal is
not very nice.
> We would also need to avoid mixing ordinary request/response with the
> notifications
Why? Just disallowing notifications while a request is in flight should
be enougth I think.
I'd like to have something which integrates nicely with the usual
select() based event loops, like the xenstore model. Sort of "here is a
file handle, and if data comes in call this function." The function in
turn could either call callbacks or return some event struct.
I did that trick in gamin to write to the fd within the signal to
pass the information in a safe way. Still I would really like to avoid
getting down to the same kind of crazyness. The problem is that not
everything is data down a file/pipe but that's the only way to integrate
decently in an Unix app programming model :-(
I resisted so far adding events in libvirt API precisely to keep the
level of complexity of the API down. If we go there, then an fd based
mechanism with a function called by the user to get the information has
my preference. That way except for the fd everything is synchronous and
purely within the application flow of control. The problem is how to
then feed the fd when "something occurs" especially since the something is
happening in a different process or even a different domain. I have no
clear answer to this, and until I have (and this is still sensible for
the variuous hypervisor) I would rather postpone getting there.
Daniel
--
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard | virtualization library http://libvirt.org/
veillard(a)redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
6:23 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Daniel Veillard wrote:
For the callback model to work you need the application to call the
library
in some way (to then check the condition and do the callback), to me threading
in the library is not a good working model because then you callback from a
different thread and lot of confusion/races ensue.
Yep, in that point threads equally evil like callbacks from signal
handlers. Well, slightly better as you can at least use pthread locking
primitives then. But forcing apps into using threads isn't very nice. I
avoid threads when possible exactly to get around the locking hassle.
I resisted so far adding events in libvirt API precisely to keep
the
level of complexity of the API down. If we go there, then an fd based
mechanism with a function called by the user to get the information has
my preference.
/me too ;)
That way except for the fd everything is synchronous and
purely within the application flow of control. The problem is how to
then feed the fd when "something occurs" especially since the something is
happening in a different process or even a different domain.
What troubling events do you have in mind? For remote you'll have the
socket to select() on. As I far I know the very same protocol is used
for qemu, so you probably have a socket too, right? For xen you can
just use the xenstore fd. Dunno on the network stuff. Who manages the
network? Probably the libvirtd daemon too? Then the notifications can
go through the libvirt <-> libvirtd socket connection, so you have a fd too.
What might be needed is allowing multiple file handles (one socket to
xenstore, one socket to libvirtd) so libvirt can collect events from
different sources. It becomes a bit difficuilt at that point. avahi
has that problem too. They have a "give me functions to register and
unregister filehandles"-style api for that. And some helper functions
to make that easy for apps using the event loops of the usual gui libs
(glib/gtk, qt).
cheers,
Gerd
6:51 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
On Wed, Jul 18, 2007 at 01:23:22PM +0200, Gerd Hoffmann wrote:
> That way except for the fd everything is synchronous and
> purely within the application flow of control. The problem is how to
> then feed the fd when "something occurs" especially since the something
is
> happening in a different process or even a different domain.
What troubling events do you have in mind? For remote you'll have the
socket to select() on. As I far I know the very same protocol is used
for qemu, so you probably have a socket too, right? For xen you can
just use the xenstore fd. Dunno on the network stuff. Who manages the
network? Probably the libvirtd daemon too? Then the notifications can
go through the libvirt <-> libvirtd socket connection, so you have a fd too.
If you give directly the fd of the socket used to talk internally to the
daemon then you must allow for case where there is data to read but no
even will be generated in the function, because since you don't have the
library filtering you can't be sure what's available is something the
application is interested in.
For example if doing a migration a lot of traffic may happen on multiple
sockets before the user see an event he's actually interested into.
There is so much nastyness once you break the purely synchronous model.
What might be needed is allowing multiple file handles (one socket
to
xenstore, one socket to libvirtd) so libvirt can collect events from
different sources. It becomes a bit difficuilt at that point. avahi
has that problem too. They have a "give me functions to register and
unregister filehandles"-style api for that. And some helper functions
to make that easy for apps using the event loops of the usual gui libs
(glib/gtk, qt).
Could be a possibility, the other one is to create one thread internal
to the library but limited purely to doing I/O, still I find this disgusting.
Daniel
--
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard | virtualization library http://libvirt.org/
veillard(a)redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
7:03 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Daniel Veillard wrote:
If you give directly the fd of the socket used to talk internally
to the
daemon then you must allow for case where there is data to read but no
even will be generated in the function, because since you don't have the
library filtering you can't be sure what's available is something the
application is interested in.
I don't see that as a problem. Just process the data and return "no
event for you yet" to the application.
For example if doing a migration a lot of traffic may happen on
multiple
sockets before the user see an event he's actually interested into.
There is so much nastyness once you break the purely synchronous model.
I don't expect to be notified truely async, i.e. while some
virDomMigrate() call is in progress. I just want be able to -- for
example -- get notified about new domains showing up while sitting
around idle without polling once per second. Not sending notifications
while other operations are in progress is fine, just queue them up and
send them once the migration is done.
cheers,
Gerd
8:20 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Hi,
I don't expect to be notified truely async, i.e. while some
virDomMigrate() call is in progress.
Uhm, while thinking about it: Even when not designing the library api
async, it still is at the wire, i.e. you can have a cross over from a
server side notification and a client-side request. Hmm. No easy way
out without opening a separate communication channel for the
notifications ...
cheers,
Gerd
7:11 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Daniel Veillard wrote:
On Wed, Jul 18, 2007 at 01:23:22PM +0200, Gerd Hoffmann wrote:
>> That way except for the fd everything is synchronous and
>> purely within the application flow of control. The problem is how to
>> then feed the fd when "something occurs" especially since the something
is
>> happening in a different process or even a different domain.
> What troubling events do you have in mind? For remote you'll have the
> socket to select() on. As I far I know the very same protocol is used
> for qemu, so you probably have a socket too, right? For xen you can
> just use the xenstore fd. Dunno on the network stuff. Who manages the
> network? Probably the libvirtd daemon too? Then the notifications can
> go through the libvirt <-> libvirtd socket connection, so you have a fd too.
If you give directly the fd of the socket used to talk internally to the
daemon then you must allow for case where there is data to read but no
even will be generated in the function, because since you don't have the
library filtering you can't be sure what's available is something the
application is interested in.
For example if doing a migration a lot of traffic may happen on multiple
sockets before the user see an event he's actually interested into.
There is so much nastyness once you break the purely synchronous model.
I think, specifically with migration, the problem case is:
client -- sends domainMigrate --> source server
migration starts
domain disappears from source server
source server -- sends domainStop event --> client
later, domain appears on destination server
destination server -- sends domainStart event --> client
source server -- sends reply to domainMigrate --> client
The domainStop event from the source server is the problem message. It
appears in between the request and the reply for an ordinary RPC.
This argues for having two separate connections, one for synchronous
RPC, one for events. That in itself is not very pleasant, particularly
if the connections are going over ssh, or if connections are otherwise
expensive to set up or maintain (hello TLS).
Oh and BTW, writing and reading TLS messages does not translate simply
into write(2) and read(2) on the underlying fd. TLS tends to chat
backwards and forwards. The remote driver goes to a lot of trouble to
deal with this.
> What might be needed is allowing multiple file handles (one
socket to
> xenstore, one socket to libvirtd) so libvirt can collect events from
> different sources. It becomes a bit difficuilt at that point. avahi
> has that problem too. They have a "give me functions to register and
> unregister filehandles"-style api for that. And some helper functions
> to make that easy for apps using the event loops of the usual gui libs
> (glib/gtk, qt).
Could be a possibility, the other one is to create one thread internal
to the library but limited purely to doing I/O, still I find this disgusting.
Mmmm, not nice.
Rich.
--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in
England and Wales under Company Registration No. 03798903
6:57 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Gerd Hoffmann wrote:
xenstore doesn't calls you back. xenstore gives you a file
handle you
can select() on and a function you can call when data is available to
figure which watch did fire.
Yes, my mistake. I was -- cough -- reading the kernel interface to
xenbus, not the xenstore userspace interface. As you rightly say the
userspace interface gives you a fd which you have to select() on, see
the C example here: http://wiki.xensource.com/xenwiki/XenStoreReference
> We would also need to avoid mixing ordinary request/response with
the
> notifications
Why? Just disallowing notifications while a request is in flight should
be enougth I think.
I'd like to have something which integrates nicely with the usual
select() based event loops, like the xenstore model. Sort of "here is a
file handle, and if data comes in call this function." The function in
turn could either call callbacks or return some event struct.
I can see this working.
In the Xen (local) case we can hand back the Xenstore fd itself for
applications to monitor.
In the Xen remote case the server side has already got an event loop
which can look for fd events. They'll be sent synchronously to the
client. The client will have to hand out the remote connection fd for
applications to select() on.
In the qemu case I'm not sure how to modify qemu_driver.c to handle
this. One for Dan Berrange to comment on.
Rich.
--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in
England and Wales under Company Registration No. 03798903
9:18 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
On Wed, Jul 18, 2007 at 12:57:37PM +0100, Richard W.M. Jones wrote:
>>We would also need to avoid mixing ordinary request/response
with the
>> notifications
>
>Why? Just disallowing notifications while a request is in flight should
>be enougth I think.
>
>I'd like to have something which integrates nicely with the usual
>select() based event loops, like the xenstore model. Sort of "here is a
>file handle, and if data comes in call this function." The function in
>turn could either call callbacks or return some event struct.
I can see this working.
In the Xen (local) case we can hand back the Xenstore fd itself for
applications to monitor.
In the Xen remote case the server side has already got an event loop
which can look for fd events. They'll be sent synchronously to the
client. The client will have to hand out the remote connection fd for
applications to select() on.
In the qemu case I'm not sure how to modify qemu_driver.c to handle
this. One for Dan Berrange to comment on.
We should not explicitly expose 'the xenstore filehandle', or 'the
libvirtd'
filehandle per-se. Instead we should provide an API which lets application
register callbacks for adding/removing file handles to an event loop.
We in fact already have this - look at the src/event.h file. So if we make
this part of the official public API, then applications merely need provide
an impl of the virEventAddHandleFunc and virEventRemoveHandleFunc. The
underlying libvirt drivers can thus invoke these callbacks as required,
be it with the XenStored filehandle, a XenAPI filehandle, a libvirtd fd
or something else.
We can provide sample impls of the virEventAddHandleFunc and
virEventRemoveHandleFunc code for GLib and/or QT in a libvirtglib.so
and libvirtqt.so add-on libraries. This allows for easy integration
with most GUI apps, while also allowing integration with custom home
brew event loops. This is how DBus provides event loop integration
without tieing itself to a specific impl & it works very nicely.
I outline this previously in the bottom half of this mail.
http://www.redhat.com/archives/libvir-list/2007-May/msg00083.html
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
9:37 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Daniel P. Berrange wrote:
We should not explicitly expose 'the xenstore filehandle', or
'the libvirtd'
filehandle per-se. Instead we should provide an API which lets application
register callbacks for adding/removing file handles to an event loop.
I outline this previously in the bottom half of this mail.
http://www.redhat.com/archives/libvir-list/2007-May/msg00083.html
YES!
That is the way to go I guess ...
cheers,
Gerd
9:45 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Daniel P. Berrange wrote:
Instead we should provide an API which lets application
register callbacks for adding/removing file handles to an event loop.
Understood ...
We in fact already have this - look at the src/event.h file. So if we
make
this part of the official public API, then applications merely need provide
an impl of the virEventAddHandleFunc and virEventRemoveHandleFunc. The
underlying libvirt drivers can thus invoke these callbacks as required,
be it with the XenStored filehandle, a XenAPI filehandle, a libvirtd fd
or something else.
If I understand what you're saying here, you are proposing that libvirt
provides a call like this following (don't worry about specifics):
virEventDomainNotify (virConnectPtr, int state, void (*f)(...));
The implementation of this call is that it sets up a xenstore watch, and
registers the xenstore fd with the previously defined
virEventAddHandleFunc (implemented by the main app / app's event loop code).
When the event happens, xenstore fd gets some data available, app's
event loop triggers this, and calls into libvirt, which then calls the
app's callback function ("f").
Is that right?
In the remote case it's more tricky. What does the remote's callback
function do when it gets an event? (This could happen pretty
asynchronously relative to other communications going on -- eg. in the
middle of a round of TLS chatter). Choices seem to be: (1) Send the
data OOB, unencrypted[1]. (2) Piggy-back the notification on to the
reply to the next call [not asynchronous]. (3) Try to work out where we
are in the TLS message and if necessary queue the event up and wait for
a "safe" place to insert the message in the protocol. (3) seems fairly
do-able.
Rich.
[1] TLS 1.1 includes support for OOB data, but it's not really clear if
GnuTLS supports that.
--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in
England and Wales under Company Registration No. 03798903
10:03 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
On Wed, Jul 18, 2007 at 03:45:10PM +0100, Richard W.M. Jones wrote:
Daniel P. Berrange wrote:
>Instead we should provide an API which lets application
>register callbacks for adding/removing file handles to an event loop.
Understood ...
>We in fact already have this - look at the src/event.h file. So if we make
>this part of the official public API, then applications merely need provide
>an impl of the virEventAddHandleFunc and virEventRemoveHandleFunc. The
>underlying libvirt drivers can thus invoke these callbacks as required,
>be it with the XenStored filehandle, a XenAPI filehandle, a libvirtd fd
>or something else.
If I understand what you're saying here, you are proposing that libvirt
provides a call like this following (don't worry about specifics):
virEventDomainNotify (virConnectPtr, int state, void (*f)(...));
The implementation of this call is that it sets up a xenstore watch, and
registers the xenstore fd with the previously defined
virEventAddHandleFunc (implemented by the main app / app's event loop code).
When the event happens, xenstore fd gets some data available, app's
event loop triggers this, and calls into libvirt, which then calls the
app's callback function ("f").
Is that right?
Yep, that's pretty much spot on.
In the remote case it's more tricky. What does the remote's
callback
function do when it gets an event? (This could happen pretty
asynchronously relative to other communications going on -- eg. in the
middle of a round of TLS chatter). Choices seem to be: (1) Send the
data OOB, unencrypted[1]. (2) Piggy-back the notification on to the
reply to the next call [not asynchronous]. (3) Try to work out where we
are in the TLS message and if necessary queue the event up and wait for
a "safe" place to insert the message in the protocol. (3) seems fairly
do-able.
The daemon has a buffer of outgoing data - and it appends to this buffer
discrete 'XDR' messages. So we should be able to simply append another
XDR message per notification & not have to overly worry about things
being mixed up in the stream. When the client is deserializing replies
during normal calls it'll have to keep an eye out for messages which are
in fact notifications, rather than its expected reply, but again that
should not be too hard since we're dealing with discrete XDR encoded
messages. So AFAICT we don't need any OOB or piggybacking stuff - just
queue up the data to send as normal.
[1] TLS 1.1 includes support for OOB data, but it's not really
clear if
GnuTLS supports that.
Yeah I get the impression that is fairly experimental stuff.
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
10:13 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
Daniel P. Berrange wrote:
The daemon has a buffer of outgoing data - and it appends to this
buffer
discrete 'XDR' messages. So we should be able to simply append another
XDR message per notification & not have to overly worry about things
being mixed up in the stream. When the client is deserializing replies
during normal calls it'll have to keep an eye out for messages which are
in fact notifications, rather than its expected reply, but again that
should not be too hard since we're dealing with discrete XDR encoded
messages. So AFAICT we don't need any OOB or piggybacking stuff - just
queue up the data to send as normal.
Obviously at the moment, remote calls are completely synchronous on the
client side. remote_internal.c:call() is a function which serialises
the args, writes them to the socket, then sits there waiting for a
reply. I guess we're not planning to make this asynchronous? (Or are we??)
Unless we change remote_internal.c, events which happen outside of calls
won't be delivered upwards asynchronously, but only when the app happens
to make a libvirt call.
Rich.
--
Emerging Technologies, Red Hat - http://et.redhat.com/~rjones/
Registered Address: Red Hat UK Ltd, Amberley Place, 107-111 Peascod
Street, Windsor, Berkshire, SL4 1TE, United Kingdom. Registered in
England and Wales under Company Registration No. 03798903
10:16 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
On Wed, Jul 18, 2007 at 04:13:38PM +0100, Richard W.M. Jones wrote:
Daniel P. Berrange wrote:
>The daemon has a buffer of outgoing data - and it appends to this buffer
>discrete 'XDR' messages. So we should be able to simply append another
>XDR message per notification & not have to overly worry about things
>being mixed up in the stream. When the client is deserializing replies
>during normal calls it'll have to keep an eye out for messages which are
>in fact notifications, rather than its expected reply, but again that
>should not be too hard since we're dealing with discrete XDR encoded
>messages. So AFAICT we don't need any OOB or piggybacking stuff - just
>queue up the data to send as normal.
Obviously at the moment, remote calls are completely synchronous on the
client side. remote_internal.c:call() is a function which serialises
the args, writes them to the socket, then sits there waiting for a
reply. I guess we're not planning to make this asynchronous? (Or are we??)
It'll be a little tricky, but I don't think we'd want to make it
asynchronous. If we just adapt it so that if the packet is gets back
is a notification we put it aside & read another packet for its real
reply. Once its got the real reply then it can dispatch any notifications
it queued up.
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
9:46 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
On Wed, Jul 18, 2007 at 03:18:03PM +0100, Daniel P. Berrange wrote:
On Wed, Jul 18, 2007 at 12:57:37PM +0100, Richard W.M. Jones wrote:
> In the qemu case I'm not sure how to modify qemu_driver.c to handle
> this. One for Dan Berrange to comment on.
We should not explicitly expose 'the xenstore filehandle', or 'the
libvirtd'
filehandle per-se. Instead we should provide an API which lets application
register callbacks for adding/removing file handles to an event loop.
We in fact already have this - look at the src/event.h file. So if we make
this part of the official public API, then applications merely need provide
an impl of the virEventAddHandleFunc and virEventRemoveHandleFunc. The
underlying libvirt drivers can thus invoke these callbacks as required,
be it with the XenStored filehandle, a XenAPI filehandle, a libvirtd fd
or something else.
We can provide sample impls of the virEventAddHandleFunc and
virEventRemoveHandleFunc code for GLib and/or QT in a libvirtglib.so
and libvirtqt.so add-on libraries. This allows for easy integration
with most GUI apps, while also allowing integration with custom home
brew event loops. This is how DBus provides event loop integration
without tieing itself to a specific impl & it works very nicely.
Hum, on one hand you have a simple "here is an fd, register it and
call this when there is something to read" and on the other hand there
is an indirect callback architecture, with possibly multiple shared libs.
Isn't this a bit over the top ? Plus I really prefer to have an API
not based on callbacks but on synchronous calls driven by the application.
My own experience is that it is way more comfortable with the later
at least for most programmers (e.g. SAX vs. Reader XML APIs).
Daniel
--
Red Hat Virtualization group http://redhat.com/virtualization/
Daniel Veillard | virtualization library http://libvirt.org/
veillard(a)redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
9:59 a.m.
New subject: [Libvir] Re: Asynchronous notifications of domain start/stop
On Wed, Jul 18, 2007 at 10:46:02AM -0400, Daniel Veillard wrote:
On Wed, Jul 18, 2007 at 03:18:03PM +0100, Daniel P. Berrange wrote:
> On Wed, Jul 18, 2007 at 12:57:37PM +0100, Richard W.M. Jones wrote:
> > In the qemu case I'm not sure how to modify qemu_driver.c to handle
> > this. One for Dan Berrange to comment on.
>
> We should not explicitly expose 'the xenstore filehandle', or 'the
libvirtd'
> filehandle per-se. Instead we should provide an API which lets application
> register callbacks for adding/removing file handles to an event loop.
>
> We in fact already have this - look at the src/event.h file. So if we make
> this part of the official public API, then applications merely need provide
> an impl of the virEventAddHandleFunc and virEventRemoveHandleFunc. The
> underlying libvirt drivers can thus invoke these callbacks as required,
> be it with the XenStored filehandle, a XenAPI filehandle, a libvirtd fd
> or something else.
>
> We can provide sample impls of the virEventAddHandleFunc and
> virEventRemoveHandleFunc code for GLib and/or QT in a libvirtglib.so
> and libvirtqt.so add-on libraries. This allows for easy integration
> with most GUI apps, while also allowing integration with custom home
> brew event loops. This is how DBus provides event loop integration
> without tieing itself to a specific impl & it works very nicely.
Hum, on one hand you have a simple "here is an fd, register it and
call this when there is something to read" and on the other hand there
is an indirect callback architecture, with possibly multiple shared libs.
Isn't this a bit over the top ? Plus I really prefer to have an API
not based on callbacks but on synchronous calls driven by the application.
Because this assumes that a single FD is all you'll ever need to monitor
which isn't neccessarily the case with all drivers. The add/remove handle
funcs let the driver provide the application with FD as & when they are
needed. From what I understand of XenAPI, the way they provide state
notifications requires multiple FDs, and we shouldn't artifically limit
ourselves since there's many other virt systems we still may support in
the future which may have have a single pre-determined 'fd'.
The callbacks really aren't that hard to understand and we already have this
stuff in the internal API since we need it for QEMU driver, so we wouldn't
be adding any new code to libvirt in that regard - merely reusing existing
infrastructure.
My own experience is that it is way more comfortable with the later
at least for most programmers (e.g. SAX vs. Reader XML APIs).
Not exactly a great comparison, as SAX / XML Readers aren't trying to
monitor an arbitrary external dynamically changing state system.
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
9:24 a.m.
On Wed, Jul 18, 2007 at 10:56:42AM +0100, Richard W.M. Jones wrote:
I had some private discussions with a developer who wants to start
using
libvirt, but requires asynch notifications of domain start/stop (and
other stuff, but this will do for the moment). Right now he's using
XenStore watches. The alternative of polling didn't look very appealing
for him.
A rough API for this might look like this:
typedef int virEventHandle;
/* Register a callback to catch domain start events. */
virEventHandle virEventDomainStart
(virConnectPtr,
void (*f) (virConnectPtr, int domid));
/* Register a callback to catch domain stop events. */
virEventHandle virEventDomainStop
(virConnectPtr,
void (*f) (virConnectPtr, int domid));
/* Unregister any type of callback. */
int virEventCancel (virConnectPtr, virEventHandle handle);
Yuk - this is going to give an ever growing number of callbacks - there
are many interesting states besides Start & Start. Pause, resume, save,
restore. Probably want something that takes
void (*virDomainStateNotify)(virDomainPtr dom, int state);
The 'state' value could be any virDomainState constant - though we should
not notify for changes betweeen 'nostate', 'running' and
'blocked' because
they all effectively just mean 'running' and Xen domains ossicilate between
them many times a second. Or maybe define a different set of 'state'
transitions to represent the actiohn triggered, rather than the resulting
state ?
Any you could register with:
virDomainRegisterStateNotify(virDomainPtr, virDomainStateNotify);
virDomainRegisterStateNotify(virDomainPtr);
A pre-requisite of using these notifications would be that the application
had registered an event loop implementation, as per
http://www.redhat.com/archives/libvir-list/2007-May/msg00083.html
See src/event.c and srv/event.h for actual impls we already have.
Dan.
--
|=- Red Hat, Engineering, Emerging Technologies, Boston. +1 978 392 2496 -=|
|=- Perl modules: http://search.cpan.org/~danberr/ -=|
|=- Projects: http://freshmeat.net/~danielpb/ -=|
|=- GnuPG: 7D3B9505 F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 -=|
6318
days inactive
6318
days old
17 comments
4 participants
participants (4)
-
Daniel P. Berrange -
Daniel Veillard -
Gerd Hoffmann -
Richard W.M. Jones