Howdy, I'm about to checkin some XML-RPC code into CVS. The code I'm checking in won't compile by default (for now) as there's still a little work to do but I wanted to get something into the tree. Here are the tasks left: 1) Convert from dict to s-expressions to handle config returns. Right now, the Xend XML-RPC code returns S-Expressions in the form of a tuple. We just need a little code to convert from these tuples to struct sexpr's. 2) Integrate my unit tests. I've got a number of unit tests that stress the various bits of parsing code. I'd like to add a make check that runs these unit tests and a make fullcheck that runs these unit tests under valgrind. 3) Support XML-RPC over a domain socket. Right now I'm using nanohttp which doesn't appear to allow different transports. There is some HTTP PUT code already in the tree so I could use that if there isn't a more clever solution. 4) Plumb the xend backend functions to XML-RPC. This is really straight forward. It's just a matter of converting: int xend_pause(virConnectPtr xend, const char *name) { return xend_op(xend, name, "op", "pause", NULL); } To: int xend_pause(virConnectPtr xend, const char *name) { if (xend->xmlrpc) return xmlRpcCall(xend->xmlrpc, "xend.domain.pause", "", "s", name) else return xend_op(xend, name, "op", "pause", NULL); } Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions. For the S-Expression/HTTP protocol, you just get a 501 and have to return a meaningless error. With XML-RPC, those exceptions are actually serialized and sent over the wire. We may want to explore how we can make that information available to the user. My current understanding, btw, is that the basic XML-RPC support for Xend is going into the tree sometime this week (Ewan is looking at the code now) and will be a part of the 3.0.2 release. This means that for the 3.0.3 cycle, we'll be defining a standard API. This process will be starting soon (within the next couple of weeks hopefully). Regards, Anthony Liguori
On Fri, Mar 10, 2006 at 10:57:59AM -0600, Anthony Liguori wrote:
Howdy,
Evening ;-)
I'm about to checkin some XML-RPC code into CVS. The code I'm checking
okay got it. Please add a Changelog entry when you commit to the base :-)
in won't compile by default (for now) as there's still a little work to do but I wanted to get something into the tree. Here are the tasks left:
1) Convert from dict to s-expressions to handle config returns. Right now, the Xend XML-RPC code returns S-Expressions in the form of a tuple. We just need a little code to convert from these tuples to struct sexpr's.
2) Integrate my unit tests. I've got a number of unit tests that stress the various bits of parsing code. I'd like to add a make check that runs these unit tests and a make fullcheck that runs these unit tests under valgrind.
3) Support XML-RPC over a domain socket. Right now I'm using nanohttp which doesn't appear to allow different transports. There is some HTTP PUT code already in the tree so I could use that if there isn't a more clever solution.
4) Plumb the xend backend functions to XML-RPC. This is really straight forward. It's just a matter of converting:
int xend_pause(virConnectPtr xend, const char *name) { return xend_op(xend, name, "op", "pause", NULL); }
To:
int xend_pause(virConnectPtr xend, const char *name) { if (xend->xmlrpc) return xmlRpcCall(xend->xmlrpc, "xend.domain.pause", "", "s", name) else return xend_op(xend, name, "op", "pause", NULL); }
Okay the steps needed look fairly clear, thanks :-)
Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions.
the new error code tries at least to extract the error message when an HTTP POST or GEt fails with an error code, but the XML-RPC should give a far more reliable framework for error handling.
For the S-Expression/HTTP protocol, you just get a 501 and have to return a meaningless error. With XML-RPC, those exceptions are actually serialized and sent over the wire. We may want to explore how we can make that information available to the user.
I would say as a buffer at least that's no problem. But exposing the structure of the error stack on the server may be a bit too much.
My current understanding, btw, is that the basic XML-RPC support for Xend is going into the tree sometime this week (Ewan is looking at the code now) and will be a part of the 3.0.2 release. This means that for
That's waht I was told too :-)
the 3.0.3 cycle, we'll be defining a standard API. This process will be starting soon (within the next couple of weeks hopefully).
Excellent ! Good good ! Hopefully we will be able to confront that with what is present in other engines like l4 and QEmu and make sure at least for the 'common' APIs the semantic is similar. Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Fri, 10 Mar 2006 17:21:52 -0500, Daniel Veillard wrote:
okay got it. Please add a Changelog entry when you commit to the base :-) Will do.
int xend_pause(virConnectPtr xend, const char *name) { if (xend->xmlrpc) return xmlRpcCall(xend->xmlrpc, "xend.domain.pause", "", "s", name) else return xend_op(xend, name, "op", "pause", NULL); }
Okay the steps needed look fairly clear, thanks :-)
One thing I should say is that I'm still contemplating changing the XML-RPC interface in Xend to make the libvirt a little easier..
Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions.
the new error code tries at least to extract the error message when an HTTP POST or GEt fails with an error code, but the XML-RPC should give a far more reliable framework for error handling.
If I recall correctly, the S-Expression/HTTP errors are always 501 Internal Server error with no extended error message. Of course, I may be wrong.
For the S-Expression/HTTP protocol, you just get a 501 and have to return a meaningless error. With XML-RPC, those exceptions are actually serialized and sent over the wire. We may want to explore how we can make that information available to the user.
I would say as a buffer at least that's no problem. But exposing the structure of the error stack on the server may be a bit too much.
What Ewan and I had discussed at the Summit, was standardizing a set of exceptions for use as errors. I think this is a good idea. I would like to always keep the stack trace in the exceptions though as it makes debugging Xend considerably simpler. For something like libvirt, we could just look at the faultCode and use that to build an appropriate error message (based on current locale of course). Perhaps we could also offer an extended error message too that included the Xend faultMessage too.
My current understanding, btw, is that the basic XML-RPC support for Xend is going into the tree sometime this week (Ewan is looking at the code now) and will be a part of the 3.0.2 release. This means that for
That's waht I was told too :-)
the 3.0.3 cycle, we'll be defining a standard API. This process will be starting soon (within the next couple of weeks hopefully).
Excellent ! Good good ! Hopefully we will be able to confront that with what is present in other engines like l4 and QEmu and make sure at least for the 'common' APIs the semantic is similar.
Oh, L4, good idea :-) I know just the person to talk to about that too... Regards, Anthony Liguori
Daniel
-- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
On Fri, Mar 10, 2006 at 04:35:11PM -0600, Anthony Liguori wrote:
On Fri, 10 Mar 2006 17:21:52 -0500, Daniel Veillard wrote: One thing I should say is that I'm still contemplating changing the XML-RPC interface in Xend to make the libvirt a little easier..
Well don't think specifically in libvirt terms, but what canonical interfaces are the most likely to be suitable for apps :-)
Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions.
the new error code tries at least to extract the error message when an HTTP POST or GEt fails with an error code, but the XML-RPC should give a far more reliable framework for error handling.
If I recall correctly, the S-Expression/HTTP errors are always 501 Internal Server error with no extended error message. Of course, I may be wrong.
There is an error message in the content of the GET/POST returned, I extract it now, running "make tests" show: /u/veillard/libvirt/python/tests ## running Python regression tests libvir: Xen Daemon error : GET operation failed: No such domain test That is a message output by the library. The python shutdown the 'test' domain and loop trying to get state every second until a failure. The failure ends up being reported (i.e. success :-) "GET operation failed" comes from libvirt, while "No such domain test" is the error message extrated from the failed HTTP request, I think it work alike for POST ones. Still this is limited and XML-RPC should be better in this respect :-)
For the S-Expression/HTTP protocol, you just get a 501 and have to return a meaningless error. With XML-RPC, those exceptions are actually serialized and sent over the wire. We may want to explore how we can make that information available to the user.
I would say as a buffer at least that's no problem. But exposing the structure of the error stack on the server may be a bit too much.
What Ewan and I had discussed at the Summit, was standardizing a set of exceptions for use as errors. I think this is a good idea. I would like to always keep the stack trace in the exceptions though as it makes debugging Xend considerably simpler.
Sure sounds a good idea
For something like libvirt, we could just look at the faultCode and use that to build an appropriate error message (based on current locale of course). Perhaps we could also offer an extended error message too that included the Xend faultMessage too.
appropriate error message yes. locale hum I'm not fond of it. libvirt is too low in the stack, I would not expect the 'message' to be directly useful to an user, but one must give as much informations as possible to the software on top to be able to analyze and present a correct, in-context error if needed.
Excellent ! Good good ! Hopefully we will be able to confront that with what is present in other engines like l4 and QEmu and make sure at least for the 'common' APIs the semantic is similar.
Oh, L4, good idea :-) I know just the person to talk to about that too...
Well Ronald Aigner who joined the list 2 days ago from Desden Univ certainly seems to come from the l4 group there, but sure get more people on this list, so we have less risk to fail our API designs ;-) Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions.
the new error code tries at least to extract the error message when an HTTP POST or GEt fails with an error code, but the XML-RPC should give a far more reliable framework for error handling.
Nice segway... I've been recently pinged a few times by our Xen CIM consumers about the lack of good errors coming out of our providers (which in turn are limited by what we get back from libxm today), especially in regards to conditions that might cause a create() operation to fail. Do you have a sense today of what errors we might expect to get reported back from libvirt? Not that this will constitute any sort of meaningful 'requirements' with which to write code, but the following is a list of errors that my Xen CIM consumers handle today for the likes VMWare. I am trying to get more info on under what specific circumstance(s) these are generated... ERR_SUCCESS ERR_UNABLE_TO_VERIFY_STATE ERR_OUT_OF_DISK_SPACE ERR_BAD_PARAMETER ERR_VM_CONTROL_OP_FAILED ERR_INVALID_PARM_NUM ERR_CANNOT_ACCESS_DISK_FILE ERR_UNKNOWN ERR_VIRTUAL_DISK_CREATE_FAILED ERR_VM_STUCK ERR_CREDENTIALS_NOT_SET ERR_UNACCEPTED_CREDENTIALS ERR_OUT_OF_MEMORY ERR_WRONG_STATE_FOR_OP ERR_VM_NOT_FOUND ERR_HOST_NOT_FOUND ERR_ACCESS_DENIED ERR_ALREADY_EXIST ERR_OPERATION_FAILED ERR_UNDOABLE_DISK_NOT_SUPPORTED ERR_VMM_CMD_FORMAT_ERROR ERR_COMMUNICATION_NOT_ESTABLISHED ERR_FILE_COPY_FAILED ERR_NAME_TOO_LONG ERR_OS_NOT_SUPPORTED ERR_MOUNT_FAILED_DIR_NOT_EMPTY ERR_CANT_DISMOUNT_BOOT_OR_SYSTEM ERR_FILE_IN_USE I think error reporting is an area where we will definitely want to drive clients' requirements down into the likes of libvirt. Thnx. - Gareth Dr. Gareth S. Bestor IBM Linux Technology Center M/S DES2-01 15300 SW Koll Parkway, Beaverton, OR 97006 503-578-3186, T/L 775-3186, Fax 503-578-3186 |---------+------------------------------> | | Daniel Veillard | | | <veillard@redhat.co| | | m> | | | Sent by: | | | libvir-list-bounces| | | @redhat.com | | | | | | | | | 03/10/06 02:21 PM | | | Please respond to | | | veillard | |---------+------------------------------>
------------------------------------------------------------------------------------------------------------------| | | | To: aliguori@us.ltcfwd.linux.ibm.com | | cc: libvir-list@redhat.com | | Subject: Re: [Libvir] XML-RPC support for libvirt | ------------------------------------------------------------------------------------------------------------------|
On Fri, Mar 10, 2006 at 10:57:59AM -0600, Anthony Liguori wrote:
Howdy,
Evening ;-)
I'm about to checkin some XML-RPC code into CVS. The code I'm checking
okay got it. Please add a Changelog entry when you commit to the base :-)
in won't compile by default (for now) as there's still a little work to do but I wanted to get something into the tree. Here are the tasks left:
1) Convert from dict to s-expressions to handle config returns. Right now, the Xend XML-RPC code returns S-Expressions in the form of a tuple. We just need a little code to convert from these tuples to struct sexpr's.
2) Integrate my unit tests. I've got a number of unit tests that stress the various bits of parsing code. I'd like to add a make check that runs these unit tests and a make fullcheck that runs these unit tests under valgrind.
3) Support XML-RPC over a domain socket. Right now I'm using nanohttp which doesn't appear to allow different transports. There is some HTTP PUT code already in the tree so I could use that if there isn't a more clever solution.
4) Plumb the xend backend functions to XML-RPC. This is really straight forward. It's just a matter of converting:
int xend_pause(virConnectPtr xend, const char *name) { return xend_op(xend, name, "op", "pause", NULL); }
To:
int xend_pause(virConnectPtr xend, const char *name) { if (xend->xmlrpc) return xmlRpcCall(xend->xmlrpc, "xend.domain.pause", "", "s", name) else return xend_op(xend, name, "op", "pause", NULL); }
Okay the steps needed look fairly clear, thanks :-)
Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions.
the new error code tries at least to extract the error message when an HTTP POST or GEt fails with an error code, but the XML-RPC should give a far more reliable framework for error handling.
For the S-Expression/HTTP protocol, you just get a 501 and have to return a meaningless error. With XML-RPC, those exceptions are actually serialized and sent over the wire. We may want to explore how we can make that information available to the user.
I would say as a buffer at least that's no problem. But exposing the structure of the error stack on the server may be a bit too much.
My current understanding, btw, is that the basic XML-RPC support for Xend is going into the tree sometime this week (Ewan is looking at the code now) and will be a part of the 3.0.2 release. This means that for
That's waht I was told too :-)
the 3.0.3 cycle, we'll be defining a standard API. This process will be starting soon (within the next couple of weeks hopefully).
Excellent ! Good good ! Hopefully we will be able to confront that with what is present in other engines like l4 and QEmu and make sure at least for the 'common' APIs the semantic is similar. Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/ -- Libvir-list mailing list Libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
On Fri, Mar 10, 2006 at 02:57:57PM -0800, Gareth S Bestor wrote:
Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions.
the new error code tries at least to extract the error message when an HTTP POST or GEt fails with an error code, but the XML-RPC should give a far more reliable framework for error handling.
Nice segway... I've been recently pinged a few times by our Xen CIM consumers about the lack of good errors coming out of our providers (which in turn are limited by what we get back from libxm today), especially in regards to conditions that might cause a create() operation to fail. Do you have a sense today of what errors we might expect to get reported back from libvirt?
I think the prerequisite read is the following page where I tried to write down how I planned and implemented error handling: http://libvirt.org/errors.html it's not coming from nowhere, it's actually the model used by libxml2 "structured" error handling the latest evolution of error processing in that library. It's not completely broken as people seems to be satisfied now with it (and its set of users is quite diverse :-)
Not that this will constitute any sort of meaningful 'requirements' with which to write code, but the following is a list of errors that my Xen CIM consumers handle today for the likes VMWare. I am trying to get more info on under what specific circumstance(s) these are generated...
ERR_SUCCESS ERR_UNABLE_TO_VERIFY_STATE ERR_OUT_OF_DISK_SPACE ERR_BAD_PARAMETER ERR_VM_CONTROL_OP_FAILED ERR_INVALID_PARM_NUM ERR_CANNOT_ACCESS_DISK_FILE ERR_UNKNOWN ERR_VIRTUAL_DISK_CREATE_FAILED ERR_VM_STUCK ERR_CREDENTIALS_NOT_SET ERR_UNACCEPTED_CREDENTIALS ERR_OUT_OF_MEMORY ERR_WRONG_STATE_FOR_OP ERR_VM_NOT_FOUND ERR_HOST_NOT_FOUND ERR_ACCESS_DENIED ERR_ALREADY_EXIST ERR_OPERATION_FAILED ERR_UNDOABLE_DISK_NOT_SUPPORTED ERR_VMM_CMD_FORMAT_ERROR ERR_COMMUNICATION_NOT_ESTABLISHED ERR_FILE_COPY_FAILED ERR_NAME_TOO_LONG ERR_OS_NOT_SUPPORTED ERR_MOUNT_FAILED_DIR_NOT_EMPTY ERR_CANT_DISMOUNT_BOOT_OR_SYSTEM ERR_FILE_IN_USE
That look actually a bit short to me for such a complete tool ;-)
I think error reporting is an area where we will definitely want to drive clients' requirements down into the likes of libvirt. Thnx.
Well you can consult libvirt current list of errors in http://libvirt.org/html/libvirt-virterror.html#virErrorNumber if we can get more details from the Xend internals then the VIR_ERR_GET_FAILED and VIR_ERR_POST_FAILED could be replaced by more precise informations. Also keep in mind that I would like as much as possible to keep genericity in the API among the different back-ends (and if know of a way to invite the WMWare folks to help here I would be grateful :-). Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
T Thanks Daniel. I agree on both counts: the list I attached (as given to me) is still pretty anemic, and our Xen CIM consumer today would also prefer generic, broadly useful common error codes and not just Xen (or VMWare) specific ones, although some of this normalization will probably be handled at the CIM level. I'll check out your suggested links. thnx again.
(and if know of a way to invite the WMWare folks to help here I would be grateful :-).
I will plant a seed and see if it grows... :-) - G Gareth S. Bestor, PhD. IBM Linux Technology Center M/S DES2-01 15300 SW Koll Parkway, Beaverton, OR 97006 503-578-3186, T/L 775-3186, Fax 503-578-3186 Daniel Veillard <veillard@redhat.com> on 03/10/2006 03:18:23 PM Please respond to veillard@redhat.com To: Gareth S Bestor/Poughkeepsie/IBM@IBMUS cc: libvir-list@redhat.com Subject: Re: [Libvir] XML-RPC support for libvirt On Fri, Mar 10, 2006 at 02:57:57PM -0800, Gareth S Bestor wrote:
Of course, using the xml-rpc code, we now have access to rich fault information. Xend never actually returns errors for things and instead throws exceptions.
the new error code tries at least to extract the error message when an HTTP POST or GEt fails with an error code, but the XML-RPC should give a far more reliable framework for error handling.
Nice segway... I've been recently pinged a few times by our Xen CIM consumers about the lack of good errors coming out of our providers (which in turn are limited by what we get back from libxm today), especially in regards to conditions that might cause a create() operation to fail. Do you have a sense today of what errors we might expect to get reported back from libvirt?
I think the prerequisite read is the following page where I tried to write down how I planned and implemented error handling: http://libvirt.org/errors.html it's not coming from nowhere, it's actually the model used by libxml2 "structured" error handling the latest evolution of error processing in that library. It's not completely broken as people seems to be satisfied now with it (and its set of users is quite diverse :-)
Not that this will constitute any sort of meaningful 'requirements' with which to write code, but the following is a list of errors that my Xen CIM consumers handle today for the likes VMWare. I am trying to get more info on under what specific circumstance(s) these are generated...
ERR_SUCCESS ERR_UNABLE_TO_VERIFY_STATE ERR_OUT_OF_DISK_SPACE ERR_BAD_PARAMETER ERR_VM_CONTROL_OP_FAILED ERR_INVALID_PARM_NUM ERR_CANNOT_ACCESS_DISK_FILE ERR_UNKNOWN ERR_VIRTUAL_DISK_CREATE_FAILED ERR_VM_STUCK ERR_CREDENTIALS_NOT_SET ERR_UNACCEPTED_CREDENTIALS ERR_OUT_OF_MEMORY ERR_WRONG_STATE_FOR_OP ERR_VM_NOT_FOUND ERR_HOST_NOT_FOUND ERR_ACCESS_DENIED ERR_ALREADY_EXIST ERR_OPERATION_FAILED ERR_UNDOABLE_DISK_NOT_SUPPORTED ERR_VMM_CMD_FORMAT_ERROR ERR_COMMUNICATION_NOT_ESTABLISHED ERR_FILE_COPY_FAILED ERR_NAME_TOO_LONG ERR_OS_NOT_SUPPORTED ERR_MOUNT_FAILED_DIR_NOT_EMPTY ERR_CANT_DISMOUNT_BOOT_OR_SYSTEM ERR_FILE_IN_USE
That look actually a bit short to me for such a complete tool ;-)
I think error reporting is an area where we will definitely want to drive clients' requirements down into the likes of libvirt. Thnx.
Well you can consult libvirt current list of errors in http://libvirt.org/html/libvirt-virterror.html#virErrorNumber if we can get more details from the Xend internals then the VIR_ERR_GET_FAILED and VIR_ERR_POST_FAILED could be replaced by more precise informations. Also keep in mind that I would like as much as possible to keep genericity in the API among the different back-ends (and if know of a way to invite the WMWare folks to help here I would be grateful :-). Daniel -- Daniel Veillard | Red Hat http://redhat.com/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
participants (4)
-
Anthony Liguori -
Anthony Liguori -
Daniel Veillard -
Gareth S Bestor