On Wed, Jul 02, 2014 at 11:56:23AM -0400, Francesco Romani wrote:
This is made only worse by the fact that VDSM is a python 2.7 application, and notoriously python 2.x behaves very badly with threads. We are already working to improve our code, but I'd like to bring the discussion here and see if and when the querying API can be improved.
We currently use these APIs for our sempling: virDomainBlockInfo virDomainGetInfo virDomainGetCPUStats virDomainBlockStats virDomainBlockStatsFlags virDomainInterfaceStats virDomainGetVcpusFlags virDomainGetMetadata
What we'd like to have is
* asynchronous APIs for querying domain stats (https://bugzilla.redhat.com/show_bug.cgi?id=1113106) This would be just awesome. Either a single callback or a different one per call is fine (let's discuss this!). please note that we are much more concerned about thread reduction then about performance numbers. We had report of thread number becoming a real harm, while performance so far is not yet a concern (https://bugzilla.redhat.com/show_bug.cgi?id=1102147#c54)
I'm not a big fan of this approach. I mean, IIRC python has this Big Python Lock, which effectively prevents two threads run concurrently.
It has the GIL, yes. Only one thread can run python code at any given time. This however it is not true for extensions modules written in C which if carefully designed (read: coded to properly release the GIL) can run concurrently. This is one of the reasons while threading in python it is tolerated for I/O, evne though never recommended.
AFAIK/IIRC the code the libvirt module for python allows this, so we should be good to go.
For the sake of completeness I'll point out that there's another theoretical option. The libvirt-gobject binding to libvirt provides async APIs to libvirt APIs. It does this by using threads internally. Since these are C level threads though, if VDSM were to use libvirt-gobject it could get async APIs and the benefits of real threads, while remaining single threaded at the python layer. That all said, I'm not sure whether libvirt-gobject has sufficient API coverage for all the APIs VDSM needs. It primarily just has bindings for the APIs used by GNOME Boxes & libvirt-sandbox so far. Also not sure if it is a widely deployed enough dep for VDSM to mandate. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|