In the virt-manager app I'm using the python bindings to libvirt for all operations. For long running operations such as creating new domains, or saving/restoring domains I fork a background thread to do the call, so that the main UI / status refresh can carry on working. Except this wasn't working. The main thread was being blocked for the entire duration of the background thread. After a lot of nasty debug sessions I discovered that the main thread was being blocked in the GTK event loop attempting to aquire the Python GIL (Global Interpreter Lock). The only way this could be occurring is if the background thread doing the libvirt call held the GIL. And indeed after looking at the source for the libvirt python bindings it appears we never release the GIL when calling into the libvirt C apis, so for the entire duration of virDomainCreateLinux the python interpreter is completely stopped. Clearly this sucks because virDomainCreateLinux can take a very long time.

I read up a little on python threading & looked at how PyGTK deals with this & have come up with a patch to make libvirt release the GIL when entering a C call & re-aquire it on completion. Indeed, since PyGTK & GTK are both LGPL, I just copied their C macros straight over. The patch appears to work - the python interpreter is no longer deadlocked when calling virDomainCreateLinux (or any other libvirt calls). Unfortuntely the virtmanager app still deadlocks, because it turns out XenD itself serializes all SEXPR HTTP requests, but once this is fixed the whole thing is fully parallelized & works without blocking the app.

Attached the patch - I've not done much python threading before so let me know if i missed anything / messed stuff up. NB, I also hacked up

the error handling callback so it re-acquires the lock before calling back into the python.

> output.write("libvirt_begin_allow_threads;\n"); 425a427 > output.write("libvirt_end_allow_threads;\n");

diff -r1.3 libvirt_wrap.h 50a51,102 > extern int libvirt_threads_enabled; > > > /* Provide simple macro statement wrappers (adapted from Perl): > * LIBVIRT_STMT_START { statements; } LIBVIRT_STMT_END; > * can be used as a single statement, as in > * if (x) LIBVIRT_STMT_START { ... } LIBVIRT_STMT_END; else ... > * > * When GCC is compiling C code in non-ANSI mode, it will use the > * compiler __extension__ to wrap the statements wihin `({' and '})' braces. > * When compiling on platforms where configure has defined > * HAVE_DOWHILE_MACROS, statements will be wrapped with `do' and `while (0)'. > * For any other platforms (SunOS4 is known to have this issue), wrap the > * statements with `if (1)' and `else (void) 0'. > */ > #if !(defined (LIBVIRT_STMT_START) && defined (LIBVIRT_STMT_END)) > # if defined (__GNUC__) && !defined (__STRICT_ANSI__) && !defined (__cplusplus) > # define LIBVIRT_STMT_START (void) __extension__ ( > # define LIBVIRT_STMT_END ) > # else /* !(__GNUC__ && !__STRICT_ANSI__ && !__cplusplus) */ > # if defined (HAVE_DOWHILE_MACROS) > # define LIBVIRT_STMT_START do > # define LIBVIRT_STMT_END while (0) > # else /* !HAVE_DOWHILE_MACROS */ > # define LIBVIRT_STMT_START if (1) > # define LIBVIRT_STMT_END else (void) 0 > # endif /* !HAVE_DOWHILE_MACROS */ > # endif /* !(__GNUC__ && !__STRICT_ANSI__ && !__cplusplus) */ > #endif > > #define libvirt_begin_allow_threads \ > LIBVIRT_STMT_START { \ > PyThreadState *_save = NULL; \ > if (PyEval_ThreadsInitialized()) \ > _save = PyEval_SaveThread(); > > #define libvirt_end_allow_threads \ > if (PyEval_ThreadsInitialized()) \ > PyEval_RestoreThread(_save); \ > } LIBVIRT_STMT_END > > #define libvirt_ensure_thread_state \ > LIBVIRT_STMT_START { \ > PyGILState_STATE _save; \ > if (PyEval_ThreadsInitialized()) \ > _save = PyGILState_Ensure(); > > #define libvirt_release_thread_state \ > if (PyEval_ThreadsInitialized()) \ > PyGILState_Release(_save); \ > } LIBVIRT_STMT_END