Re: [libvirt] [PATCH] Fix threading problems in python bindings

On Wed, Nov 25, 2009 at 12:06:03PM +0000, Daniel P. Berrange wrote: Thanks, but I am still experiencing deadlocks when I run more than one domain. Thread 3 (Thread 0x52022940 (LWP 29704)): #0 0x0000003cc1c0c9b1 in sem_wait () from /lib64/libpthread.so.0 #1 0x0000003cc24b8148 in PyThread_acquire_lock (lock=<value optimized out>, waitflag=<value optimized out>) at Python/thread_pthread.h:313 #2 0x0000003cc2496084 in PyEval_RestoreThread (tstate=<value optimized out>) at Python/ceval.c:305 #3 0x0000003cc24b1eac in PyGILState_Ensure () at Python/pystate.c:508 #4 0x00002ae384d41605 in libvirt_virEventUpdateHandleFunc (watch=2, event=0) at libvirt-override.c:2279 #5 0x00002ae384fbd016 in remoteIO (conn=0xf57b560, priv=0x2aaaaad7a010, flags=0, thiscall=0x2aaaac0008e0) at remote/remote_driver.c:8437 #6 0x00002ae384fbdbab in call (conn=0xf57b560, priv=0x2aaaaad7a010, flags=0, proc_nr=<value optimized out>, args_filter=0x2ae384fd0af0 <xdr_remote_domain_interface_stats_args>, args=0x52020120 "\220ZX\017", ret_filter=0x2ae384fcdfe0 <xdr_remote_domain_interface_stats_ret>, ret=0x520200e0 "") at remote/remote_driver.c:8508 #7 0x00002ae384fc084b in remoteDomainInterfaceStats (domain=0xf57fdc0, path=0xf630e34 "vnet1", stats=0x520201a0) at remote/remote_driver.c:3345 #8 0x00002ae384fafd82 in virDomainInterfaceStats (dom=0xf57fdc0, path=0xf630e34 "vnet1", stats=0x52020210, size=64) at libvirt.c:4083 #9 0x00002ae384d433e7 in libvirt_virDomainInterfaceStats ( self=<value optimized out>, args=<value optimized out>) at libvirt-override.c:103 #10 0x0000003cc24950ba in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3563 #11 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #12 0x0000003cc244c45a in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #13 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #14 0x0000003cc249352c in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3840 #15 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #16 0x0000003cc249473f in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3656 #17 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #18 0x0000003cc244c367 in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #19 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #20 0x0000003cc243c1ef in instancemethod_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:2447 #21 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #22 0x0000003cc248fc3d in PyEval_CallObjectWithKeywords ( func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Python/ceval.c:3430 #23 0x0000003cc243fc10 in PyInstance_New (klass=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:575 #24 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #25 0x0000003cc24928ce in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3771 #26 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #27 0x0000003cc244c367 in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #28 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #29 0x0000003cc243c1ef in instancemethod_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:2447 #30 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #31 0x0000003cc248fc3d in PyEval_CallObjectWithKeywords ( func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Python/ceval.c:3430 #32 0x0000003cc243fc10 in PyInstance_New (klass=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:575 #33 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #34 0x0000003cc24928ce in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3771 #35 0x0000003cc2494b66 in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3645 #36 0x0000003cc2494b66 in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3645 #37 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #38 0x0000003cc244c367 in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #39 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #40 0x0000003cc243c1ef in instancemethod_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:2447 #41 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #42 0x0000003cc248fc3d in PyEval_CallObjectWithKeywords ( func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Python/ceval.c:3430 #43 0x0000003cc24bbbbd in t_bootstrap (boot_raw=<value optimized out>) at Modules/threadmodule.c:434 #44 0x0000003cc1c064a7 in start_thread () from /lib64/libpthread.so.0 #45 0x0000003cc14d3c2d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x50c20940 (LWP 29764)): #0 0x0000003cc1c0ab99 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae384f7dbb6 in virCondWait (c=0xf7e4cdc, m=0x80) at util/threads-pthread.c:84 #2 0x00002ae384fbcf62 in remoteIO (conn=0xf57b560, priv=0x2aaaaad7a010, flags=0, thiscall=0xf7a4cc0) at remote/remote_driver.c:8378 #3 0x00002ae384fbdbab in call (conn=0xf57b560, priv=0x2aaaaad7a010, flags=0, proc_nr=<value optimized out>, args_filter=0x2ae384fd07b0 <xdr_remote_domain_get_info_args>, args=0x50c1e100 "\020{^\017", ret_filter=0x2ae384fcdf60 <xdr_remote_domain_get_info_ret>, ret=0x50c1e0d0 "") at remote/remote_driver.c:8508 #4 0x00002ae384fc1fe0 in remoteDomainGetInfo (domain=0xf5e91f0, info=0x50c1e1c0) at remote/remote_driver.c:2261 #5 0x00002ae384fb023b in virDomainGetInfo (domain=0xf5e91f0, info=0x50c1e1c0) at libvirt.c:2786 #6 0x00002ae384d42748 in libvirt_virDomainGetVcpus ( self=<value optimized out>, args=<value optimized out>) at libvirt-override.c:354 #7 0x0000003cc24950ba in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3563 #8 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #9 0x0000003cc244c45a in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #10 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #11 0x0000003cc249352c in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3840 #12 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #13 0x0000003cc249473f in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3656 #14 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #15 0x0000003cc244c367 in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #16 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #17 0x0000003cc243c1ef in instancemethod_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:2447 #18 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #19 0x0000003cc248fc3d in PyEval_CallObjectWithKeywords ( func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Python/ceval.c:3430 #20 0x0000003cc243fc10 in PyInstance_New (klass=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:575 #21 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #22 0x0000003cc24928ce in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3771 #23 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #24 0x0000003cc244c367 in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #25 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #26 0x0000003cc243c1ef in instancemethod_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:2447 #27 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #28 0x0000003cc248fc3d in PyEval_CallObjectWithKeywords ( func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Python/ceval.c:3430 #29 0x0000003cc243fc10 in PyInstance_New (klass=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:575 #30 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #31 0x0000003cc24928ce in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3771 #32 0x0000003cc2494b66 in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3645 #33 0x0000003cc2494b66 in PyEval_EvalFrame (f=<value optimized out>) at Python/ceval.c:3645 #34 0x0000003cc2495fe5 in PyEval_EvalCodeEx (co=<value optimized out>, globals=<value optimized out>, locals=<value optimized out>, args=<value optimized out>, argcount=<value optimized out>, kws=<value optimized out>, kwcount=<value optimized out>, defs=<value optimized out>, defcount=<value optimized out>, closure=<value optimized out>) at Python/ceval.c:2736 #35 0x0000003cc244c367 in function_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/funcobject.c:548 #36 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #37 0x0000003cc243c1ef in instancemethod_call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/classobject.c:2447 #38 0x0000003cc24360f0 in PyObject_Call (func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Objects/abstract.c:1795 #39 0x0000003cc248fc3d in PyEval_CallObjectWithKeywords ( func=<value optimized out>, arg=<value optimized out>, kw=<value optimized out>) at Python/ceval.c:3430 #40 0x0000003cc24bbbbd in t_bootstrap (boot_raw=<value optimized out>) at Modules/threadmodule.c:434 #41 0x0000003cc1c064a7 in start_thread () from /lib64/libpthread.so.0 #42 0x0000003cc14d3c2d in clone () from /lib64/libc.so.6