[libvirt] segment fault from libvirtmod
Hi, The Makefile.am in python forget to add probes.o if WITH_DTRACE but after I added it and tried to connect, using libvirt.open("qemu:///system") in python , it reported: "segment fault" I tried to figure out, but failed. If anyone can help this, thanks. The following is the backtrace: #0 0x0000003c62a810a4 in free () from /lib64/libc.so.6 #1 0x00007ffff1836f29 in virFree (ptrptr=0x6a6a28) at util/memory.c:310 #2 0x00007ffff1849692 in virResetError (err=0x6a6a20) at util/virterror.c:387 #3 0x00007ffff13e2761 in do_open (name=0x7ffff7ed6444 "qemu:///system", auth=0x0, flags=0) at libvirt.c:1093 #4 0x00007ffff13e4d44 in virConnectOpen (name=0x7ffff7ed6444 "qemu:///system") at libvirt.c:1350 #5 0x00007ffff1824879 in libvirt_virConnectOpen (self=<optimized out>, args=<optimized out>) at libvirt.c:3637 #6 0x0000003c652dffbb in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #7 0x0000003c652e0580 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #8 0x0000003c652e15a5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0 #9 0x0000003c652e16d2 in PyEval_EvalCode () from /usr/lib64/libpython2.7.so.1.0 #10 0x0000003c652fb9ec in ?? () from /usr/lib64/libpython2.7.so.1.0 #11 0x0000003c652fc7f0 in PyRun_FileExFlags () from /usr/lib64/libpython2.7.so.1.0 #12 0x0000003c652fd26f in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.7.so.1.0 #13 0x0000003c6530e745 in Py_Main () from /usr/lib64/libpython2.7.so.1.0 #14 0x0000003c62a2169d in __libc_start_main () from /lib64/libc.so.6 #15 0x0000000000400651 in _start () Guannan Ren
On 06.02.2012 10:08, Guannan Ren wrote:
Hi,
The Makefile.am in python forget to add probes.o if WITH_DTRACE but after I added it and tried to connect, using libvirt.open("qemu:///system") in python , it reported: "segment fault" I tried to figure out, but failed. If anyone can help this, thanks.
The following is the backtrace:
#0 0x0000003c62a810a4 in free () from /lib64/libc.so.6 #1 0x00007ffff1836f29 in virFree (ptrptr=0x6a6a28) at util/memory.c:310 #2 0x00007ffff1849692 in virResetError (err=0x6a6a20) at util/virterror.c:387 #3 0x00007ffff13e2761 in do_open (name=0x7ffff7ed6444 "qemu:///system", auth=0x0, flags=0) at libvirt.c:1093 #4 0x00007ffff13e4d44 in virConnectOpen (name=0x7ffff7ed6444 "qemu:///system") at libvirt.c:1350 #5 0x00007ffff1824879 in libvirt_virConnectOpen (self=<optimized out>, args=<optimized out>) at libvirt.c:3637 #6 0x0000003c652dffbb in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #7 0x0000003c652e0580 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #8 0x0000003c652e15a5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0 #9 0x0000003c652e16d2 in PyEval_EvalCode () from /usr/lib64/libpython2.7.so.1.0 #10 0x0000003c652fb9ec in ?? () from /usr/lib64/libpython2.7.so.1.0 #11 0x0000003c652fc7f0 in PyRun_FileExFlags () from /usr/lib64/libpython2.7.so.1.0 #12 0x0000003c652fd26f in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.7.so.1.0 #13 0x0000003c6530e745 in Py_Main () from /usr/lib64/libpython2.7.so.1.0 #14 0x0000003c62a2169d in __libc_start_main () from /lib64/libc.so.6 #15 0x0000000000400651 in _start ()
Guannan Ren
Running git bisect showed it was caused by c700613b8d463212d142c97108b7a2352e23e559. However, I think it only exposed the design problem we are facing here. IMHO problem is when an application tries to call virConnectOpen() (or virConnectOpenAuth()) from two concurrent threads. This is what will happen: 1st thread: call virConnectOpen(); global variable @initialized == zero; thus virInitialize() is called, error object is set up (via virErrorInitialize()) which is thread local variable. However, @initialized is set to 1 in the first place. 2nd thread: call virConnectOpen() as well; @initialized is already set, therefore no thread local variable is set and any error report will fail. Even if I am right, I am not sure how to fix this. Michal
On 02/06/2012 07:59 AM, Michal Privoznik wrote:
Running git bisect showed it was caused by c700613b8d463212d142c97108b7a2352e23e559. However, I think it only exposed the design problem we are facing here.
IMHO problem is when an application tries to call virConnectOpen() (or virConnectOpenAuth()) from two concurrent threads. This is what will happen:
1st thread: call virConnectOpen(); global variable @initialized == zero; thus virInitialize() is called, error object is set up (via virErrorInitialize()) which is thread local variable. However, @initialized is set to 1 in the first place.
2nd thread: call virConnectOpen() as well; @initialized is already set, therefore no thread local variable is set and any error report will fail.
Ouch - I agree with your analysis of a thread-safety issue. I think we can fix it by making virInitialize() delegate most of its work to a virOnce() initialization function. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On Mon, Feb 06, 2012 at 03:59:19PM +0100, Michal Privoznik wrote:
On 06.02.2012 10:08, Guannan Ren wrote:
Hi,
The Makefile.am in python forget to add probes.o if WITH_DTRACE but after I added it and tried to connect, using libvirt.open("qemu:///system") in python , it reported: "segment fault" I tried to figure out, but failed. If anyone can help this, thanks.
The following is the backtrace:
#0 0x0000003c62a810a4 in free () from /lib64/libc.so.6 #1 0x00007ffff1836f29 in virFree (ptrptr=0x6a6a28) at util/memory.c:310 #2 0x00007ffff1849692 in virResetError (err=0x6a6a20) at util/virterror.c:387 #3 0x00007ffff13e2761 in do_open (name=0x7ffff7ed6444 "qemu:///system", auth=0x0, flags=0) at libvirt.c:1093 #4 0x00007ffff13e4d44 in virConnectOpen (name=0x7ffff7ed6444 "qemu:///system") at libvirt.c:1350 #5 0x00007ffff1824879 in libvirt_virConnectOpen (self=<optimized out>, args=<optimized out>) at libvirt.c:3637 #6 0x0000003c652dffbb in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #7 0x0000003c652e0580 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #8 0x0000003c652e15a5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0 #9 0x0000003c652e16d2 in PyEval_EvalCode () from /usr/lib64/libpython2.7.so.1.0 #10 0x0000003c652fb9ec in ?? () from /usr/lib64/libpython2.7.so.1.0 #11 0x0000003c652fc7f0 in PyRun_FileExFlags () from /usr/lib64/libpython2.7.so.1.0 #12 0x0000003c652fd26f in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.7.so.1.0 #13 0x0000003c6530e745 in Py_Main () from /usr/lib64/libpython2.7.so.1.0 #14 0x0000003c62a2169d in __libc_start_main () from /lib64/libc.so.6 #15 0x0000000000400651 in _start ()
Guannan Ren
Running git bisect showed it was caused by c700613b8d463212d142c97108b7a2352e23e559. However, I think it only exposed the design problem we are facing here.
IMHO problem is when an application tries to call virConnectOpen() (or virConnectOpenAuth()) from two concurrent threads. This is what will happen:
1st thread: call virConnectOpen(); global variable @initialized == zero; thus virInitialize() is called, error object is set up (via virErrorInitialize()) which is thread local variable. However, @initialized is set to 1 in the first place.
2nd thread: call virConnectOpen() as well; @initialized is already set, therefore no thread local variable is set and any error report will fail.
Even if I am right, I am not sure how to fix this.
In a mulithreaded application, you should be calling virInitialize() explicitly, instead of relying on lazy init. IMHO, the python module should simply be calling virInitialize at time of import. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Feb 06, 2012 at 03:59:19PM +0100, Michal Privoznik wrote:
On 06.02.2012 10:08, Guannan Ren wrote:
Hi,
The Makefile.am in python forget to add probes.o if WITH_DTRACE but after I added it and tried to connect, using libvirt.open("qemu:///system") in python , it reported: "segment fault" I tried to figure out, but failed. If anyone can help this, thanks.
The following is the backtrace:
#0 0x0000003c62a810a4 in free () from /lib64/libc.so.6 #1 0x00007ffff1836f29 in virFree (ptrptr=0x6a6a28) at util/memory.c:310 #2 0x00007ffff1849692 in virResetError (err=0x6a6a20) at util/virterror.c:387 #3 0x00007ffff13e2761 in do_open (name=0x7ffff7ed6444 "qemu:///system", auth=0x0, flags=0) at libvirt.c:1093 #4 0x00007ffff13e4d44 in virConnectOpen (name=0x7ffff7ed6444 "qemu:///system") at libvirt.c:1350 #5 0x00007ffff1824879 in libvirt_virConnectOpen (self=<optimized out>, args=<optimized out>) at libvirt.c:3637 #6 0x0000003c652dffbb in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #7 0x0000003c652e0580 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #8 0x0000003c652e15a5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0 #9 0x0000003c652e16d2 in PyEval_EvalCode () from /usr/lib64/libpython2.7.so.1.0 #10 0x0000003c652fb9ec in ?? () from /usr/lib64/libpython2.7.so.1.0 #11 0x0000003c652fc7f0 in PyRun_FileExFlags () from /usr/lib64/libpython2.7.so.1.0 #12 0x0000003c652fd26f in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.7.so.1.0 #13 0x0000003c6530e745 in Py_Main () from /usr/lib64/libpython2.7.so.1.0 #14 0x0000003c62a2169d in __libc_start_main () from /lib64/libc.so.6 #15 0x0000000000400651 in _start ()
Guannan Ren
Running git bisect showed it was caused by c700613b8d463212d142c97108b7a2352e23e559. However, I think it only exposed the design problem we are facing here.
IMHO problem is when an application tries to call virConnectOpen() (or virConnectOpenAuth()) from two concurrent threads. This is what will happen:
1st thread: call virConnectOpen(); global variable @initialized == zero; thus virInitialize() is called, error object is set up (via virErrorInitialize()) which is thread local variable. However, @initialized is set to 1 in the first place.
2nd thread: call virConnectOpen() as well; @initialized is already set, therefore no thread local variable is set and any error report will fail.
Even if I am right, I am not sure how to fix this. In a mulithreaded application, you should be calling virInitialize() explicitly, instead of relying on lazy init. IMHO, the python module should simply be calling virInitialize at time of import.
Regards, Daniel It called virInitialize() in the first place when libvirtmod is imported. Right now, I found the pthread_key_t is changed between the time it is initialized and time to get thread local data using the key. It
On 02/07/2012 05:35 PM, Daniel P. Berrange wrote: probably point to other location even in single thread.
On 06.02.2012 15:59, Michal Privoznik wrote:
On 06.02.2012 10:08, Guannan Ren wrote:
Hi,
The Makefile.am in python forget to add probes.o if WITH_DTRACE but after I added it and tried to connect, using libvirt.open("qemu:///system") in python , it reported: "segment fault" I tried to figure out, but failed. If anyone can help this, thanks.
The following is the backtrace:
#0 0x0000003c62a810a4 in free () from /lib64/libc.so.6 #1 0x00007ffff1836f29 in virFree (ptrptr=0x6a6a28) at util/memory.c:310 #2 0x00007ffff1849692 in virResetError (err=0x6a6a20) at util/virterror.c:387 #3 0x00007ffff13e2761 in do_open (name=0x7ffff7ed6444 "qemu:///system", auth=0x0, flags=0) at libvirt.c:1093 #4 0x00007ffff13e4d44 in virConnectOpen (name=0x7ffff7ed6444 "qemu:///system") at libvirt.c:1350 #5 0x00007ffff1824879 in libvirt_virConnectOpen (self=<optimized out>, args=<optimized out>) at libvirt.c:3637 #6 0x0000003c652dffbb in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #7 0x0000003c652e0580 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #8 0x0000003c652e15a5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0 #9 0x0000003c652e16d2 in PyEval_EvalCode () from /usr/lib64/libpython2.7.so.1.0 #10 0x0000003c652fb9ec in ?? () from /usr/lib64/libpython2.7.so.1.0 #11 0x0000003c652fc7f0 in PyRun_FileExFlags () from /usr/lib64/libpython2.7.so.1.0 #12 0x0000003c652fd26f in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.7.so.1.0 #13 0x0000003c6530e745 in Py_Main () from /usr/lib64/libpython2.7.so.1.0 #14 0x0000003c62a2169d in __libc_start_main () from /lib64/libc.so.6 #15 0x0000000000400651 in _start ()
Guannan Ren
Running git bisect showed it was caused by c700613b8d463212d142c97108b7a2352e23e559. However, I think it only exposed the design problem we are facing here.
IMHO problem is when an application tries to call virConnectOpen() (or virConnectOpenAuth()) from two concurrent threads. This is what will happen:
1st thread: call virConnectOpen(); global variable @initialized == zero; thus virInitialize() is called, error object is set up (via virErrorInitialize()) which is thread local variable. However, @initialized is set to 1 in the first place.
2nd thread: call virConnectOpen() as well; @initialized is already set, therefore no thread local variable is set and any error report will fail.
Even if I am right, I am not sure how to fix this.
Michal
Sorry for mystification. Here's the debug output which shows what's wrong: http://fpaste.org/6pBs/ However, I am not sure what is causing it.
On Mon, Feb 06, 2012 at 03:59:19PM +0100, Michal Privoznik wrote:
On 06.02.2012 10:08, Guannan Ren wrote:
Hi,
The Makefile.am in python forget to add probes.o if WITH_DTRACE but after I added it and tried to connect, using libvirt.open("qemu:///system") in python , it reported: "segment fault" I tried to figure out, but failed. If anyone can help this, thanks.
The following is the backtrace:
#0 0x0000003c62a810a4 in free () from /lib64/libc.so.6 #1 0x00007ffff1836f29 in virFree (ptrptr=0x6a6a28) at util/memory.c:310 #2 0x00007ffff1849692 in virResetError (err=0x6a6a20) at util/virterror.c:387 #3 0x00007ffff13e2761 in do_open (name=0x7ffff7ed6444 "qemu:///system", auth=0x0, flags=0) at libvirt.c:1093 #4 0x00007ffff13e4d44 in virConnectOpen (name=0x7ffff7ed6444 "qemu:///system") at libvirt.c:1350 #5 0x00007ffff1824879 in libvirt_virConnectOpen (self=<optimized out>, args=<optimized out>) at libvirt.c:3637 #6 0x0000003c652dffbb in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #7 0x0000003c652e0580 in PyEval_EvalFrameEx () from /usr/lib64/libpython2.7.so.1.0 #8 0x0000003c652e15a5 in PyEval_EvalCodeEx () from /usr/lib64/libpython2.7.so.1.0 #9 0x0000003c652e16d2 in PyEval_EvalCode () from /usr/lib64/libpython2.7.so.1.0 #10 0x0000003c652fb9ec in ?? () from /usr/lib64/libpython2.7.so.1.0 #11 0x0000003c652fc7f0 in PyRun_FileExFlags () from /usr/lib64/libpython2.7.so.1.0 #12 0x0000003c652fd26f in PyRun_SimpleFileExFlags () from /usr/lib64/libpython2.7.so.1.0 #13 0x0000003c6530e745 in Py_Main () from /usr/lib64/libpython2.7.so.1.0 #14 0x0000003c62a2169d in __libc_start_main () from /lib64/libc.so.6 #15 0x0000000000400651 in _start ()
Guannan Ren
Running git bisect showed it was caused by c700613b8d463212d142c97108b7a2352e23e559. However, I think it only exposed the design problem we are facing here.
Oh no, this commit is clearly wrong. @@ -35,8 +39,14 @@ EXTRA_DIST = \ $(DOCS) if WITH_PYTHON -mylibs = $(top_builddir)/src/libvirt.la -myqemulibs = $(top_builddir)/src/libvirt-qemu.la +mylibs = \ + $(top_builddir)/src/libvirt.la \ + $(top_builddir)/src/libvirt_util.la \ + $(top_builddir)/gnulib/lib/libgnu.la +myqemulibs = \ + $(top_builddir)/src/libvirt-qemu.la \ + $(top_builddir)/src/libvirt_util.la \ + $(top_builddir)/gnulib/lib/libgnu.la The commit as made the python library link against 'libvirt_util.la' directly. The libvirt_util.la library is *already* part of libvirt.so so this change resulted in 2 copies of libvirt-util.la being linked in which causes unpredictable results with global variables / initializers This part of the change should be reverted. Anything that the python code needs from libvirt_util.la, should be added to src/libvirt_private.syms if it is not already there. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (4)
-
Daniel P. Berrange -
Eric Blake -
Guannan Ren -
Michal Privoznik