[libvirt] Segfault in libxl_osevent_occurred_timeout [Was: Re: [libvirt-users] About debugging of libvirt.]
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel] Jim, cooldharma06 reports having issues when destroying VMs with libvirt 1.2.0 and Xen 4.2.1 (is that the case, cooldharma06)? Look at the stack trace at the end of this message, or here: https://bugzilla.redhat.com/show_bug.cgi?id=1044838 Ian, since it looks like events are involved... Any idea? cooldharma06, about the Xen 4.2.1, thing, is there a specific reason why you use it instead, of, e.g., 4.3.x ? Since you are building from source anyway, could you at least try Xen 4.3, to see whether the issue could be a bug in 4.2's libxl? I'm asking because, with both Xen 4.3 and the curret git tip, I don't have issues destroying domains with virsh. Thanks and Regards, Dario On gio, 2013-12-19 at 15:42 +0530, cool dharma06 wrote:
i did the debugging as you said. Kindly refer the following logs:
(gdb) c Continuing. thread apply all bt [New Thread 0x7fc337c6b700 (LWP 29520)]
Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) thread apply all bt
Thread 12 (Thread 0x7fc337c6b700 (LWP 29520)): #0 0x00007fc33509f18d in read () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc32b1555c4 in read_all (fd=26, data=0x7fc3380e0a70, data@entry=0x20, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:365 #2 0x00007fc32b1556d8 in read_message (h=h@entry=0x7fc324013ee0, nonblocking=nonblocking@entry=0) at xs.c:1071 #3 0x00007fc32b156005 in read_thread (arg=0x7fc324013ee0) at xs.c:1137 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 11 (Thread 0x7fc330e7f700 (LWP 29170)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7fc33067e700 (LWP 29171)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7fc32fe7d700 (LWP 29172)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7fc32f67c700 (LWP 29173)): #0 0x00007fc33497cb9f in realloc () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fc3349eec85 in __vasprintf_chk () from /lib/x86_64-linux-gnu/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc3371aac0a in vasprintf (__ap=0x7fc32f67b6d0, __fmt=0x7fc337391017 "%llu: %s : %s:%d : %s\n", __ptr=0x7fc32f67b830) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:199 #3 virVasprintfInternal (report=report@entry=false, domcode=0, filename=0x0, funcname=0x0, linenr=0, strp=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n", list=list@entry=0x7fc32f67b6d0) at util/virstring.c:337 #4 0x00007fc3371aad1b in virAsprintfInternal (report=report@entry=false, domcode=domcode@entry=0, filename=filename@entry=0x0, funcname=funcname@entry=0x0, linenr=linenr@entry=0, strp=strp@entry=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n") at util/virstring.c:358 #5 0x00007fc33718cb59 in virLogFormatString (str=<optimized out>, priority=VIR_LOG_DEBUG, funcname=0x7fc337394734 "virObjectRef", linenr=293, msg=0x7fc32f67b830) at util/virlog.c:719 #6 virLogVMessage (source=VIR_LOG_FROM_TRACE, priority=VIR_LOG_DEBUG, filename=0x7fc337394597 "util/virobject.c", linenr=293, funcname=0x7fc337394734 "virObjectRef", metadata=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p", vargs=vargs@entry=0x7fc32f67b8c0) at util/virlog.c:846 #7 0x00007fc33718cfc7 in virLogMessage (source=source@entry=VIR_LOG_FROM_TRACE, priority=priority@entry=VIR_LOG_DEBUG, filename=filename@entry=0x7fc337394597 "util/virobject.c", linenr=linenr@entry=293, funcname=funcname@entry=0x7fc337394734 "virObjectRef", metadata=metadata@entry=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p") at util/virlog.c:778 #8 0x00007fc33719c3d7 in virObjectRef (anyobj=0x7fc338102720) at util/virobject.c:293 #9 0x00007fc32af1824f in libxlDriverConfigGet (driver=driver@entry=0x7fc338108930) at libxl/libxl_conf.c:1124 #10 0x00007fc32af1a50e in libxlVmCleanup (driver=driver@entry=0x7fc338108930, vm=vm@entry=0x7fc32400f530, reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:264 #11 0x00007fc32af20274 in libxlVmReap (driver=0x7fc338108930, vm=0x7fc32400f530, reason=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:338 #12 0x00007fc32af207de in libxlDomainDestroyFlags (dom=0x7fc3380e3770, flags=<optimized out>) at libxl/libxl_driver.c:1507 #13 0x00007fc33723cd37 in virDomainDestroy (domain=domain@entry=0x7fc3380e3770) at libvirt.c:2342 #14 0x00007fc337cb9ba4 in remoteDispatchDomainDestroy (args=<optimized out>, rerr=0x7fc32f67bcd0, client=0x7fc32400ee60, server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:3165 #15 remoteDispatchDomainDestroyHelper (server=<optimized out>, client=0x7fc32400ee60, msg=<optimized out>, rerr=0x7fc32f67bcd0, args=<optimized out>, ret=<optimized out>) at remote_dispatch.h:3143 #16 0x00007fc3372b4d79 in virNetServerProgramDispatchCall (msg=0x7fc324014070, client=0x7fc32400ee60, server=0x7fc3380dbba0, prog=0x7fc324007750) at rpc/virnetserverprogram.c:435 #17 virNetServerProgramDispatch (prog=0x7fc324007750, server=server@entry=0x7fc3380dbba0, client=0x7fc32400ee60, msg=0x7fc324014070) at rpc/virnetserverprogram.c:305 #18 0x00007fc3372aea78 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7fc3380dbba0) at rpc/virnetserver.c:165 #19 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fc3380dbba0) at rpc/virnetserver.c:186 #20 0x00007fc3371af02e in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:144 #21 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #22 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #23 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #24 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fc32ee7b700 (LWP 29174)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 ---Type <return> to continue, or q <return> to quit--- #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fc32e67a700 (LWP 29175)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fc32de79700 (LWP 29176)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fc32d678700 (LWP 29177)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fc32ce77700 (LWP 29178)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit--- Thread 2 (Thread 0x7fc32c676700 (LWP 29179)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fc337c4d7c0 (LWP 29169)): #0 0x0000000000000000 in ?? () #1 0x00007fc32ace881e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7fc3380edf38) at libxl_event.c:1039 #2 0x00007fc32af199c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7fc3380e3310) at libxl/libxl_domain.c:216 #3 0x00007fc33717cf7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007fc33717b70b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007fc3372b0065 in virNetServerRun (srv=0x7fc3380dbba0) at rpc/virnetserver.c:1112 #7 0x00007fc337c9851c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb) (gdb)
****
regards, cooldharma06.
On Thu, Dec 19, 2013 at 3:30 PM, Daniel P. Berrange <berrange@redhat.com> wrote: On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote: > I am newbie to this gdb debugging. now i did the bt in the debugging. In my > side still it showing the error. Following are the stacktrace which i got. > Kindly refer the following logs for full information: > > **** > [New Thread 0x7f8503695700 (LWP 20012)] > > Program received signal SIGSEGV, Segmentation fault. > 0x0000000000000000 in ?? () > (gdb) bt > #0 0x0000000000000000 in ?? () > #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized > out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 > #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized > out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 > #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at > util/vireventpoll.c:451 > #4 virEventPollRunOnce () at util/vireventpoll.c:644 > #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 > #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at > rpc/virnetserver.c:1112 > #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) > at libvirtd.c:1513 > (gdb)
In this case, that stack trace looks like it ought to be sufficient to identify the problem - a clear NULL pointer reference. In general though you want to capture a stack trace of all threads in a program
eg instead of 'bt', do 'thread apply all bt' next time.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Dario Faggioli writes ("Segfault in libxl_osevent_occurred_timeout [Was: Re: [libvirt-users] About debugging of libvirt.]"):
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel]
4.2.1 is lacking libxl: fix stale timeout event callback race libxl: fix stale fd event callback race These are both in stable-4.2 as 6f0f339dd4378d062a211969f45cd23af12bf386 a87ef897295ec17788e41e9a8f4c0ada7a5a45f8 Around the time I was developing those in xen-unstable.git there were some libvirt patches which were necessary too. Ian.
On gio, 2013-12-19 at 11:46 +0000, Ian Jackson wrote:
Dario Faggioli writes ("Segfault in libxl_osevent_occurred_timeout [Was: Re: [libvirt-users] About debugging of libvirt.]"):
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel]
4.2.1 is lacking libxl: fix stale timeout event callback race libxl: fix stale fd event callback race
These are both in stable-4.2 as 6f0f339dd4378d062a211969f45cd23af12bf386 a87ef897295ec17788e41e9a8f4c0ada7a5a45f8
I see. Thanks Ian.
Around the time I was developing those in xen-unstable.git there were some libvirt patches which were necessary too.
That should be ok, as he's using the very latest libvirt. For Xen, cooldharma06, you really should at least upgrade to 4.2.3, which is the latest available 4.2.x. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
Dario Faggioli wrote:
On gio, 2013-12-19 at 11:46 +0000, Ian Jackson wrote:
Dario Faggioli writes ("Segfault in libxl_osevent_occurred_timeout [Was: Re: [libvirt-users] About debugging of libvirt.]"):
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel]
4.2.1 is lacking libxl: fix stale timeout event callback race libxl: fix stale fd event callback race
These are both in stable-4.2 as 6f0f339dd4378d062a211969f45cd23af12bf386 a87ef897295ec17788e41e9a8f4c0ada7a5a45f8
I see.
Thanks Ian.
Yep, both of those are definitely needed on the Xen side.
Around the time I was developing those in xen-unstable.git there were some libvirt patches which were necessary too.
That should be ok, as he's using the very latest libvirt.
Correct, all of the libvirt-related patches have been in for a while, so he should have them. Regards, Jim
On gio, 2013-12-19 at 11:46 +0000, Ian Jackson wrote:
Dario Faggioli writes ("Segfault in libxl_osevent_occurred_timeout [Was: Re: [libvirt-users] About debugging of libvirt.]"):
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel]
4.2.1 is lacking libxl: fix stale timeout event callback race libxl: fix stale fd event callback race
These are both in stable-4.2 as 6f0f339dd4378d062a211969f45cd23af12bf386 a87ef897295ec17788e41e9a8f4c0ada7a5a45f8
I see. Thanks Ian.
Around the time I was developing those in xen-unstable.git there were some libvirt patches which were necessary too.
That should be ok, as he's using the very latest libvirt. For Xen, cooldharma06, you really should at least upgrade to 4.2.3, considering it is the latest 4.2.x release, which is really what ought to be used. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
thanks to all and that is working with libvirt-1.2.0 and xen-4.3.1. thanks to all people for their valuable suggestions. regards, cooldharma06. On Thu, Dec 19, 2013 at 5:01 PM, Dario Faggioli <dario.faggioli@citrix.com>wrote:
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel]
Jim,
cooldharma06 reports having issues when destroying VMs with libvirt 1.2.0 and Xen 4.2.1 (is that the case, cooldharma06)?
Look at the stack trace at the end of this message, or here: https://bugzilla.redhat.com/show_bug.cgi?id=1044838
Ian, since it looks like events are involved... Any idea?
cooldharma06, about the Xen 4.2.1, thing, is there a specific reason why you use it instead, of, e.g., 4.3.x ? Since you are building from source anyway, could you at least try Xen 4.3, to see whether the issue could be a bug in 4.2's libxl? I'm asking because, with both Xen 4.3 and the curret git tip, I don't have issues destroying domains with virsh.
Thanks and Regards, Dario
On gio, 2013-12-19 at 15:42 +0530, cool dharma06 wrote:
i did the debugging as you said. Kindly refer the following logs:
(gdb) c Continuing. thread apply all bt [New Thread 0x7fc337c6b700 (LWP 29520)]
Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) thread apply all bt
Thread 12 (Thread 0x7fc337c6b700 (LWP 29520)): #0 0x00007fc33509f18d in read () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc32b1555c4 in read_all (fd=26, data=0x7fc3380e0a70, data@entry=0x20, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:365 #2 0x00007fc32b1556d8 in read_message (h=h@entry=0x7fc324013ee0, nonblocking=nonblocking@entry=0) at xs.c:1071 #3 0x00007fc32b156005 in read_thread (arg=0x7fc324013ee0) at xs.c:1137 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 11 (Thread 0x7fc330e7f700 (LWP 29170)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7fc33067e700 (LWP 29171)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7fc32fe7d700 (LWP 29172)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7fc32f67c700 (LWP 29173)): #0 0x00007fc33497cb9f in realloc () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fc3349eec85 in __vasprintf_chk () from /lib/x86_64-linux-gnu/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc3371aac0a in vasprintf (__ap=0x7fc32f67b6d0, __fmt=0x7fc337391017 "%llu: %s : %s:%d : %s\n", __ptr=0x7fc32f67b830) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:199 #3 virVasprintfInternal (report=report@entry=false, domcode=0, filename=0x0, funcname=0x0, linenr=0, strp=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n", list=list@entry=0x7fc32f67b6d0) at util/virstring.c:337 #4 0x00007fc3371aad1b in virAsprintfInternal (report=report@entry=false, domcode=domcode@entry=0, filename=filename@entry=0x0, funcname=funcname@entry=0x0, linenr=linenr@entry=0, strp=strp@entry=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n") at util/virstring.c:358 #5 0x00007fc33718cb59 in virLogFormatString (str=<optimized out>, priority=VIR_LOG_DEBUG, funcname=0x7fc337394734 "virObjectRef", linenr=293, msg=0x7fc32f67b830) at util/virlog.c:719 #6 virLogVMessage (source=VIR_LOG_FROM_TRACE, priority=VIR_LOG_DEBUG, filename=0x7fc337394597 "util/virobject.c", linenr=293, funcname=0x7fc337394734 "virObjectRef", metadata=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p", vargs=vargs@entry=0x7fc32f67b8c0) at util/virlog.c:846 #7 0x00007fc33718cfc7 in virLogMessage (source=source@entry=VIR_LOG_FROM_TRACE, priority=priority@entry=VIR_LOG_DEBUG, filename=filename@entry=0x7fc337394597 "util/virobject.c", linenr=linenr@entry=293, funcname=funcname@entry=0x7fc337394734 "virObjectRef", metadata=metadata@entry=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p") at util/virlog.c:778 #8 0x00007fc33719c3d7 in virObjectRef (anyobj=0x7fc338102720) at util/virobject.c:293 #9 0x00007fc32af1824f in libxlDriverConfigGet (driver=driver@entry=0x7fc338108930) at libxl/libxl_conf.c:1124 #10 0x00007fc32af1a50e in libxlVmCleanup (driver=driver@entry=0x7fc338108930, vm=vm@entry=0x7fc32400f530, reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:264 #11 0x00007fc32af20274 in libxlVmReap (driver=0x7fc338108930, vm=0x7fc32400f530, reason=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:338 #12 0x00007fc32af207de in libxlDomainDestroyFlags (dom=0x7fc3380e3770, flags=<optimized out>) at libxl/libxl_driver.c:1507 #13 0x00007fc33723cd37 in virDomainDestroy (domain=domain@entry=0x7fc3380e3770) at libvirt.c:2342 #14 0x00007fc337cb9ba4 in remoteDispatchDomainDestroy (args=<optimized out>, rerr=0x7fc32f67bcd0, client=0x7fc32400ee60, server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:3165 #15 remoteDispatchDomainDestroyHelper (server=<optimized out>, client=0x7fc32400ee60, msg=<optimized out>, rerr=0x7fc32f67bcd0, args=<optimized out>, ret=<optimized out>) at remote_dispatch.h:3143 #16 0x00007fc3372b4d79 in virNetServerProgramDispatchCall (msg=0x7fc324014070, client=0x7fc32400ee60, server=0x7fc3380dbba0, prog=0x7fc324007750) at rpc/virnetserverprogram.c:435 #17 virNetServerProgramDispatch (prog=0x7fc324007750, server=server@entry=0x7fc3380dbba0, client=0x7fc32400ee60, msg=0x7fc324014070) at rpc/virnetserverprogram.c:305 #18 0x00007fc3372aea78 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7fc3380dbba0) at rpc/virnetserver.c:165 #19 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fc3380dbba0) at rpc/virnetserver.c:186 #20 0x00007fc3371af02e in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:144 #21 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #22 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #23 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #24 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fc32ee7b700 (LWP 29174)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 ---Type <return> to continue, or q <return> to quit--- #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fc32e67a700 (LWP 29175)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fc32de79700 (LWP 29176)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fc32d678700 (LWP 29177)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fc32ce77700 (LWP 29178)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit--- Thread 2 (Thread 0x7fc32c676700 (LWP 29179)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fc337c4d7c0 (LWP 29169)): #0 0x0000000000000000 in ?? () #1 0x00007fc32ace881e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7fc3380edf38) at libxl_event.c:1039 #2 0x00007fc32af199c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7fc3380e3310) at libxl/libxl_domain.c:216 #3 0x00007fc33717cf7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007fc33717b70b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007fc3372b0065 in virNetServerRun (srv=0x7fc3380dbba0) at rpc/virnetserver.c:1112 #7 0x00007fc337c9851c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb) (gdb)
****
regards, cooldharma06.
On Thu, Dec 19, 2013 at 3:30 PM, Daniel P. Berrange <berrange@redhat.com> wrote: On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote: > I am newbie to this gdb debugging. now i did the bt in the debugging. In my > side still it showing the error. Following are the stacktrace which i got. > Kindly refer the following logs for full information: > > **** > [New Thread 0x7f8503695700 (LWP 20012)] > > Program received signal SIGSEGV, Segmentation fault. > 0x0000000000000000 in ?? () > (gdb) bt > #0 0x0000000000000000 in ?? () > #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized > out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 > #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized > out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 > #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at > util/vireventpoll.c:451 > #4 virEventPollRunOnce () at util/vireventpoll.c:644 > #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 > #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at > rpc/virnetserver.c:1112 > #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) > at libvirtd.c:1513 > (gdb)
In this case, that stack trace looks like it ought to be sufficient to identify the problem - a clear NULL pointer reference. In general though you want to capture a stack trace of all threads in a program
eg instead of 'bt', do 'thread apply all bt' next time.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
participants (4)
-
cool dharma06 -
Dario Faggioli -
Ian Jackson -
Jim Fehlig