[libvirt] Segfault in libxl_osevent_occurred_timeout [Was: Re: [libvirt-users] About debugging of libvirt.]

i did the debugging as you said. Kindly refer the following logs: (gdb) c Continuing. thread apply all bt [New Thread 0x7fc337c6b700 (LWP 29520)] Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) thread apply all bt Thread 12 (Thread 0x7fc337c6b700 (LWP 29520)): #0 0x00007fc33509f18d in read () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc32b1555c4 in read_all (fd=26, data=0x7fc3380e0a70, data@entry=0x20, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:365 #2 0x00007fc32b1556d8 in read_message (h=h@entry=0x7fc324013ee0, nonblocking=nonblocking@entry=0) at xs.c:1071 #3 0x00007fc32b156005 in read_thread (arg=0x7fc324013ee0) at xs.c:1137 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 11 (Thread 0x7fc330e7f700 (LWP 29170)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 10 (Thread 0x7fc33067e700 (LWP 29171)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 9 (Thread 0x7fc32fe7d700 (LWP 29172)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 8 (Thread 0x7fc32f67c700 (LWP 29173)): #0 0x00007fc33497cb9f in realloc () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fc3349eec85 in __vasprintf_chk () from /lib/x86_64-linux-gnu/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc3371aac0a in vasprintf (__ap=0x7fc32f67b6d0, __fmt=0x7fc337391017 "%llu: %s : %s:%d : %s\n", __ptr=0x7fc32f67b830) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:199 #3 virVasprintfInternal (report=report@entry=false, domcode=0, filename=0x0, funcname=0x0, linenr=0, strp=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n", list=list@entry=0x7fc32f67b6d0) at util/virstring.c:337 #4 0x00007fc3371aad1b in virAsprintfInternal (report=report@entry=false, domcode=domcode@entry=0, filename=filename@entry=0x0, funcname=funcname@entry=0x0, linenr=linenr@entry=0, strp=strp@entry=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n") at util/virstring.c:358 #5 0x00007fc33718cb59 in virLogFormatString (str=<optimized out>, priority=VIR_LOG_DEBUG, funcname=0x7fc337394734 "virObjectRef", linenr=293, msg=0x7fc32f67b830) at util/virlog.c:719 #6 virLogVMessage (source=VIR_LOG_FROM_TRACE, priority=VIR_LOG_DEBUG, filename=0x7fc337394597 "util/virobject.c", linenr=293, funcname=0x7fc337394734 "virObjectRef", metadata=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p", vargs=vargs@entry=0x7fc32f67b8c0) at util/virlog.c:846 #7 0x00007fc33718cfc7 in virLogMessage (source=source@entry=VIR_LOG_FROM_TRACE, priority=priority@entry=VIR_LOG_DEBUG, filename=filename@entry=0x7fc337394597 "util/virobject.c", linenr=linenr@entry=293, funcname=funcname@entry=0x7fc337394734 "virObjectRef", metadata=metadata@entry=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p") at util/virlog.c:778 #8 0x00007fc33719c3d7 in virObjectRef (anyobj=0x7fc338102720) at util/virobject.c:293 #9 0x00007fc32af1824f in libxlDriverConfigGet (driver=driver@entry=0x7fc338108930) at libxl/libxl_conf.c:1124 #10 0x00007fc32af1a50e in libxlVmCleanup (driver=driver@entry=0x7fc338108930, vm=vm@entry=0x7fc32400f530, reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:264 #11 0x00007fc32af20274 in libxlVmReap (driver=0x7fc338108930, vm=0x7fc32400f530, reason=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:338 #12 0x00007fc32af207de in libxlDomainDestroyFlags (dom=0x7fc3380e3770, flags=<optimized out>) at libxl/libxl_driver.c:1507 #13 0x00007fc33723cd37 in virDomainDestroy (domain=domain@entry=0x7fc3380e3770) at libvirt.c:2342 #14 0x00007fc337cb9ba4 in remoteDispatchDomainDestroy (args=<optimized out>, rerr=0x7fc32f67bcd0, client=0x7fc32400ee60, server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:3165 #15 remoteDispatchDomainDestroyHelper (server=<optimized out>, client=0x7fc32400ee60, msg=<optimized out>, rerr=0x7fc32f67bcd0, args=<optimized out>, ret=<optimized out>) at remote_dispatch.h:3143 #16 0x00007fc3372b4d79 in virNetServerProgramDispatchCall (msg=0x7fc324014070, client=0x7fc32400ee60, server=0x7fc3380dbba0, prog=0x7fc324007750) at rpc/virnetserverprogram.c:435 #17 virNetServerProgramDispatch (prog=0x7fc324007750, server=server@entry=0x7fc3380dbba0, client=0x7fc32400ee60, msg=0x7fc324014070) at rpc/virnetserverprogram.c:305 #18 0x00007fc3372aea78 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7fc3380dbba0) at rpc/virnetserver.c:165 #19 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fc3380dbba0) at rpc/virnetserver.c:186 #20 0x00007fc3371af02e in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:144 #21 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #22 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #23 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #24 0x0000000000000000 in ?? () Thread 7 (Thread 0x7fc32ee7b700 (LWP 29174)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 ---Type <return> to continue, or q <return> to quit--- #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 6 (Thread 0x7fc32e67a700 (LWP 29175)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 5 (Thread 0x7fc32de79700 (LWP 29176)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 4 (Thread 0x7fc32d678700 (LWP 29177)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 3 (Thread 0x7fc32ce77700 (LWP 29178)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- Thread 2 (Thread 0x7fc32c676700 (LWP 29179)): #0 0x00007fc33509c2d4 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 1 (Thread 0x7fc337c4d7c0 (LWP 29169)): #0 0x0000000000000000 in ?? () #1 0x00007fc32ace881e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7fc3380edf38) at libxl_event.c:1039 #2 0x00007fc32af199c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7fc3380e3310) at libxl/libxl_domain.c:216 #3 0x00007fc33717cf7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007fc33717b70b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007fc3372b0065 in virNetServerRun (srv=0x7fc3380dbba0) at rpc/virnetserver.c:1112 #7 0x00007fc337c9851c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb) (gdb) **** regards, cooldharma06. On Thu, Dec 19, 2013 at 3:30 PM, Daniel P. Berrange <berrange(a)redhat.com&gt; wrote: On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote: > I am newbie to this gdb debugging. now i did the bt in the debugging. In my > side still it showing the error. Following are the stacktrace which i got. > Kindly refer the following logs for full information: > > **** > [New Thread 0x7f8503695700 (LWP 20012)] > > Program received signal SIGSEGV, Segmentation fault. > 0x0000000000000000 in ?? () > (gdb) bt > #0 0x0000000000000000 in ?? () > #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized > out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 > #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized > out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 > #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at > util/vireventpoll.c:451 > #4 virEventPollRunOnce () at util/vireventpoll.c:644 > #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 > #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at > rpc/virnetserver.c:1112 > #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) > at libvirtd.c:1513 > (gdb) In this case, that stack trace looks like it ought to be sufficient to identify the problem - a clear NULL pointer reference. In general though you want to capture a stack trace of all threads in a program eg instead of 'bt', do 'thread apply all bt' next time. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|