On Thu, Dec 19, 2013 at 5:01 PM, Dario Faggioli <dario.faggioli@citrix.com> wrote:

[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel]

Jim,

cooldharma06 reports having issues when destroying VMs with libvirt
1.2.0 and Xen 4.2.1 (is that the case, cooldharma06)?

Look at the stack trace at the end of this message, or here:
https://bugzilla.redhat.com/show_bug.cgi?id=1044838

Ian, since it looks like events are involved... Any idea?

cooldharma06, about the Xen 4.2.1, thing, is there a specific reason why
you use it instead, of, e.g., 4.3.x ? Since you are building from source
anyway, could you at least try Xen 4.3, to see whether the issue could
be a bug in 4.2's libxl? I'm asking because, with both Xen 4.3 and the
curret git tip, I don't have issues destroying domains with virsh.

Thanks and Regards,
Dario

On gio, 2013-12-19 at 15:42 +0530, cool dharma06 wrote:
> i did the debugging as you said. Kindly refer the following logs:
>
>
> (gdb) c
> Continuing.
> thread apply all bt
> [New Thread 0x7fc337c6b700 (LWP 29520)]
>
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x0000000000000000 in ?? ()
> (gdb) thread apply all bt
>
>
> Thread 12 (Thread 0x7fc337c6b700 (LWP 29520)):
> #0 0x00007fc33509f18d in read ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc32b1555c4 in read_all (fd=26, data=0x7fc3380e0a70,
> data@entry=0x20, len=len@entry=16, nonblocking=nonblocking@entry=0) at
> xs.c:365
> #2 0x00007fc32b1556d8 in read_message (h=h@entry=0x7fc324013ee0,
> nonblocking=nonblocking@entry=0) at xs.c:1071
> #3 0x00007fc32b156005 in read_thread (arg=0x7fc324013ee0) at
> xs.c:1137
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 11 (Thread 0x7fc330e7f700 (LWP 29170)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 10 (Thread 0x7fc33067e700 (LWP 29171)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 9 (Thread 0x7fc32fe7d700 (LWP 29172)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 8 (Thread 0x7fc32f67c700 (LWP 29173)):
> #0 0x00007fc33497cb9f in realloc ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #1 0x00007fc3349eec85 in __vasprintf_chk ()
> from /lib/x86_64-linux-gnu/libc.so.6
> ---Type <return> to continue, or q <return> to quit---
> #2 0x00007fc3371aac0a in vasprintf (__ap=0x7fc32f67b6d0,
> __fmt=0x7fc337391017 "%llu: %s : %s:%d : %s\n", __ptr=0x7fc32f67b830)
> at /usr/include/x86_64-linux-gnu/bits/stdio2.h:199
> #3 virVasprintfInternal (report=report@entry=false, domcode=0,
> filename=0x0, funcname=0x0, linenr=0, strp=0x7fc32f67b830,
> fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n",
> list=list@entry=0x7fc32f67b6d0) at util/virstring.c:337
> #4 0x00007fc3371aad1b in virAsprintfInternal
> (report=report@entry=false, domcode=domcode@entry=0,
> filename=filename@entry=0x0, funcname=funcname@entry=0x0,
> linenr=linenr@entry=0, strp=strp@entry=0x7fc32f67b830,
> fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n") at
> util/virstring.c:358
> #5 0x00007fc33718cb59 in virLogFormatString (str=<optimized out>,
> priority=VIR_LOG_DEBUG, funcname=0x7fc337394734 "virObjectRef",
> linenr=293, msg=0x7fc32f67b830)
> at util/virlog.c:719
> #6 virLogVMessage (source=VIR_LOG_FROM_TRACE, priority=VIR_LOG_DEBUG,
> filename=0x7fc337394597 "util/virobject.c", linenr=293,
> funcname=0x7fc337394734 "virObjectRef",
> metadata=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p",
> vargs=vargs@entry=0x7fc32f67b8c0) at util/virlog.c:846
> #7 0x00007fc33718cfc7 in virLogMessage
> (source=source@entry=VIR_LOG_FROM_TRACE,
> priority=priority@entry=VIR_LOG_DEBUG,
> filename=filename@entry=0x7fc337394597 "util/virobject.c",
> linenr=linenr@entry=293, funcname=funcname@entry=0x7fc337394734
> "virObjectRef",
> metadata=metadata@entry=0x0, fmt=fmt@entry=0x7fc3373945f1
> "OBJECT_REF: obj=%p") at util/virlog.c:778
> #8 0x00007fc33719c3d7 in virObjectRef (anyobj=0x7fc338102720) at
> util/virobject.c:293
> #9 0x00007fc32af1824f in libxlDriverConfigGet
> (driver=driver@entry=0x7fc338108930) at libxl/libxl_conf.c:1124
> #10 0x00007fc32af1a50e in libxlVmCleanup
> (driver=driver@entry=0x7fc338108930, vm=vm@entry=0x7fc32400f530,
> reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED)
> at libxl/libxl_driver.c:264
> #11 0x00007fc32af20274 in libxlVmReap (driver=0x7fc338108930,
> vm=0x7fc32400f530, reason=VIR_DOMAIN_SHUTOFF_DESTROYED) at
> libxl/libxl_driver.c:338
> #12 0x00007fc32af207de in libxlDomainDestroyFlags (dom=0x7fc3380e3770,
> flags=<optimized out>) at libxl/libxl_driver.c:1507
> #13 0x00007fc33723cd37 in virDomainDestroy
> (domain=domain@entry=0x7fc3380e3770) at libvirt.c:2342
> #14 0x00007fc337cb9ba4 in remoteDispatchDomainDestroy (args=<optimized
> out>, rerr=0x7fc32f67bcd0, client=0x7fc32400ee60, server=<optimized
> out>, msg=<optimized out>)
> at remote_dispatch.h:3165
> #15 remoteDispatchDomainDestroyHelper (server=<optimized out>,
> client=0x7fc32400ee60, msg=<optimized out>, rerr=0x7fc32f67bcd0,
> args=<optimized out>,
> ret=<optimized out>) at remote_dispatch.h:3143
> #16 0x00007fc3372b4d79 in virNetServerProgramDispatchCall
> (msg=0x7fc324014070, client=0x7fc32400ee60, server=0x7fc3380dbba0,
> prog=0x7fc324007750)
> at rpc/virnetserverprogram.c:435
> #17 virNetServerProgramDispatch (prog=0x7fc324007750,
> server=server@entry=0x7fc3380dbba0, client=0x7fc32400ee60,
> msg=0x7fc324014070) at rpc/virnetserverprogram.c:305
> #18 0x00007fc3372aea78 in virNetServerProcessMsg (msg=<optimized out>,
> prog=<optimized out>, client=<optimized out>, srv=0x7fc3380dbba0) at
> rpc/virnetserver.c:165
> #19 virNetServerHandleJob (jobOpaque=<optimized out>,
> opaque=0x7fc3380dbba0) at rpc/virnetserver.c:186
> #20 0x00007fc3371af02e in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:144
> #21 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #22 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #23 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #24 0x0000000000000000 in ?? ()
>
>
> Thread 7 (Thread 0x7fc32ee7b700 (LWP 29174)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0cb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> ---Type <return> to continue, or q <return> to quit---
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 6 (Thread 0x7fc32e67a700 (LWP 29175)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 5 (Thread 0x7fc32de79700 (LWP 29176)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 4 (Thread 0x7fc32d678700 (LWP 29177)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 3 (Thread 0x7fc32ce77700 (LWP 29178)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> ---Type <return> to continue, or q <return> to quit---
> Thread 2 (Thread 0x7fc32c676700 (LWP 29179)):
> #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18,
> m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117
> #2 0x00007fc3371af0eb in virThreadPoolWorker
> (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103
> #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at
> util/virthreadpthread.c:161
> #4 0x00007fc335097b50 in start_thread ()
> from /lib/x86_64-linux-gnu/libpthread.so.0
> #5 0x00007fc3349daa3d in clone ()
> from /lib/x86_64-linux-gnu/libc.so.6
> #6 0x0000000000000000 in ?? ()
>
>
> Thread 1 (Thread 0x7fc337c4d7c0 (LWP 29169)):
> #0 0x0000000000000000 in ?? ()
> #1 0x00007fc32ace881e in libxl_osevent_occurred_timeout
> (ctx=<optimized out>, for_libxl=0x7fc3380edf38) at libxl_event.c:1039
> #2 0x00007fc32af199c7 in libxlDomainObjTimerCallback
> (timer=<optimized out>, timer_info=0x7fc3380e3310) at
> libxl/libxl_domain.c:216
> #3 0x00007fc33717cf7d in virEventPollDispatchTimeouts () at
> util/vireventpoll.c:451
> #4 virEventPollRunOnce () at util/vireventpoll.c:644
> #5 0x00007fc33717b70b in virEventRunDefaultImpl () at
> util/virevent.c:274
> #6 0x00007fc3372b0065 in virNetServerRun (srv=0x7fc3380dbba0) at
> rpc/virnetserver.c:1112
> #7 0x00007fc337c9851c in main (argc=<optimized out>, argv=<optimized
> out>) at libvirtd.c:1513
> (gdb)
> (gdb)
>
>
> ****
>
>
> regards,
> cooldharma06.
>
>
> On Thu, Dec 19, 2013 at 3:30 PM, Daniel P. Berrange
> <berrange@redhat.com> wrote:
> On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote:
> > I am newbie to this gdb debugging. now i did the bt in the
> debugging. In my
> > side still it showing the error. Following are the
> stacktrace which i got.
> > Kindly refer the following logs for full information:
> >
> > ****
> > [New Thread 0x7f8503695700 (LWP 20012)]
> >
> > Program received signal SIGSEGV, Segmentation fault.
> > 0x0000000000000000 in ?? ()
> > (gdb) bt
> > #0 0x0000000000000000 in ?? ()
> > #1 0x00007f84f671281e in libxl_osevent_occurred_timeout
> (ctx=<optimized
> > out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039
> > #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback
> (timer=<optimized
> > out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216
> > #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at
> > util/vireventpoll.c:451
> > #4 virEventPollRunOnce () at util/vireventpoll.c:644
> > #5 0x00007f8502ba570b in virEventRunDefaultImpl () at
> util/virevent.c:274
> > #6 0x00007f8502cda065 in virNetServerRun
> (srv=0x7f8503ba6ba0) at
> > rpc/virnetserver.c:1112
> > #7 0x00007f85036c251c in main (argc=<optimized out>,
> argv=<optimized out>)
> > at libvirtd.c:1513
> > (gdb)
>
>
> In this case, that stack trace looks like it ought to be
> sufficient
> to identify the problem - a clear NULL pointer reference. In
> general
> though you want to capture a stack trace of all threads in a
> program
>
> eg instead of 'bt', do 'thread apply all bt' next time.
>
> Regards,
> Daniel
> --
> |: http://berrange.com -o-
> http://www.flickr.com/photos/dberrange/ :|
> |: http://libvirt.org -o-
> http://virt-manager.org :|
> |: http://autobuild.org -o-
> http://search.cpan.org/~danberr/ :|
> |: http://entangle-photo.org -o-
> http://live.gnome.org/gtk-vnc :|
>
>

--
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)