[libvirt-users] About debugging of libvirt.
i am using libvirt-1.1.4 with xen(libxl). When destroying the VM its showing the following error : virsh # destroy 4 error: Failed to destroy domain 4 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor virsh # list error: failed to connect to the hypervisor error: no valid connection error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused already i send one request to this community to solve this problem. they suggested to do gdb. Any guide to do the gdb libvirtd. By googling i did the following things: 1. i enable the logoutput and loglevel in libvirtd.conf 2. restarted the libvirtd with the changed configuration file. 3. ps -aux | grep libvirtd 4. gdb libvirtd pid 5. then what i have to do to find the reason for this error. Suggest me some steps to solve this problem. regards, cooldharma06.
On mer, 2013-12-18 at 11:47 +0530, cool dharma06 wrote:
i am using libvirt-1.1.4 with xen(libxl). When destroying the VM its showing the following error :
virsh # destroy 4
error: Failed to destroy domain 4 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor
Libvirt 1.1.4 built from sources? Also, what version of Xen (and thus of libxl)? Build from sources or installed via distro packages?
virsh # list error: failed to connect to the hypervisor error: no valid connection error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
Mmm... I'm doing some libvirt testing/development on the libxl driver right in these days, and I've issued a lot of `virsh start' and `virsh stop' without seeing anything like this (I'm on trunk for both libvirt and xen, though).
already i send one request to this community to solve this problem. they suggested to do gdb.
Well, sure gdb could be useful... But what about checking the logs? Look in /var/log/libvirt/libxl.
Any guide to do the gdb libvirtd. By googling i did the following things:
1. i enable the logoutput and loglevel in libvirtd.conf 2. restarted the libvirtd with the changed configuration file. 3. ps -aux | grep libvirtd 4. gdb libvirtd pid 5. then what i have to do to find the reason for this error.
You have to trigger the crash (e.g., by issuing the offending command in another terminal window) and try to figure out where it is that libvirtd died, check the backtrace, etc. Once you have figured out that, you can set breakpoints right before the statements where it crashes, and try to figure out (by looking at variable values, etc) what is making it explode. Again, I'd recommend checking the logs (first) to help with that. Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
On 18.12.2013 07:17, cool dharma06 wrote:
i am using libvirt-1.1.4 with xen(libxl). When destroying the VM its showing the following error :
virsh # destroy 4 error: Failed to destroy domain 4 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor
virsh # list error: failed to connect to the hypervisor error: no valid connection error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
already i send one request to this community to solve this problem. they suggested to do gdb.
Any guide to do the gdb libvirtd. By googling i did the following things:
1. i enable the logoutput and loglevel in libvirtd.conf
http://wiki.libvirt.org/page/DebugLogs
2. restarted the libvirtd with the changed configuration file. 3. ps -aux | grep libvirtd 4. gdb libvirtd pid 5. then what i have to do to find the reason for this error.
You need to hit 'c[enter]' in the gdb console (and by [enter] I mean pressing the enter key not actually typing it onto console). Then you need to run 'virsh destroy 4' in a different console so you reproduce the crash. After this, when you return do gdb you should see a stack trace. It will point to line which caused the trouble. If not, you may need to install libvirt debug infos (actual command to do this is distro specific). Btw: you can join steps 3 and 4 into one: gdb -p $(pgrep libvirtd) Once you have the stack trace and the logs, please do open a new bug against libvirt: https://bugzilla.redhat.com/enter_bug.cgi?product=Virtualization%20Tools and attach the stack trace and the debug logs. Michal
i installed both xen-4.2.1 and libvirt-1.2.0 from the source only. i moved with the latest version of libvrirt. i did the gdb in libvirt-1.2.0 and i raised the bug. Please find the following link for the bug report: https://bugzilla.redhat.com/show_bug.cgi?id=1044838 regards, cooldharma06. On Wed, Dec 18, 2013 at 6:33 PM, Michal Privoznik <mprivozn@redhat.com>wrote:
On 18.12.2013 07:17, cool dharma06 wrote:
i am using libvirt-1.1.4 with xen(libxl). When destroying the VM its showing the following error :
virsh # destroy 4 error: Failed to destroy domain 4 error: End of file while reading data: Input/output error error: One or more references were leaked after disconnect from the hypervisor error: Failed to reconnect to the hypervisor
virsh # list error: failed to connect to the hypervisor error: no valid connection error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': Connection refused
already i send one request to this community to solve this problem. they suggested to do gdb.
Any guide to do the gdb libvirtd. By googling i did the following things:
1. i enable the logoutput and loglevel in libvirtd.conf
http://wiki.libvirt.org/page/DebugLogs
2. restarted the libvirtd with the changed configuration file. 3. ps -aux | grep libvirtd 4. gdb libvirtd pid 5. then what i have to do to find the reason for this error.
You need to hit 'c[enter]' in the gdb console (and by [enter] I mean pressing the enter key not actually typing it onto console). Then you need to run 'virsh destroy 4' in a different console so you reproduce the crash. After this, when you return do gdb you should see a stack trace. It will point to line which caused the trouble. If not, you may need to install libvirt debug infos (actual command to do this is distro specific).
Btw: you can join steps 3 and 4 into one:
gdb -p $(pgrep libvirtd)
Once you have the stack trace and the logs, please do open a new bug against libvirt:
https://bugzilla.redhat.com/enter_bug.cgi?product=Virtualization%20Tools
and attach the stack trace and the debug logs.
Michal
On gio, 2013-12-19 at 10:57 +0530, cool dharma06 wrote:
i installed both xen-4.2.1 and libvirt-1.2.0 from the source only. i moved with the latest version of libvrirt. i did the gdb in libvirt-1.2.0 and i raised the bug. Please find the following link for the bug report:
Thanks for doing that but, as I replied in the BZ, it's not very useful as it stands. As Michal said in his e-mail, you need debugging symbols, so that we can have an actual stack trace to look at. Like this, it's a needle in an haystack! :-/ Regards, Dario -- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
I am newbie to this gdb debugging. now i did the bt in the debugging. In my side still it showing the error. Following are the stacktrace which i got. Kindly refer the following logs for full information: **** [New Thread 0x7f8503695700 (LWP 20012)] Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at rpc/virnetserver.c:1112 #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb) ****** regards, cooldharma06. On Thu, Dec 19, 2013 at 2:52 PM, Dario Faggioli <dario.faggioli@citrix.com>wrote:
On gio, 2013-12-19 at 10:57 +0530, cool dharma06 wrote:
i installed both xen-4.2.1 and libvirt-1.2.0 from the source only. i moved with the latest version of libvrirt. i did the gdb in libvirt-1.2.0 and i raised the bug. Please find the following link for the bug report:
Thanks for doing that but, as I replied in the BZ, it's not very useful as it stands.
As Michal said in his e-mail, you need debugging symbols, so that we can have an actual stack trace to look at. Like this, it's a needle in an haystack! :-/
Regards, Dario
-- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote:
I am newbie to this gdb debugging. now i did the bt in the debugging. In my side still it showing the error. Following are the stacktrace which i got. Kindly refer the following logs for full information:
**** [New Thread 0x7f8503695700 (LWP 20012)]
Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at rpc/virnetserver.c:1112 #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb)
In this case, that stack trace looks like it ought to be sufficient to identify the problem - a clear NULL pointer reference. In general though you want to capture a stack trace of all threads in a program eg instead of 'bt', do 'thread apply all bt' next time. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
i did the debugging as you said. Kindly refer the following logs: (gdb) c Continuing. thread apply all bt [New Thread 0x7fc337c6b700 (LWP 29520)] Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) thread apply all bt Thread 12 (Thread 0x7fc337c6b700 (LWP 29520)): #0 0x00007fc33509f18d in read () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc32b1555c4 in read_all (fd=26, data=0x7fc3380e0a70, data@entry=0x20, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:365 #2 0x00007fc32b1556d8 in read_message (h=h@entry=0x7fc324013ee0, nonblocking=nonblocking@entry=0) at xs.c:1071 #3 0x00007fc32b156005 in read_thread (arg=0x7fc324013ee0) at xs.c:1137 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 11 (Thread 0x7fc330e7f700 (LWP 29170)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 10 (Thread 0x7fc33067e700 (LWP 29171)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 9 (Thread 0x7fc32fe7d700 (LWP 29172)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 8 (Thread 0x7fc32f67c700 (LWP 29173)): #0 0x00007fc33497cb9f in realloc () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fc3349eec85 in __vasprintf_chk () from /lib/x86_64-linux-gnu/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc3371aac0a in vasprintf (__ap=0x7fc32f67b6d0, __fmt=0x7fc337391017 "%llu: %s : %s:%d : %s\n", __ptr=0x7fc32f67b830) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:199 #3 virVasprintfInternal (report=report@entry=false, domcode=0, filename=0x0, funcname=0x0, linenr=0, strp=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n", list=list@entry=0x7fc32f67b6d0) at util/virstring.c:337 #4 0x00007fc3371aad1b in virAsprintfInternal (report=report@entry=false, domcode=domcode@entry=0, filename=filename@entry=0x0, funcname=funcname@entry=0x0, linenr=linenr@entry=0, strp=strp@entry=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n") at util/virstring.c:358 #5 0x00007fc33718cb59 in virLogFormatString (str=<optimized out>, priority=VIR_LOG_DEBUG, funcname=0x7fc337394734 "virObjectRef", linenr=293, msg=0x7fc32f67b830) at util/virlog.c:719 #6 virLogVMessage (source=VIR_LOG_FROM_TRACE, priority=VIR_LOG_DEBUG, filename=0x7fc337394597 "util/virobject.c", linenr=293, funcname=0x7fc337394734 "virObjectRef", metadata=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p", vargs=vargs@entry=0x7fc32f67b8c0) at util/virlog.c:846 #7 0x00007fc33718cfc7 in virLogMessage (source=source@entry=VIR_LOG_FROM_TRACE, priority=priority@entry=VIR_LOG_DEBUG, filename=filename@entry=0x7fc337394597 "util/virobject.c", linenr=linenr@entry=293, funcname=funcname@entry=0x7fc337394734 "virObjectRef", metadata=metadata@entry=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p") at util/virlog.c:778 #8 0x00007fc33719c3d7 in virObjectRef (anyobj=0x7fc338102720) at util/virobject.c:293 #9 0x00007fc32af1824f in libxlDriverConfigGet (driver=driver@entry=0x7fc338108930) at libxl/libxl_conf.c:1124 #10 0x00007fc32af1a50e in libxlVmCleanup (driver=driver@entry=0x7fc338108930, vm=vm@entry=0x7fc32400f530, reason=reason@entry =VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:264 #11 0x00007fc32af20274 in libxlVmReap (driver=0x7fc338108930, vm=0x7fc32400f530, reason=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:338 #12 0x00007fc32af207de in libxlDomainDestroyFlags (dom=0x7fc3380e3770, flags=<optimized out>) at libxl/libxl_driver.c:1507 #13 0x00007fc33723cd37 in virDomainDestroy (domain=domain@entry=0x7fc3380e3770) at libvirt.c:2342 #14 0x00007fc337cb9ba4 in remoteDispatchDomainDestroy (args=<optimized out>, rerr=0x7fc32f67bcd0, client=0x7fc32400ee60, server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:3165 #15 remoteDispatchDomainDestroyHelper (server=<optimized out>, client=0x7fc32400ee60, msg=<optimized out>, rerr=0x7fc32f67bcd0, args=<optimized out>, ret=<optimized out>) at remote_dispatch.h:3143 #16 0x00007fc3372b4d79 in virNetServerProgramDispatchCall (msg=0x7fc324014070, client=0x7fc32400ee60, server=0x7fc3380dbba0, prog=0x7fc324007750) at rpc/virnetserverprogram.c:435 #17 virNetServerProgramDispatch (prog=0x7fc324007750, server=server@entry=0x7fc3380dbba0, client=0x7fc32400ee60, msg=0x7fc324014070) at rpc/virnetserverprogram.c:305 #18 0x00007fc3372aea78 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7fc3380dbba0) at rpc/virnetserver.c:165 #19 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fc3380dbba0) at rpc/virnetserver.c:186 #20 0x00007fc3371af02e in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:144 #21 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #22 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #23 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #24 0x0000000000000000 in ?? () Thread 7 (Thread 0x7fc32ee7b700 (LWP 29174)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 ---Type <return> to continue, or q <return> to quit--- #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 6 (Thread 0x7fc32e67a700 (LWP 29175)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 5 (Thread 0x7fc32de79700 (LWP 29176)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 4 (Thread 0x7fc32d678700 (LWP 29177)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 3 (Thread 0x7fc32ce77700 (LWP 29178)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () ---Type <return> to continue, or q <return> to quit--- Thread 2 (Thread 0x7fc32c676700 (LWP 29179)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? () Thread 1 (Thread 0x7fc337c4d7c0 (LWP 29169)): #0 0x0000000000000000 in ?? () #1 0x00007fc32ace881e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7fc3380edf38) at libxl_event.c:1039 #2 0x00007fc32af199c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7fc3380e3310) at libxl/libxl_domain.c:216 #3 0x00007fc33717cf7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007fc33717b70b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007fc3372b0065 in virNetServerRun (srv=0x7fc3380dbba0) at rpc/virnetserver.c:1112 #7 0x00007fc337c9851c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb) (gdb) **** regards, cooldharma06. On Thu, Dec 19, 2013 at 3:30 PM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote:
I am newbie to this gdb debugging. now i did the bt in the debugging. In my side still it showing the error. Following are the stacktrace which i got. Kindly refer the following logs for full information:
**** [New Thread 0x7f8503695700 (LWP 20012)]
Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) bt #0 0x0000000000000000 in ?? () #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at rpc/virnetserver.c:1112 #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb)
In this case, that stack trace looks like it ought to be sufficient to identify the problem - a clear NULL pointer reference. In general though you want to capture a stack trace of all threads in a program
eg instead of 'bt', do 'thread apply all bt' next time.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
[Moving this to libvir, libvir-users in Bcc. Also, added xen-devel] Jim, cooldharma06 reports having issues when destroying VMs with libvirt 1.2.0 and Xen 4.2.1 (is that the case, cooldharma06)? Look at the stack trace at the end of this message, or here: https://bugzilla.redhat.com/show_bug.cgi?id=1044838 Ian, since it looks like events are involved... Any idea? cooldharma06, about the Xen 4.2.1, thing, is there a specific reason why you use it instead, of, e.g., 4.3.x ? Since you are building from source anyway, could you at least try Xen 4.3, to see whether the issue could be a bug in 4.2's libxl? I'm asking because, with both Xen 4.3 and the curret git tip, I don't have issues destroying domains with virsh. Thanks and Regards, Dario On gio, 2013-12-19 at 15:42 +0530, cool dharma06 wrote:
i did the debugging as you said. Kindly refer the following logs:
(gdb) c Continuing. thread apply all bt [New Thread 0x7fc337c6b700 (LWP 29520)]
Program received signal SIGSEGV, Segmentation fault. 0x0000000000000000 in ?? () (gdb) thread apply all bt
Thread 12 (Thread 0x7fc337c6b700 (LWP 29520)): #0 0x00007fc33509f18d in read () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc32b1555c4 in read_all (fd=26, data=0x7fc3380e0a70, data@entry=0x20, len=len@entry=16, nonblocking=nonblocking@entry=0) at xs.c:365 #2 0x00007fc32b1556d8 in read_message (h=h@entry=0x7fc324013ee0, nonblocking=nonblocking@entry=0) at xs.c:1071 #3 0x00007fc32b156005 in read_thread (arg=0x7fc324013ee0) at xs.c:1137 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 11 (Thread 0x7fc330e7f700 (LWP 29170)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 10 (Thread 0x7fc33067e700 (LWP 29171)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 9 (Thread 0x7fc32fe7d700 (LWP 29172)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 8 (Thread 0x7fc32f67c700 (LWP 29173)): #0 0x00007fc33497cb9f in realloc () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007fc3349eec85 in __vasprintf_chk () from /lib/x86_64-linux-gnu/libc.so.6 ---Type <return> to continue, or q <return> to quit--- #2 0x00007fc3371aac0a in vasprintf (__ap=0x7fc32f67b6d0, __fmt=0x7fc337391017 "%llu: %s : %s:%d : %s\n", __ptr=0x7fc32f67b830) at /usr/include/x86_64-linux-gnu/bits/stdio2.h:199 #3 virVasprintfInternal (report=report@entry=false, domcode=0, filename=0x0, funcname=0x0, linenr=0, strp=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n", list=list@entry=0x7fc32f67b6d0) at util/virstring.c:337 #4 0x00007fc3371aad1b in virAsprintfInternal (report=report@entry=false, domcode=domcode@entry=0, filename=filename@entry=0x0, funcname=funcname@entry=0x0, linenr=linenr@entry=0, strp=strp@entry=0x7fc32f67b830, fmt=fmt@entry=0x7fc337391017 "%llu: %s : %s:%d : %s\n") at util/virstring.c:358 #5 0x00007fc33718cb59 in virLogFormatString (str=<optimized out>, priority=VIR_LOG_DEBUG, funcname=0x7fc337394734 "virObjectRef", linenr=293, msg=0x7fc32f67b830) at util/virlog.c:719 #6 virLogVMessage (source=VIR_LOG_FROM_TRACE, priority=VIR_LOG_DEBUG, filename=0x7fc337394597 "util/virobject.c", linenr=293, funcname=0x7fc337394734 "virObjectRef", metadata=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p", vargs=vargs@entry=0x7fc32f67b8c0) at util/virlog.c:846 #7 0x00007fc33718cfc7 in virLogMessage (source=source@entry=VIR_LOG_FROM_TRACE, priority=priority@entry=VIR_LOG_DEBUG, filename=filename@entry=0x7fc337394597 "util/virobject.c", linenr=linenr@entry=293, funcname=funcname@entry=0x7fc337394734 "virObjectRef", metadata=metadata@entry=0x0, fmt=fmt@entry=0x7fc3373945f1 "OBJECT_REF: obj=%p") at util/virlog.c:778 #8 0x00007fc33719c3d7 in virObjectRef (anyobj=0x7fc338102720) at util/virobject.c:293 #9 0x00007fc32af1824f in libxlDriverConfigGet (driver=driver@entry=0x7fc338108930) at libxl/libxl_conf.c:1124 #10 0x00007fc32af1a50e in libxlVmCleanup (driver=driver@entry=0x7fc338108930, vm=vm@entry=0x7fc32400f530, reason=reason@entry=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:264 #11 0x00007fc32af20274 in libxlVmReap (driver=0x7fc338108930, vm=0x7fc32400f530, reason=VIR_DOMAIN_SHUTOFF_DESTROYED) at libxl/libxl_driver.c:338 #12 0x00007fc32af207de in libxlDomainDestroyFlags (dom=0x7fc3380e3770, flags=<optimized out>) at libxl/libxl_driver.c:1507 #13 0x00007fc33723cd37 in virDomainDestroy (domain=domain@entry=0x7fc3380e3770) at libvirt.c:2342 #14 0x00007fc337cb9ba4 in remoteDispatchDomainDestroy (args=<optimized out>, rerr=0x7fc32f67bcd0, client=0x7fc32400ee60, server=<optimized out>, msg=<optimized out>) at remote_dispatch.h:3165 #15 remoteDispatchDomainDestroyHelper (server=<optimized out>, client=0x7fc32400ee60, msg=<optimized out>, rerr=0x7fc32f67bcd0, args=<optimized out>, ret=<optimized out>) at remote_dispatch.h:3143 #16 0x00007fc3372b4d79 in virNetServerProgramDispatchCall (msg=0x7fc324014070, client=0x7fc32400ee60, server=0x7fc3380dbba0, prog=0x7fc324007750) at rpc/virnetserverprogram.c:435 #17 virNetServerProgramDispatch (prog=0x7fc324007750, server=server@entry=0x7fc3380dbba0, client=0x7fc32400ee60, msg=0x7fc324014070) at rpc/virnetserverprogram.c:305 #18 0x00007fc3372aea78 in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7fc3380dbba0) at rpc/virnetserver.c:165 #19 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7fc3380dbba0) at rpc/virnetserver.c:186 #20 0x00007fc3371af02e in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:144 #21 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #22 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #23 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #24 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fc32ee7b700 (LWP 29174)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbd80, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0cb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 ---Type <return> to continue, or q <return> to quit--- #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 6 (Thread 0x7fc32e67a700 (LWP 29175)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 5 (Thread 0x7fc32de79700 (LWP 29176)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 4 (Thread 0x7fc32d678700 (LWP 29177)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fc32ce77700 (LWP 29178)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db870) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit--- Thread 2 (Thread 0x7fc32c676700 (LWP 29179)): #0 0x00007fc33509c2d4 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0 #1 0x00007fc3371ae9ea in virCondWait (c=c@entry=0x7fc3380dbe18, m=m@entry=0x7fc3380dbd58) at util/virthreadpthread.c:117 #2 0x00007fc3371af0eb in virThreadPoolWorker (opaque=opaque@entry=0x7fc3380db640) at util/virthreadpool.c:103 #3 0x00007fc3371ae686 in virThreadHelper (data=<optimized out>) at util/virthreadpthread.c:161 #4 0x00007fc335097b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #5 0x00007fc3349daa3d in clone () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fc337c4d7c0 (LWP 29169)): #0 0x0000000000000000 in ?? () #1 0x00007fc32ace881e in libxl_osevent_occurred_timeout (ctx=<optimized out>, for_libxl=0x7fc3380edf38) at libxl_event.c:1039 #2 0x00007fc32af199c7 in libxlDomainObjTimerCallback (timer=<optimized out>, timer_info=0x7fc3380e3310) at libxl/libxl_domain.c:216 #3 0x00007fc33717cf7d in virEventPollDispatchTimeouts () at util/vireventpoll.c:451 #4 virEventPollRunOnce () at util/vireventpoll.c:644 #5 0x00007fc33717b70b in virEventRunDefaultImpl () at util/virevent.c:274 #6 0x00007fc3372b0065 in virNetServerRun (srv=0x7fc3380dbba0) at rpc/virnetserver.c:1112 #7 0x00007fc337c9851c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1513 (gdb) (gdb)
****
regards, cooldharma06.
On Thu, Dec 19, 2013 at 3:30 PM, Daniel P. Berrange <berrange@redhat.com> wrote: On Thu, Dec 19, 2013 at 03:09:14PM +0530, cool dharma06 wrote: > I am newbie to this gdb debugging. now i did the bt in the debugging. In my > side still it showing the error. Following are the stacktrace which i got. > Kindly refer the following logs for full information: > > **** > [New Thread 0x7f8503695700 (LWP 20012)] > > Program received signal SIGSEGV, Segmentation fault. > 0x0000000000000000 in ?? () > (gdb) bt > #0 0x0000000000000000 in ?? () > #1 0x00007f84f671281e in libxl_osevent_occurred_timeout (ctx=<optimized > out>, for_libxl=0x7f8503bbbc38) at libxl_event.c:1039 > #2 0x00007f84f69439c7 in libxlDomainObjTimerCallback (timer=<optimized > out>, timer_info=0x7f8503bb5930) at libxl/libxl_domain.c:216 > #3 0x00007f8502ba6f7d in virEventPollDispatchTimeouts () at > util/vireventpoll.c:451 > #4 virEventPollRunOnce () at util/vireventpoll.c:644 > #5 0x00007f8502ba570b in virEventRunDefaultImpl () at util/virevent.c:274 > #6 0x00007f8502cda065 in virNetServerRun (srv=0x7f8503ba6ba0) at > rpc/virnetserver.c:1112 > #7 0x00007f85036c251c in main (argc=<optimized out>, argv=<optimized out>) > at libvirtd.c:1513 > (gdb)
In this case, that stack trace looks like it ought to be sufficient to identify the problem - a clear NULL pointer reference. In general though you want to capture a stack trace of all threads in a program
eg instead of 'bt', do 'thread apply all bt' next time.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- <<This happens because I choose it to happen!>> (Raistlin Majere) ----------------------------------------------------------------- Dario Faggioli, Ph.D, http://about.me/dario.faggioli Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)
participants (4)
-
cool dharma06 -
Daniel P. Berrange -
Dario Faggioli -
Michal Privoznik