[libvirt] libvirtd crash when attach-disk to VM
Hi all, I got a depressed problem(libvirtd crash with SIGABRT or SIGSEGV sometimes) when attach a nbd disk to a VM by using cmd as follow: virsh attach-disk 228 --source /dev/nbd0 --target vdd --sourcetype block --driver qemu --subdriver raw or just using virsh attach-disk 228 /dev/nbd0 vdd. and this problem occurs everytime when I attach a nbd disk to VM. the device nbd0 is create by qemu-nbd cmd: qemu-nbd -c /dev/nbd0 /home/hzwangpan/zero root@114-113-199-15:/home/hzwangpan# qemu-nbd -V qemu-nbd version 0.0.1 the 'zero' file was created by qemu-img cmd: qemu-img create -f raw zero 512M qemu-img version 1.1.0, Copyright (c) 2004-2008 Fabrice Bellard Some info of packages' version is listed below: root@114-113-199-15:/home/hzwangpan# dpkg -l | grep libvi ii libvirt-bin 0.9.12-4 programs for the libvirt library ii libvirt-dev 0.9.12-4 development files for the libvirt library ii libvirt0 0.9.12-4 library for interfacing with different virtualization systems ii libvirt0-dbg 0.9.12-4 library for interfacing with different virtualization systems root@114-113-199-15:/home/hzwangpan# virsh version Compiled against library: libvir 0.9.12 Using library: libvir 0.9.12 Using API: QEMU 0.9.12 Running hypervisor: QEMU 1.1.0 root@114-113-199-15:/home/hzwangpan# kvm -version QEMU emulator version 1.1.0 (qemu-kvm-1.1.0+dfsg-3, Debian), Copyright (c) 2003-2008 Fabrice Bellard root@114-113-199-15:/home/hzwangpan# uname -a Linux 114-113-199-15 3.2.0-3-amd64 #1 SMP Thu Jun 28 09:07:26 UTC 2012 x86_64 GNU/Linux root@114-113-199-15:/home/hzwangpan# cat /etc/issue Debian GNU/Linux wheezy/sid \n \l the backtrace info when libvirtd crash is list here: 1) one situation: *** glibc detected *** /usr/sbin/libvirtd: malloc(): memory corruption (fast): 0x000000000087bc10 *** ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(+0x75b46)[0x7ffff4608b46] /lib/x86_64-linux-gnu/libc.so.6(+0x79428)[0x7ffff460c428] /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x70)[0x7ffff460d960] /lib/x86_64-linux-gnu/libc.so.6(__strdup+0x22)[0x7ffff4612912] /usr/lib/libvirt.so.0(virJSONValueObjectAppend+0x39)[0x7ffff77b75c9] /usr/lib/libvirt.so.0(virJSONValueObjectAppendString+0x37)[0x7ffff77b7e87] /usr/sbin/libvirtd[0x4aa884] /usr/sbin/libvirtd[0x4ac3a7] /usr/sbin/libvirtd[0x49d303] /usr/sbin/libvirtd[0x4a8bfe] /usr/sbin/libvirtd[0x4b0814] /usr/sbin/libvirtd[0x484a5d] /usr/sbin/libvirtd[0x461d9e] /usr/lib/libvirt.so.0(virDomainAttachDevice+0xdd)[0x7ffff7846f5d] /usr/sbin/libvirtd[0x43ccfe] /usr/lib/libvirt.so.0(+0x129866)[0x7ffff788a866] /usr/lib/libvirt.so.0(+0x1254d1)[0x7ffff78864d1] /usr/lib/libvirt.so.0(+0x6273e)[0x7ffff77c373e] /usr/lib/libvirt.so.0(+0x621c9)[0x7ffff77c31c9] /lib/x86_64-linux-gnu/libpthread.so.0(+0x6b50)[0x7ffff4d27b50] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7ffff466b6dd] ======= Memory map: ======== 00400000-0053d000 r-xp 00000000 08:01 69457 /usr/sbin/libvirtd 0073c000-0073e000 r--p 0013c000 08:01 69457 /usr/sbin/libvirtd 0073e000-00746000 rw-p 0013e000 08:01 69457 /usr/sbin/libvirtd 00746000-00aa1000 rw-p 00000000 00:00 0 [heap] 7fffe0000000-7fffe021f000 rw-p 00000000 00:00 0 7fffe021f000-7fffe4000000 ---p 00000000 00:00 0 7fffe77ff000-7fffe7800000 ---p 00000000 00:00 0 7fffe7800000-7fffe8000000 rw-p 00000000 00:00 0 7fffe8000000-7fffe8175000 rw-p 00000000 00:00 0 7fffe8175000-7fffec000000 ---p 00000000 00:00 0 7fffec138000-7fffec139000 ---p 00000000 00:00 0 7fffec139000-7fffec939000 rw-p 00000000 00:00 0 7fffec939000-7fffec96e000 r--s 00000000 08:06 286722 /var/cache/nscd/passwd 7fffec96e000-7fffec99a000 rw-p 00000000 00:00 0 7fffec99a000-7fffec99b000 ---p 00000000 00:00 0 7fffec99b000-7fffed19b000 rw-p 00000000 00:00 0 7fffed19b000-7fffed315000 r-xp 00000000 08:01 124989 /usr/lib/x86_64-linux-gnu/libdb-5.1.so 7fffed315000-7fffed515000 ---p 0017a000 08:01 124989 /usr/lib/x86_64-linux-gnu/libdb-5.1.so 7fffed515000-7fffed51b000 r--p 0017a000 08:01 124989 /usr/lib/x86_64-linux-gnu/libdb-5.1.so 7fffed51b000-7fffed51e000 rw-p 00180000 08:01 124989 /usr/lib/x86_64-linux-gnu/libdb-5.1.so 7fffed51e000-7fffed524000 r-xp 00000000 08:01 133549 /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25 7fffed524000-7fffed723000 ---p 00006000 08:01 133549 /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25 7fffed723000-7fffed724000 r--p 00005000 08:01 133549 /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25 7fffed724000-7fffed725000 rw-p 00006000 08:01 133549 /usr/lib/x86_64-linux-gnu/sasl2/libsasldb.so.2.0.25 7fffed725000-7fffed75a000 r--s 00000000 08:06 286723 /var/cache/nscd/group 7fffed75a000-7fffed75b000 ---p 00000000 00:00 0 7fffed75b000-7fffedf5b000 rw-p 00000000 00:00 0 7fffedf5b000-7fffedf5c000 ---p 00000000 00:00 0 7fffedf5c000-7fffee75c000 rw-p 00000000 00:00 0 7fffee75c000-7fffee75d000 ---p 00000000 00:00 0 7fffee75d000-7fffeef5d000 rw-p 00000000 00:00 0 7fffeef5d000-7fffeef5e000 ---p 00000000 00:00 0 7fffeef5e000-7fffef75e000 rw-p 00000000 00:00 0 7fffef75e000-7fffef75f000 ---p 00000000 00:00 0 7fffef75f000-7fffeff5f000 rw-p 00000000 00:00 0 7fffeff5f000-7fffeff60000 ---p 00000000 00:00 0 7fffeff60000-7ffff0760000 rw-p 00000000 00:00 0 7ffff0760000-7ffff0761000 ---p 00000000 00:00 0 7ffff0761000-7ffff0f61000 rw-p 00000000 00:00 0 7ffff0f61000-7ffff0f62000 ---p 00000000 00:00 0 7ffff0f62000-7ffff1762000 rw-p 00000000 00:00 0 7ffff1762000-7ffff1763000 ---p 00000000 00:00 0 7ffff1763000-7ffff1f63000 rw-p 00000000 00:00 0 7ffff1f63000-7ffff1f64000 ---p 00000000 00:00 0 7ffff1f64000-7ffff2764000 rw-p 00000000 00:00 0 7ffff2764000-7ffff2799000 r--s 00000000 08:06 303114 /var/cache/nscd/hosts 7ffff2799000-7ffff27ae000 r-xp 00000000 08:01 57404 /lib/x86_64-linux-gnu/libgcc_s.so.1 7ffff27ae000-7ffff29ae000 ---p 00015000 08:01 57404 /lib/x86_64-linux-gnu/libgcc_s.so.1 7ffff29ae000-7ffff29af000 rw-p 00015000 08:01 57404 /lib/x86_64-linux-gnu/libgcc_s.so.1 7ffff29af000-7ffff29c7000 r-xp 00000000 08:01 110124 /usr/lib/libfa.so.1.3.5 7ffff29c7000-7ffff2bc7000 ---p 00018000 08:01 110124 /usr/lib/libfa.so.1.3.5 7ffff2bc7000-7ffff2bc8000 rw-p 00018000 08:01 110124 /usr/lib/libfa.so.1.3.5 7ffff2bc8000-7ffff2be9000 r-xp 00000000 08:01 57814 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7ffff2be9000-7ffff2de9000 ---p 00021000 08:01 57814 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7ffff2de9000-7ffff2dea000 r--p 00021000 08:01 57814 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7ffff2dea000-7ffff2deb000 rw-p 00022000 08:01 57814 /lib/x86_64-linux-gnu/liblzma.so.5.0.0 7ffff2deb000-7ffff2e6c000 r-xp 00000000 08:01 57526 /lib/x86_64-linux-gnu/libm-2.13.so 7ffff2e6c000-7ffff306b000 ---p 00081000 08:01 57526 /lib/x86_64-linux-gnu/libm-2.13.so 7ffff306b000-7ffff306c000 r--p 00080000 08:01 57526 /lib/x86_64-linux-gnu/libm-2.13.so 7ffff306c000-7ffff306d000 rw-p 00081000 08:01 57526 /lib/x86_64-linux-gnu/libm-2.13.so 7ffff306d000-7ffff30a9000 r-xp 00000000 08:01 76972 /usr/lib/x86_64-linux-gnu/libxslt.so.1.1.26 7ffff30a9000-7ffff32a8000 ---p 0003c000 08:01 76972 /usr/lib/x86_64-linux-gnu/libxslt.so.1.1.26 7ffff32a8000-7ffff32aa000 r--p 0003b000 08:01 76972 /usr/lib/x86_64-linux-gnu/libxslt.so.1.1.26 7ffff32aa000-7ffff32ab000 rw-p 0003d000 08:01 76972 /usr/lib/x86_64-linux-gnu/libxslt.so.1.1.26 7ffff32ab000-7ffff32bf000 r-xp 00000000 08:01 76971 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.15 7ffff32bf000-7ffff34be000 ---p 00014000 08:01 76971 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.15 7ffff34be000-7ffff34bf000 r--p 00013000 08:01 76971 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.15 7ffff34bf000-7ffff34c0000 rw-p 00014000 08:01 76971 /usr/lib/x86_64-linux-gnu/libexslt.so.0.8.15 7ffff34c0000-7ffff3519000 r-xp 00000000 08:01 93870 /usr/lib/libaugeas.so.0.15.0 7ffff3519000-7ffff3718000 ---p 00059000 08:01 93870 /usr/lib/libaugeas.so.0.15.0 7ffff3718000-7ffff371b000 rw-p 00058000 08:01 93870 /usr/lib/libaugeas.so.0.15.0 7ffff371b000-7ffff372e000 r-xp 00000000 08:01 57421 /lib/x86_64-linux-gnu/libresolv-2.13.so Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ffff1761700 (LWP 8216)] 0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff45c86f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff45ff2fb in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff4608b46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007ffff460c428 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ffff460d960 in malloc () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x00007ffff4612912 in strdup () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x00007ffff77b75c9 in virJSONValueObjectAppend (object=object@entry=0x91d810, key=key@entry=0x4fef81 "execute", value=value@entry=0x85de90) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:274 #8 0x00007ffff77b7e87 in virJSONValueObjectAppendString (object=object@entry=0x91d810, key=key@entry=0x4fef81 "execute", value=value@entry=0x51196f "human-monitor-command") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:296 #9 0x00000000004aa884 in qemuMonitorJSONMakeCommandRaw (wrap=wrap@entry=false, cmdname=cmdname@entry=0x51196f "human-monitor-command") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:404 #10 0x00000000004ac3a7 in qemuMonitorJSONHumanCommandWithFd (mon=mon@entry=0x7fffe80010b0, cmd_str=<optimized out>, scm_fd=-1, reply_str=0x7ffff1760920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:886 #11 0x000000000049d303 in qemuMonitorHMPCommandWithFd (mon=mon@entry=0x7fffe80010b0, cmd=<optimized out>, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7ffff1760920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:910 #12 0x00000000004a8bfe in qemuMonitorTextAddDrive (mon=mon@entry=0x7fffe80010b0, drivestr=drivestr@entry=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_text.c:2836 #13 0x00000000004b0814 in qemuMonitorJSONAddDrive (mon=0x7fffe80010b0, drivestr=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:2979 #14 0x00000000004a1bad in qemuMonitorAddDrive (mon=<optimized out>, drivestr=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:2571 #15 0x0000000000484a5d in qemuDomainAttachPciDiskDevice (conn=conn@entry=0x7fffe00111f0, driver=driver@entry=0x81fec0, vm=vm@entry=0x82f6b0, disk=disk@entry=0x7fffe02024d0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_hotplug.c:250 #16 0x0000000000461d9e in qemuDomainAttachDeviceDiskLive (vm=0x82f6b0, driver=0x81fec0, conn=0x7fffe00111f0, dev=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5179 #17 qemuDomainAttachDeviceLive (dev=0x7fffe001d5b0, vm=0x82f6b0, dom=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5238 #18 qemuDomainModifyDeviceFlags (dom=<optimized out>, xml=0x7fffe001d5b0 "\001", flags=<optimized out>, action=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5779 #19 0x00007ffff7846f5d in virDomainAttachDevice (domain=domain@entry=0x7fffe0201fb0, xml=0x7fffe0201e50 "<disk type='block'>\n <driver name='qemu' type='raw'/>\n <source dev='/dev/nbd0'/>\n <target dev='vdd'/>\n</disk>\n") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/libvirt.c:9288 #20 0x000000000043ccfe in remoteDispatchDomainAttachDevice (args=0x7fffe0201ff0, rerr=0x7ffff1760c90, client=<optimized out>, server=<optimized out>, msg=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:320 #21 remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7ffff1760c90, args=0x7fffe0201ff0, ret=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:298 #22 0x00007ffff788a866 in virNetServerProgramDispatchCall (msg=0x7fffe8093d20, client=0x7fffe8053050, server=0x76e920, prog=0x778880) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:416 #23 virNetServerProgramDispatch (prog=0x778880, server=server@entry=0x76e920, client=0x7fffe8053050, msg=0x7fffe8093d20) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:289 #24 0x00007ffff78864d1 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x76e920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserver.c:161 #25 0x00007ffff77c373e in virThreadPoolWorker (opaque=opaque@entry=0x7789a0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threadpool.c:144 #26 0x00007ffff77c31c9 in virThreadHelper (data=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threads-pthread.c:161 #27 0x00007ffff4d27b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #28 0x00007ffff466b6dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #29 0x0000000000000000 in ?? () (gdb) f 7 #7 0x00007ffff77b75c9 in virJSONValueObjectAppend (object=object@entry=0x91d810, key=key@entry=0x4fef81 "execute", value=value@entry=0x85de90) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:274 warning: Source file is more recent than executable. 274 if (!(newkey = strdup(key))) (gdb) l 269 return -1; 270 271 if (virJSONValueObjectHasKey(object, key)) 272 return -1; 273 274 if (!(newkey = strdup(key))) 275 return -1; 276 277 if (VIR_REALLOC_N(object->data.object.pairs, 278 object->data.object.npairs + 1) < 0) { (gdb) 2) another situation: Program received signal SIGABRT, Aborted. [Switching to Thread 0x7ffff075f700 (LWP 11697)] 0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff45c86f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff45ff2fb in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff4608b46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007ffff460d87c in free () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ffff77ba4c9 in virFree (ptrptr=ptrptr@entry=0x7ffff075e8e8) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/memory.c:310 #6 0x000000000049d321 in qemuMonitorHMPCommandWithFd (mon=mon@entry=0x9214e0, cmd=<optimized out>, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7ffff075e920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:916 #7 0x00000000004a8bfe in qemuMonitorTextAddDrive (mon=mon@entry=0x9214e0, drivestr=drivestr@entry=0x8f7090 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_text.c:2836 #8 0x00000000004b0814 in qemuMonitorJSONAddDrive (mon=0x9214e0, drivestr=0x8f7090 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:2979 #9 0x00000000004a1bad in qemuMonitorAddDrive (mon=<optimized out>, drivestr=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:2571 #10 0x0000000000484a5d in qemuDomainAttachPciDiskDevice (conn=conn@entry=0x9213c0, driver=driver@entry=0x8162c0, vm=vm@entry=0x820530, disk=disk@entry=0x8f5290) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_hotplug.c:250 #11 0x0000000000461d9e in qemuDomainAttachDeviceDiskLive (vm=0x820530, driver=0x8162c0, conn=0x9213c0, dev=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5179 #12 qemuDomainAttachDeviceLive (dev=0x8465b0, vm=0x820530, dom=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5238 #13 qemuDomainModifyDeviceFlags (dom=<optimized out>, xml=0x8465b0 "\001", flags=<optimized out>, action=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5779 #14 0x00007ffff7846f5d in virDomainAttachDevice (domain=domain@entry=0x8f6080, xml=0x8f5610 "<disk type='block'>\n <driver name='qemu' type='raw'/>\n <source dev='/dev/nbd0'/>\n <target dev='vdd'/>\n</disk>\n") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/libvirt.c:9288 #15 0x000000000043ccfe in remoteDispatchDomainAttachDevice (args=0x8f60c0, rerr=0x7ffff075ec90, client=<optimized out>, server=<optimized out>, msg=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:320 #16 remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7ffff075ec90, args=0x8f60c0, ret=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:298 #17 0x00007ffff788a866 in virNetServerProgramDispatchCall (msg=0x7fffe00929d0, client=0x7fffe0052880, server=0x76e920, prog=0x778880) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:416 #18 virNetServerProgramDispatch (prog=0x778880, server=server@entry=0x76e920, client=0x7fffe0052880, msg=0x7fffe00929d0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:289 #19 0x00007ffff78864d1 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x76e920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserver.c:161 #20 0x00007ffff77c373e in virThreadPoolWorker (opaque=opaque@entry=0x7789a0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threadpool.c:144 #21 0x00007ffff77c31c9 in virThreadHelper (data=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threads-pthread.c:161 #22 0x00007ffff4d27b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #23 0x00007ffff466b6dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #24 0x0000000000000000 in ?? () (gdb) f 6 #6 0x000000000049d321 in qemuMonitorHMPCommandWithFd (mon=mon@entry=0x9214e0, cmd=<optimized out>, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7ffff075e920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:916 916 VIR_FREE(json_cmd); (gdb) l 911 } else { 912 ret = qemuMonitorTextCommandWithFd(mon, cmd, scm_fd, reply); 913 } 914 915 cleanup: 916 VIR_FREE(json_cmd); 917 return ret; 918 } 919 920 /* Ensure proper locking around callbacks. */ (gdb) p json_cmd $3 = 0x933c20 "drive_add dummy file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw" (gdb) anybody can give me any clue? Thanks in advance! 2012-08-16 Wangpan
On 08/16/12 11:15, Wangpan wrote:
Hi all, I got a depressed problem(libvirtd crash with SIGABRT or SIGSEGV sometimes) when attach a nbd disk to a VM by using cmd as follow: virsh attach-disk 228 --source /dev/nbd0 --target vdd --sourcetype block --driver qemu --subdriver raw or just using virsh attach-disk 228 /dev/nbd0 vdd. and this problem occurs everytime when I attach a nbd disk to VM.
...
Some info of packages' version is listed below: root@114-113-199-15:/home/hzwangpan# dpkg -l | grep libvi ii libvirt-bin 0.9.12-4 programs for the libvirt library ii libvirt-dev 0.9.12-4 development files for the libvirt library ii libvirt0 0.9.12-4 library for interfacing with different virtualization systems ii libvirt0-dbg 0.9.12-4 library for interfacing with different virtualization systems
You probably (looking at the version numbers) came across a known bug: https://bugzilla.redhat.com/show_bug.cgi?id=822068
0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff45c86f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff45ff2fb in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff4608b46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007ffff460c428 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ffff460d960 in malloc () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x00007ffff4612912 in strdup () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x00007ffff77b75c9 in virJSONValueObjectAppend (object=object@entry=0x91d810, key=key@entry=0x4fef81 "execute", value=value@entry=0x85de90) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:274 #8 0x00007ffff77b7e87 in virJSONValueObjectAppendString (object=object@entry=0x91d810, key=key@entry=0x4fef81 "execute", value=value@entry=0x51196f "human-monitor-command") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:296 #9 0x00000000004aa884 in qemuMonitorJSONMakeCommandRaw (wrap=wrap@entry=false, cmdname=cmdname@entry=0x51196f "human-monitor-command") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:404 #10 0x00000000004ac3a7 in qemuMonitorJSONHumanCommandWithFd (mon=mon@entry=0x7fffe80010b0, cmd_str=<optimized out>, scm_fd=-1, reply_str=0x7ffff1760920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:886 #11 0x000000000049d303 in qemuMonitorHMPCommandWithFd (mon=mon@entry=0x7fffe80010b0, cmd=<optimized out>, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7ffff1760920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:910 #12 0x00000000004a8bfe in qemuMonitorTextAddDrive (mon=mon@entry=0x7fffe80010b0, drivestr=drivestr@entry=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_text.c:2836 #13 0x00000000004b0814 in qemuMonitorJSONAddDrive (mon=0x7fffe80010b0, drivestr=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:2979 #14 0x00000000004a1bad in qemuMonitorAddDrive (mon=<optimized out>, drivestr=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:2571 #15 0x0000000000484a5d in qemuDomainAttachPciDiskDevice (conn=conn@entry=0x7fffe00111f0, driver=driver@entry=0x81fec0, vm=vm@entry=0x82f6b0, disk=disk@entry=0x7fffe02024d0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_hotplug.c:250 #16 0x0000000000461d9e in qemuDomainAttachDeviceDiskLive (vm=0x82f6b0, driver=0x81fec0, conn=0x7fffe00111f0, dev=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5179 #17 qemuDomainAttachDeviceLive (dev=0x7fffe001d5b0, vm=0x82f6b0, dom=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5238 #18 qemuDomainModifyDeviceFlags (dom=<optimized out>, xml=0x7fffe001d5b0 "\001", flags=<optimized out>, action=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5779 #19 0x00007ffff7846f5d in virDomainAttachDevice (domain=domain@entry=0x7fffe0201fb0, xml=0x7fffe0201e50 "<disk type='block'>\n <driver name='qemu' type='raw'/>\n <source dev='/dev/nbd0'/>\n <target dev='vdd'/>\n</disk>\n") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/libvirt.c:9288 #20 0x000000000043ccfe in remoteDispatchDomainAttachDevice (args=0x7fffe0201ff0, rerr=0x7ffff1760c90, client=<optimized out>, server=<optimized out>, msg=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:320 #21 remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7ffff1760c90, args=0x7fffe0201ff0, ret=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:298 #22 0x00007ffff788a866 in virNetServerProgramDispatchCall (msg=0x7fffe8093d20, client=0x7fffe8053050, server=0x76e920, prog=0x778880) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:416 #23 virNetServerProgramDispatch (prog=0x778880, server=server@entry=0x76e920, client=0x7fffe8053050, msg=0x7fffe8093d20) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:289 #24 0x00007ffff78864d1 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x76e920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserver.c:161 #25 0x00007ffff77c373e in virThreadPoolWorker (opaque=opaque@entry=0x7789a0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threadpool.c:144 #26 0x00007ffff77c31c9 in virThreadHelper (data=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threads-pthread.c:161 #27 0x00007ffff4d27b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #28 0x00007ffff466b6dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #29 0x0000000000000000 in ?? () (gdb) f 7
This backtrace is identical with that attached to the bug. The bug is fixed by commit: commit 0f4660c8787cc41fe67f869984c0ae11d680037e Author: Peter Krempa <pkrempa@redhat.com> Date: Thu Jun 14 10:29:36 2012 +0200 qemu: Fix off-by-one error while unescaping monitor strings While unescaping the commands the commands passed through to the monitor function qemuMonitorUnescapeArg() initialized lenght of the input string to strlen()+1 which is fine for alloc but not for iteration of the string. That is included in the 0.9.13 release. To fix this issue please upgrade or propose to backport that patch into your distro. At any rate thanks for the exhausting bug report, it definitely helped identifying the issue and would be useful in fixing it. Peter
Thanks Peter, I have verified this bug on libvirt-0.9.13 by compiling the source tarball, It's OK now! 2012-08-19 Wangpan
You probably (looking at the version numbers) came across a known bug: https://bugzilla.redhat.com/show_bug.cgi?id=822068
0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 (gdb) bt #0 0x00007ffff45c5475 in raise () from /lib/x86_64-linux-gnu/libc.so.6 #1 0x00007ffff45c86f0 in abort () from /lib/x86_64-linux-gnu/libc.so.6 #2 0x00007ffff45ff2fb in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #3 0x00007ffff4608b46 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #4 0x00007ffff460c428 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 #5 0x00007ffff460d960 in malloc () from /lib/x86_64-linux-gnu/libc.so.6 #6 0x00007ffff4612912 in strdup () from /lib/x86_64-linux-gnu/libc.so.6 #7 0x00007ffff77b75c9 in virJSONValueObjectAppend (object=object@entry=0x91d810, key=key@entry=0x4fef81 "execute", value=value@entry=0x85de90) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:274 #8 0x00007ffff77b7e87 in virJSONValueObjectAppendString (object=object@entry=0x91d810, key=key@entry=0x4fef81 "execute", value=value@entry=0x51196f "human-monitor-command") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/json.c:296 #9 0x00000000004aa884 in qemuMonitorJSONMakeCommandRaw (wrap=wrap@entry=false, cmdname=cmdname@entry=0x51196f "human-monitor-command") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:404 #10 0x00000000004ac3a7 in qemuMonitorJSONHumanCommandWithFd (mon=mon@entry=0x7fffe80010b0, cmd_str=<optimized out>, scm_fd=-1, reply_str=0x7ffff1760920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:886 #11 0x000000000049d303 in qemuMonitorHMPCommandWithFd (mon=mon@entry=0x7fffe80010b0, cmd=<optimized out>, scm_fd=scm_fd@entry=-1, reply=reply@entry=0x7ffff1760920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:910 #12 0x00000000004a8bfe in qemuMonitorTextAddDrive (mon=mon@entry=0x7fffe80010b0, drivestr=drivestr@entry=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_text.c:2836 #13 0x00000000004b0814 in qemuMonitorJSONAddDrive (mon=0x7fffe80010b0, drivestr=0x7fffe0202020 "file=/dev/nbd0,if=none,id=drive-virtio-disk3,format=raw") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor_json.c:2979 #14 0x00000000004a1bad in qemuMonitorAddDrive (mon=<optimized out>, drivestr=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_monitor.c:2571 #15 0x0000000000484a5d in qemuDomainAttachPciDiskDevice (conn=conn@entry=0x7fffe00111f0, driver=driver@entry=0x81fec0, vm=vm@entry=0x82f6b0, disk=disk@entry=0x7fffe02024d0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_hotplug.c:250 #16 0x0000000000461d9e in qemuDomainAttachDeviceDiskLive (vm=0x82f6b0, driver=0x81fec0, conn=0x7fffe00111f0, dev=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5179 #17 qemuDomainAttachDeviceLive (dev=0x7fffe001d5b0, vm=0x82f6b0, dom=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5238 #18 qemuDomainModifyDeviceFlags (dom=<optimized out>, xml=0x7fffe001d5b0 "\001", flags=<optimized out>, action=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/qemu/qemu_driver.c:5779 #19 0x00007ffff7846f5d in virDomainAttachDevice (domain=domain@entry=0x7fffe0201fb0, xml=0x7fffe0201e50 "<disk type='block'>\n <driver name='qemu' type='raw'/>\n <source dev='/dev/nbd0'/>\n <target dev='vdd'/>\n</disk>\n") at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/libvirt.c:9288 #20 0x000000000043ccfe in remoteDispatchDomainAttachDevice (args=0x7fffe0201ff0, rerr=0x7ffff1760c90, client=<optimized out>, server=<optimized out>, msg=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:320 #21 remoteDispatchDomainAttachDeviceHelper (server=<optimized out>, client=<optimized out>, msg=<optimized out>, rerr=0x7ffff1760c90, args=0x7fffe0201ff0, ret=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./daemon/remote_dispatch.h:298 #22 0x00007ffff788a866 in virNetServerProgramDispatchCall (msg=0x7fffe8093d20, client=0x7fffe8053050, server=0x76e920, prog=0x778880) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:416 #23 virNetServerProgramDispatch (prog=0x778880, server=server@entry=0x76e920, client=0x7fffe8053050, msg=0x7fffe8093d20) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserverprogram.c:289 #24 0x00007ffff78864d1 in virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x76e920) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/rpc/virnetserver.c:161 #25 0x00007ffff77c373e in virThreadPoolWorker (opaque=opaque@entry=0x7789a0) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threadpool.c:144 #26 0x00007ffff77c31c9 in virThreadHelper (data=<optimized out>) at /build/buildd-libvirt_0.9.12-4-amd64-KyxbcZ/libvirt-0.9.12/./src/util/threads-pthread.c:161 #27 0x00007ffff4d27b50 in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0 #28 0x00007ffff466b6dd in clone () from /lib/x86_64-linux-gnu/libc.so.6 #29 0x0000000000000000 in ?? () (gdb) f 7
This backtrace is identical with that attached to the bug.
The bug is fixed by commit: commit 0f4660c8787cc41fe67f869984c0ae11d680037e Author: Peter Krempa <pkrempa@redhat.com> Date: Thu Jun 14 10:29:36 2012 +0200
qemu: Fix off-by-one error while unescaping monitor strings
While unescaping the commands the commands passed through to the monitor function qemuMonitorUnescapeArg() initialized lenght of the input string to strlen()+1 which is fine for alloc but not for iteration of the string.
That is included in the 0.9.13 release. To fix this issue please upgrade or propose to backport that patch into your distro. At any rate thanks for the exhausting bug report, it definitely helped identifying the issue and would be useful in fixing it.
Peter
participants (2)
-
Peter Krempa -
Wangpan