[Libvir] crash on domain shutdown (via ruby)
I am observing a crash with libvirt ruby bindings. Please see stack trace below. I am using libvirt-0.4.2 and ruby-libvirt-0.0.6. I have upgraded to new versions of both today, so I am not sure which one is to blame. The problem happening when I try to do domain shutdown. I can reproduce the problem, so if somebody will be looking into that, I can aid debugging it. Sincerely, Vadim *** glibc detected *** ruby: free(): invalid pointer: 0xb7e5e054 *** ======= Backtrace: ========= /lib/i686/nosegneg/libc.so.6[0x5166a6] /lib/i686/nosegneg/libc.so.6(cfree+0x90)[0x519c10] /usr/lib/libvirt.so.0(virResetError+0x2e)[0x421f3de] /usr/lib/ruby/site_ruby/1.8/i386-linux/_libvirt.so[0x11ca3e] /usr/lib/ruby/site_ruby/1.8/i386-linux/_libvirt.so(libvirt_dom_shutdown +0x5e)[0x11f9fe] /usr/lib/libruby.so.1.8[0xb45db5] /usr/lib/libruby.so.1.8[0xb4d49b] /usr/lib/libruby.so.1.8[0xb4e09d] /usr/lib/libruby.so.1.8[0xb45dc8] /usr/lib/libruby.so.1.8[0xb4d49b] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55e76] /usr/lib/libruby.so.1.8[0xb5695e] /usr/lib/libruby.so.1.8[0xb56c1c] /usr/lib/libruby.so.1.8[0xb4dd74] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55f92] /usr/lib/libruby.so.1.8[0xb58a88] /usr/lib/libruby.so.1.8[0xb59658] /usr/lib/libruby.so.1.8[0xb459c7] /usr/lib/libruby.so.1.8[0xb4d49b] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55e76] /usr/lib/libruby.so.1.8[0xb5433a] /usr/lib/libruby.so.1.8[0xb5695e] /usr/lib/libruby.so.1.8[0xb58a88] /usr/lib/libruby.so.1.8[0xb59658] /usr/lib/libruby.so.1.8[0xb459c7] /usr/lib/libruby.so.1.8[0xb4d49b] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55e76] /usr/lib/libruby.so.1.8[0xb55bd0] /usr/lib/libruby.so.1.8[0xb58a88] /usr/lib/libruby.so.1.8(rb_yield+0x21)[0xb59eb1] /usr/lib/libruby.so.1.8(rb_ary_each+0x31)[0xb33031] /usr/lib/libruby.so.1.8[0xb45db5] /usr/lib/libruby.so.1.8[0xb4d49b] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb5655d] /usr/lib/libruby.so.1.8[0xb4dd74] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55f92] /usr/lib/libruby.so.1.8[0xb54b8c] /usr/lib/libruby.so.1.8[0xb5695e] /usr/lib/libruby.so.1.8[0xb4dd74] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55f92] /usr/lib/libruby.so.1.8[0xb551c0] /usr/lib/libruby.so.1.8[0xb567c0] /usr/lib/libruby.so.1.8[0xb4dd74] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55f92] /usr/lib/libruby.so.1.8[0xb4dd74] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55f92] /usr/lib/libruby.so.1.8[0xb5467a] /usr/lib/libruby.so.1.8[0xb4dd74] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8(rb_call_super+0xa2)[0xb4f152] /usr/lib/libruby.so.1.8[0xb54b37] /usr/lib/libruby.so.1.8[0xb4dd74] /usr/lib/libruby.so.1.8[0xb4e1b8] /usr/lib/libruby.so.1.8[0xb55e76] ======= Memory map: ======== 00110000-00111000 r-xp 00000000 08:01 6095946 /usr/lib/ruby/1.8/ i386-linux/fcntl.so 00111000-00112000 rw-p 00000000 08:01 6095946 /usr/lib/ruby/1.8/ i386-linux/fcntl.so 00112000-00114000 r-xp 00000000 08:01 6095942 /usr/lib/ruby/1.8/ i386-linux/digest.so 00114000-00115000 rw-p 00001000 08:01 6095942 /usr/lib/ruby/1.8/ i386-linux/digest.so 00116000-00117000 r-xp 00116000 00:00 0 [vdso] 00117000-00123000 r-xp 00000000 08:01 6193153 /usr/lib/ruby/ site_ruby/1.8/i386-linux/_libvirt.so 00123000-00124000 rw-p 0000c000 08:01 6193153 /usr/lib/ruby/ site_ruby/1.8/i386-linux/_libvirt.so 00124000-00250000 r-xp 00000000 08:01 5618582 /usr/lib/libxml2.so. 2.6.26 00250000-00256000 rw-p 0012b000 08:01 5618582 /usr/lib/libxml2.so. 2.6.26 00256000-0025f000 r-xp 00000000 08:01 6095962 /usr/lib/ruby/1.8/ i386-linux/zlib.so 0025f000-00260000 rw-p 00008000 08:01 6095962 /usr/lib/ruby/1.8/ i386-linux/zlib.so 00260000-00264000 r-xp 00000000 08:01 6193994 /usr/lib/ruby/1.8/ i386-linux/digest/sha2.so 00264000-00265000 rw-p 00003000 08:01 6193994 /usr/lib/ruby/1.8/ i386-linux/digest/sha2.so 00265000-0028a000 r-xp 00000000 08:01 5614351 /usr/lib/ libk5crypto.so.3.1 0028a000-0028b000 rw-p 00025000 08:01 5614351 /usr/lib/ libk5crypto.so.3.1 0028b000-00296000 r-xp 00000000 08:01 14915360 /lib/ libgcc_s-4.1.2-20070626.so.1 00296000-00297000 rw-p 0000a000 08:01 14915360 /lib/ libgcc_s-4.1.2-20070626.so.1 0029b000-002a6000 r-xp 00000000 08:01 6095957 /usr/lib/ruby/1.8/ i386-linux/socket.so 002a6000-002a7000 rw-p 0000a000 08:01 6095957 /usr/lib/ruby/1.8/ i386-linux/socket.so 002bb000-00332000 r-xp 00000000 08:01 5609971 /usr/lib/libgnutls.so. 13.0.6 00332000-00338000 rw-p 00076000 08:01 5609971 /usr/lib/libgnutls.so. 13.0.6 00338000-0038b000 r-xp 00000000 08:01 5609906 /usr/lib/libgcrypt.so. 11.2.2 0038b000-0038d000 rw-p 00053000 08:01 5609906 /usr/lib/libgcrypt.so. 11.2.2 0048d000-004a6000 r-xp 00000000 08:01 14909450 /lib/ld-2.5.so 004a6000-004a7000 r--p 00019000 08:01 14909450 /lib/ld-2.5.so 004a7000-004a8000 rw-p 0001a000 08:01 14909450 /lib/ld-2.5.so 004aa000-004ac000 r-xp 00000000 08:01 14915357 /lib/libcom_err.so.2.1 004ac000-004ad000 rw-p 00001000 08:01 14915357 /lib/libcom_err.so.2.1 004af000-005ec000 r-xp 00000000 08:01 14909452 /lib/i686/nosegneg/ libc-2.5.so 005ec000-005ee000 r--p 0013d000 08:01 14909452 /lib/i686/nosegneg/ libc-2.5.so 005ee000-005ef000 rw-p 0013f000 08:01 14909452 /lib/i686/nosegneg/ libc-2.5.so 005ef000-005f2000 rw-p 005ef000 00:00 0 005f4000-005f6000 r-xp 00000000 08:01 14909481 /lib/libdl-2.5.so 005f6000-005f7000 r--p 00001000 08:01 14909481 /lib/libdl-2.5.so 005f7000-005f8000 rw-p 00002000 08:01 14909481 /lib/libdl-2.5.so 005fa000-0060d000 r-xp 00000000 08:01 14909458 /lib/i686/nosegneg/ libpthread-2.5.so 0060d000-0060e000 r--p 00012000 08:01 14909458 /lib/i686/nosegneg/ libpthread-2.5.so 0060e000-0060f000 rw-p 00013000 08:01 14909458 /lib/i686/nosegneg/ libpthread-2.5.so 0060f000-00611000 rw-p 0060f000 00:00 0 00613000-00638000 r-xp 00000000 08:01 14909471 /lib/i686/nosegneg/ libm-2.5.so 00638000-00639000 r--p 00024000 08:01 14909471 /lib/i686/nosegneg/ libm-2.5.so 00639000-0063a000 rw-p 00025000 08:01 14909471 /lib/i686/nosegneg/ libm-2.5.so 0063c000-00651000 r-xp 00000000 08:01 14915356 /lib/libselinux.so.1 00651000-00653000 rw-p 00015000 08:01 14915356 /lib/libselinux.so.1 00655000-00690000 r-xp 00000000 08:01 14915355 /lib/libsepol.so.1 00690000-00691000 rw-p 0003a000 08:01 14915355 /lib/libsepol.so.1 00691000-0069b000 rw-p 00691000 00:00 0 0069d000-006af000 r-xp 00000000 08:01 5609961 /usr/lib/libz.so.1.2.3 006af000-006b0000 rw-p 00011000 08:01 5609961 /usr/lib/libz.so.1.2.3 006b2000-006b6000 r-xp 00000000 08:01 5614350 /usr/lib/ libxenstore.so.3.0.0 006b6000-006b7000 rw-p 00003000 08:01 5614350 /usr/lib/ libxenstore.so.3.0.0 006b7000-006ba000 rw-p 006b7000 00:00 0 006bd000-006d0000 r-xp 00000000 08:01 14909475 /lib/libnsl-2.5.so 006d0000-006d1000 r--p 00012000 08:01 14909475 /lib/libnsl-2.5.so 006d1000-006d2000 rw-p 00013000 08:01 14909475 /lib/libnsl-2.5.so 006d2000-006d4000 rw-p 006d2000 00:00 0 006d6000-006e5000 r-xp 00000000 08:01 14909651 /lib/libresolv-2.5.so 006e5000-006e6000 r--p 0000e000 08:01 14909651 /lib/libresolv-2.5.so 006e6000-006e7000 rw-p 0000f000 08:01 14909651 /lib/libresolv-2.5.so 006e7000-006e9000 rw-p 006e7000 00:00 0 006ec000-006f0000 r-xp 00000000 08:01 6095958 /usr/lib/ruby/1.8/ i386-linux/stringio.so 006f0000-006f1000 rw-p 00003000 08:01 6095958 /usr/lib/ruby/1.8/ i386-linux/stringio.so 006f9000-00711000 r-xp 00000000 08:01 5621763 /usr/lib/libsasl2.so. 2.0.22 00711000-00712000 rw-p 00017000 08:01 5621763 /usr/lib/libsasl2.so. 2.0.22 007e6000-00821000 r-xp 00000000 08:01 6095951 /usr/lib/ruby/1.8/ i386-linux/openssl.so 00821000-00823000 rw-p 0003a000 08:01 6095951 /usr/lib/ruby/1.8/ i386-linux/openssl.so 008c0000-008c5000 r-xp 00000000 08:01 14909479 /lib/libcrypt-2.5.so 008c5000-008c6000 r--p 00004000 08:01 14909479 /lib/libcrypt-2.5.so 008c6000-008c7000 rw-p 00005000 08:01 14909479 /lib/libcrypt-2.5.so 008c7000-008ee000 rw-p 008c7000 00:00 0 008f0000-0091d000 r-xp 00000000 08:01 5614349 /usr/lib/ libgssapi_krb5.so.2.2 0091d000-0091e000 rw-p 0002d000 08:01 5614349 /usr/lib/ libgssapi_krb5.so.2.2 00937000-0093b000 r-xp 00000000 08:01 6095959 /usr/lib/ruby/1.8/ i386-linux/strscan.so 0093b000-0093c000 rw-p 00003000 08:01 6095959 /usr/lib/ruby/1.8/ i386-linux/strscan.so 00948000-00950000 r-xp 00000000 08:01 5622056 /usr/lib/ libkrb5support.so.0.1 00950000-00951000 rw-p 00007000 08:01 5622056 /usr/lib/ libkrb5support.so.0.1 00985000-00987000 r-xp 00000000 08:01 14909636 /lib/libkeyutils-1.2.so 00987000-00988000 rw-p 00001000 08:01 14909636 /lib/libkeyutils-1.2.so 00b21000-00bfd000 r-xp 00000000 08:01 5623120 /usr/lib/libruby.so. 1.8.5 00bfd000-00c01000 rw-p 000db000 08:01 5623120 /usr/lib/libruby.so. 1.8.5 00c01000-00c10000 rw-p 00c01000 00:00 0 00c22000-00c3d000 r-xp 00000000 08:01 6095960 /usr/lib/ruby/1.8/ i386-linux/syck.so 00c3d000-00c3e000 rw-p 0001b000 08:01 6095960 /usr/lib/ruby/1.8/ i386-linux/syck.so 00c91000-00c94000 r-xp 00000000 08:01 5609879 /usr/lib/libgpg- error.so.0.3.0 00c94000-00c95000 rw-p 00002000 08:01 5609879 /usr/lib/libgpg- error.so.0.3.0 00cda000-00cdc000 r-xp 00000000 08:01 6095945 /usr/lib/ruby/1.8/ i386-linux/etc.so 00cdc000-00cdd000 rw-p 00001000 08:01 6095945 /usr/lib/ruby/1.8/ i386-linux/etc.so 00d2f000-00d31000 r-xp 00000000 08:01 6193294 /usr/lib/ruby/1.8/ i386-linux/digest/md5.so 00d31000-00d32000 rw-p 00001000 08:01 6193294 /usr/lib/ruby/1.8/ i386-linux/digest/md5.so 00def000-00df3000 r-xp 00000000 08:01 6193993 /usr/lib/ruby/1.8/ i386-linux/digest/sha1.so 00df3000-00df4000 rw-p 00003000 08:01 6193993 /usr/lib/ruby/1.8/ i386-linux/digest/sha1.so 041f2000-04275000 r-xp 00000000 08:01 5618887 /usr/lib/libvirt.so. 0.4.2 04275000-04277000 rw-p 00083000 08:01 5618887 /usr/lib/libvirt.so. 0.4.2 04277000-04279000 rw-p 04277000 00:00 0 048f5000-04a12000 r-xp 00000000 08:01 14909468 /lib/libcrypto.so. 0.9.8b 04a12000-04a25000 rw-p 0011c000 08:01 14909468 /lib/libcrypto.so. 0.9.8b 04a25000-04a28000 rw-p 04a25000 00:00 0 04a9a000-04b2a000 r-xp 00000000 08:01 5614339 /usr/lib/libkrb5.so.3.3 04b2a000-04b2d000 rw-p 0008f000 08:01 5614339 /usr/lib/libkrb5.so.3.3 04b2f000-04b70000 r-xp 00000000 08:01 14909496 /lib/libssl.so.0.9.8b 04b70000-04b74000 rw-p 00040000 08:01 14909496 /lib/libssl.so.0.9.8b 08048000-08049000 r-xp 00000000 08:01 5623121 /usr/bin/ruby 08049000-0804a000 rw-p 00000000 08:01 5623121 /usr/bin/ruby 08279000-0860e000 rw-p 08279000 00:00 0 b7000000-b7021000 rw-p b7000000 00:00 0 b7021000-b7100000 ---p b7021000 00:00 0 b7147000-b7348000 rw-p b7147000 00:00 0 b7348000-b7349000 ---p b7348000 00:00 0 b7349000-b7e66000 rw-p b7349000 00:00 0 b7e67000-b7f06000 rw-p b7e67000 00:00 0 b7f06000-b7f3b000 r--s 00000000 08:01 10750807 /var/db/nscd/hosts b7f3b000-b7fc6000 rw-p b7f3b000 00:00 0 b7fcc000-b7fd2000 rw-p b7fcc000 00:00 0 bf864000-bf8b6000 rw-p bf864000 00:00 0 [stack] Aborted -- "La perfection est atteinte non quand il ne reste rien a ajouter, mais quand il ne reste rien a enlever." (Antoine de Saint-Exupery)
On Apr 10, 2008, at 15:00, Vadim Zaliva wrote:
I am observing a crash with libvirt ruby bindings. Please see stack trace below. I am using libvirt-0.4.2 and ruby-libvirt-0.0.6. I have upgraded to new versions of both today, so I am not sure which one is to blame. The problem happening when I try to do domain shutdown.
I can reproduce the problem, so if somebody will be looking into that, I can aid debugging it.
Here is my test code: require 'libvirt' conn = Libvirt::open("") dom = conn.lookup_domain_by_name("eee") if dom == nil puts "not found" else dom.shutdown() puts "complete" end $ sudo ruby crash2.rb *** glibc detected *** ruby: free(): invalid pointer: 0xb7f2dcb4 ** Vadim -- "La perfection est atteinte non quand il ne reste rien a ajouter, mais quand il ne reste rien a enlever." (Antoine de Saint-Exupery)
Vadim Zaliva wrote:
On Apr 10, 2008, at 15:00, Vadim Zaliva wrote:
I am observing a crash with libvirt ruby bindings. Please see stack trace below. I am using libvirt-0.4.2 and ruby-libvirt-0.0.6. I have upgraded to new versions of both today, so I am not sure which one is to blame. The problem happening when I try to do domain shutdown.
I can reproduce the problem, so if somebody will be looking into that, I can aid debugging it.
OK, interesting. I took your test code and tried it out here, and I wasn't able to reproduce. I'm running libvirt-0.4.1-7 (recompiled from F-9) and ruby-libvirt-0.0.6, along with a 2.6.24 Fedora 8 kernel and KVM guests. Besides your libvirt and ruby-libvirt package versions, what else can you tell us about your environment? What kind of guests are you trying to shutdown (KVM, Xen, Qemu, etc.), and what relevant package versions are you using? Chris Lalancette
Hi Vadim, seems you're having the honor of testing the new improved error reporting in ruby-libvirt. The way virResetError was used was incorrect: it can only be called if virConnCopyLastError returned a positive result. For some reason, in your example virDomainShutdown fails, but the error status on the connection seems to still be VIR_ERR_OK. I just committed a change to ruby-libvirt to address that; can you give that a try ? The virResetError docs are not clear on when it can be called, and when it must not be called. It seems like a useless optimization that the memcpy of the error only happens when there was an error - the virConnCopyLastError/virCopyLastError routines also need to strdup the various strings in a virError, otherwise it is possible that the same string gets freed multiple times when there are multiple matching copy/reset calls. David
On Apr 10, 2008, at 22:16, David Lutterkort wrote:
seems you're having the honor of testing the new improved error reporting in ruby-libvirt. ... I just committed a change to ruby-libvirt to address that; can you give that a try ?
David, I will give it a try tomorrow morning and will report back. Vadim -- "La perfection est atteinte non quand il ne reste rien a ajouter, mais quand il ne reste rien a enlever." (Antoine de Saint-Exupery)
On Fri, Apr 11, 2008 at 05:16:03AM +0000, David Lutterkort wrote:
The way virResetError was used was incorrect: it can only be called if virConnCopyLastError returned a positive result. For some reason, in your example virDomainShutdown fails, but the error status on the connection seems to still be VIR_ERR_OK.
You have to be careful because some libvirt calls return error indications without first setting up any virterror. These cases are bugs in libvirt of course, but wearisome to find. (Well, I suggested a method using static analysis to find them, but don't have the time to implement it right now). In any case you have to be prepared for this situation. The OCaml bindings turn these cases into a "fake" VIR_ERR_INTERNAL_ERROR / VIR_FROM_NONE / VIR_ERR_ERROR. http://hg.et.redhat.com/applications/virt/applications/virt-top--devel?f=6fe... (line 68) Rich. -- Richard Jones, Emerging Technologies, Red Hat http://et.redhat.com/~rjones virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://et.redhat.com/~rjones/virt-top
On Apr 10, 2008, at 22:16, David Lutterkort wrote:
I just committed a change to ruby-libvirt to address that; can you give that a try ?
After applying your latest patch I could not reproduce the problem. Thanks! Vadim -- "La perfection est atteinte non quand il ne reste rien a ajouter, mais quand il ne reste rien a enlever." (Antoine de Saint-Exupery)
On Apr 11, 2008, at 15:02, Vadim Zaliva wrote:
I just committed a change to ruby-libvirt to address that; can you give that a try ?
After applying your latest patch I could not reproduce the problem. Thanks!
I was too quick to speak. I got another crash: *** glibc detected *** ruby: double free or corruption (out): 0x08e78608 *** ======= Backtrace: ========= /lib/i686/nosegneg/libc.so.6[0x5166a6] /lib/i686/nosegneg/libc.so.6(cfree+0x90)[0x519c10] /usr/lib/libvirt.so.0(virResetError+0x44)[0x64af3f4] /usr/lib/libvirt.so.0(virConnResetLastError+0x24)[0x64af634] /usr/lib/libvirt.so.0[0x649f618] /usr/lib/libvirt.so.0(virDomainLookupByUUID+0xb7)[0x648e7d7] /usr/lib/libvirt.so.0(virDomainLookupByUUIDString+0x14d)[0x648f90d] /usr/lib/ruby/site_ruby/1.8/i386-linux/ _libvirt.so(libvirt_conn_lookup_domain_by_uuid+0x3d)[0x15f8cd] Vadim -- "La perfection est atteinte non quand il ne reste rien a ajouter, mais quand il ne reste rien a enlever." (Antoine de Saint-Exupery)
On Fri, 2008-04-11 at 15:32 -0700, Vadim Zaliva wrote:
I was too quick to speak. I got another crash:
*** glibc detected *** ruby: double free or corruption (out): 0x08e78608 *** ======= Backtrace: ========= /lib/i686/nosegneg/libc.so.6[0x5166a6] /lib/i686/nosegneg/libc.so.6(cfree+0x90)[0x519c10] /usr/lib/libvirt.so.0(virResetError+0x44)[0x64af3f4] /usr/lib/libvirt.so.0(virConnResetLastError+0x24)[0x64af634] /usr/lib/libvirt.so.0[0x649f618] /usr/lib/libvirt.so.0(virDomainLookupByUUID+0xb7)[0x648e7d7] /usr/lib/libvirt.so.0(virDomainLookupByUUIDString+0x14d)[0x648f90d] /usr/lib/ruby/site_ruby/1.8/i386-linux/ _libvirt.so(libvirt_conn_lookup_domain_by_uuid+0x3d)[0x15f8cd]
Did your program while it ran have any errors before ? This is somewhere in the guts of libvirt, and I would guess it's a double free caused by the ruby bindings first calling virResetError on an unrelated error, and then libvirt crashing on a second error since the first virResetError freed various strings it didn't really own. Try commenting the call to virResetError in _libvirt.c in the ruby bindings out entirely, and see if you still get a crash. David
On Apr 11, 2008, at 16:55, David Lutterkort wrote:
Try commenting the call to virResetError in _libvirt.c in the ruby bindings out entirely, and see if you still get a crash.
Yes, that helped. Do you plan to fix in in mercurial soon? Vadim -- "La perfection est atteinte non quand il ne reste rien a ajouter, mais quand il ne reste rien a enlever." (Antoine de Saint-Exupery)
On Mon, 2008-04-14 at 11:04 -0700, Vadim Zaliva wrote:
On Apr 11, 2008, at 16:55, David Lutterkort wrote:
Try commenting the call to virResetError in _libvirt.c in the ruby bindings out entirely, and see if you still get a crash.
Yes, that helped. Do you plan to fix in in mercurial soon?
I just committed a change so that we don't use the copy methods anymore; the ruby bindings now use the get error methods instead. The copy methods offer little advantage over the get methods. David
participants (4)
-
Chris Lalancette -
David Lutterkort -
Richard W.M. Jones -
Vadim Zaliva