virInitialize leads to segfault when unloading libvirt-cim providers.

Contact Information = Sharad Mishra/
snmishra@us.ibm.com
,  

---uname output---

Linux elm3b193.beaverton.ibm.com 2.6.18-194.el5 #1 SMP Tue Mar 16 21:52:39 EDT

2010 x86_64 x86_64 x86_64 GNU/Linux

Machine Type = IBM system x blade, machine type - 8853, model - MC1 

---Debugger---

A debugger is not configured

---Steps to Reproduce---

 1. Install libvirt and libvirt-cim

2 start libvirtd

3. start tog-pegasus

4. run wbemcli command to enum instance on KVM_ComputerSystem class. This is

just a way to load the provider to memory. All  we need to do is run

virConnectOpen() and virConnectClose() in this provider. So in this step all

that is required is for virConnectOpen and virConnectClose to run.

5. Stop tog-pegasus. 

A core is gererated.

---System Management Component Data--- 

Userspace tool common name: libvirt/libvirt-cim/tog-pegasus 

System management type: IMM 

The userspace tool has the following bit modes: 64 

Userspace rpm: libvirt-0.6.3-33.el5,  libvirt-cim-0.5.14-1 and

tog-pegasus-2.9.2-2.el5 

Userspace tool obtained from project website:  na 

*Additional Instructions for Sharad Mishra/
snmishra@us.ibm.com
, : 

-Post a private note with access information to the machine that the bug is

occuring on.

-Attach ltrace and strace of userspace application.

-Attach screen shoot of System Managment Utility

Here is the gist of investigation done so far ---

Here is the gist of his investigation -

I've looked into the libvirt source code and the reason for issues with the

unload of libvirt-cim providers is there (and cimprovider -r forces an unload,

just as cimserver shutdown does).

virConnectOpen() does call virInitialize() which then will initialize the

gcrypt library after initializes the thread setup ->virThreadInitialize(),

creates the error reporting structure ->virErrorInitialize() and initialize a

random number generator ->virRandomInitialize().

The issue is hidden in virErrorInitialize(). It calls virErrorInitialize()

which uses the wrapper function virThreadLocalInit(&virLastErr,

virLastErrFreeData);.

virThreadLocalInit() calls pthread_create() with a destructor function

(virLastErrFreeData()) which will be called on thread exits.

Trouble here is that pthread_key_delete() is never called in libvirt, but next

time the system runs through the key list and libvirt has been unloaded the

system will jump to an already unloaded function (the pointer were

virLastErrFreeData() was loaded formerly) and consequently crash the process.

Fixing this will not be easy, but a start would be a libvirt function on its

externalize interface that allows cleanup short before library unload, or

alternatively libvirt does the thread key delete on library unload.

Trace output from gdb -

/var/log/messages: 

Aug  6 11:35:58 SYX-e41f13fe5558 cimserver[21602]: OpenPegasus stopped.

Aug  6 11:35:59 SYX-e41f13fe5558 kernel: cimserver[21376]: segfault at

00002aaaab165f00 rip 00002aaaab165f00 rsp 00000000457d40b8 error 14

call stack trace:

command: gdb -batch -x /tmp/gdb.27236.cmd /usr/sbin/cimserver

/var/log/core/core.25567

time:    Sat Aug  6 11:35:59 UTC 2011

stdout/stderr:

-----------------------------------------------------------------

[New Thread 25567]

Core was generated by `/usr/sbin/cimserver'.

Program terminated with signal 11, Segmentation fault.

#0  0x00002aaaab4f6f00 in ?? ()

#0  0x00002aaaab4f6f00 in ?? ()

#1  0x00002ab4fb3c6ad9 in __nptl_deallocate_tsd () from /lib64/libpthread.so.0

#2  0x00002ab4fb3c774b in start_thread () from /lib64/libpthread.so.0

#3  0x00002ab4fc07af6d in clone () from /lib64/libc.so.6

-----------------------------------------------------------------

 exit_code=0

-----------------------------------------------------------------
libvirt-cim-bounces@redhat.com wrote on 08/30/2011 01:43:06 AM:

> Daniel Veillard <veillard@redhat.com> 
> Sent by: libvirt-cim-bounces@redhat.com

> 
> 08/30/11 01:43 AM
> 

> Please respond to

> veillard@redhat.com; Please respond to

> List for discussion and development of libvirt CIM <libvirt-cim@redhat.com>
> 

> To
> 

> List for discussion and development of libvirt CIM <libvirt-cim@redhat.com>
> 

> cc
> 

> Subject
> 

> Re: [Libvirt-cim] [PATCH] DO NOT PUSH UPSTREAM
> 

> On Fri, Aug 26, 2011 at 05:38:29PM -0000, Sharad Mishra wrote:

> > # HG changeset patch

> > # User Sharad Mishra <snmishra@us.ibm.com>

> > # Date 1314380301 25200

> > # Node ID 73e7bb11410f3c10d3b0b61678531314262f09b7

> > # Parent  2984ede9c082bab48b3494ef4b9f5561fadef2ad

> > DO NOT PUSH UPSTREAM

> > 

> > This is the workaround that was given to zGryphon

> > team to prevent unloading of idle providers.

> > There is a bug in libvirt that needs to be fixed.

> > That fix will be the actual fix for this problem

> > that cause segfault when those providers are unloaded

> > that make a call to virConnectOpen.

> 

>   I'm curious about the libvirt bug, which one is it ?

> 

> Daniel

> 

> -- 

> Daniel Veillard      | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/

> daniel@veillard.com  | Rpmfind RPM search engine http://rpmfind.net/

> http://veillard.com/ | virtualization library  http://libvirt.org/

> 

> _______________________________________________

> Libvirt-cim mailing list

> Libvirt-cim@redhat.com

> https://www.redhat.com/mailman/listinfo/libvirt-cim