On Wed, Jul 04, 2012 at 10:24:38AM +0200, Wido den Hollander wrote:
On 03-07-12 17:21, Daniel P. Berrange wrote:
On Tue, Jul 03, 2012 at 05:11:48PM +0200, Wido den Hollander wrote:
Yes, there is memory corruption somewhere. I never used valgrind before, but the output seems to show.
I ran libvirtd inside a screen, I've attached the screenlog with all the output.
At the end you'll see there is a misread. I was storing the base64 encoded value of "wido".
Thanks so there are two errors shown by valgrind
==6825== Invalid read of size 1 ==6825== at 0x5769DBA: vfprintf (vfprintf.c:1624) ==6825== by 0x58289D0: __vasprintf_chk (vasprintf_chk.c:68) ==6825== by 0x509C727: virVasprintf (stdio2.h:199) ==6825== by 0x508CFEA: virLogVMessage (logging.c:749) ==6825== by 0x508D349: virLogMessage (logging.c:696) ==6825== by 0x127AB0BC: secretSaveValue (secret_driver.c:297) ==6825== by 0x127AB30A: secretSetValue (secret_driver.c:861) ==6825== by 0x5130CF4: virSecretSetValue (libvirt.c:14457) ==6825== by 0x41CC8F: remoteDispatchSecretSetValueHelper (remote_dispatch.h:10962) ==6825== by 0x5174F84: virNetServerProgramDispatch (virnetserverprogram.c:416) ==6825== by 0x5171260: virNetServerHandleJob (virnetserver.c:161) ==6825== by 0x50990AD: virThreadPoolWorker (threadpool.c:143) ==6825== Address 0x19cb3c64 is 0 bytes after a block of size 4 alloc'd ==6825== at 0x4C29DB4: calloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==6825== by 0x508E32F: virAllocN (memory.c:128) ==6825== by 0x127AB21B: secretSetValue (secret_driver.c:838) ==6825== by 0x5130CF4: virSecretSetValue (libvirt.c:14457) ==6825== by 0x41CC8F: remoteDispatchSecretSetValueHelper (remote_dispatch.h:10962) ==6825== by 0x5174F84: virNetServerProgramDispatch (virnetserverprogram.c:416) ==6825== by 0x5171260: virNetServerHandleJob (virnetserver.c:161) ==6825== by 0x50990AD: virThreadPoolWorker (threadpool.c:143) ==6825== by 0x5098755: virThreadHelper (threads-pthread.c:161) ==6825== by 0x550AE99: start_thread (pthread_create.c:308) ==6825== by 0x58124BC: clone (clone.S:112)
This one is harmless - this is because the VIR_DEBUG line you added uses '%s' to print secret_value, but this is not a NULL terminated string. So we can ignore this.
==6825== ==6825== Invalid read of size 4 ==6825== at 0xA57E4B9: base64_encode (in /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0) ==6825== by 0x10DDBC98: base64_encode_alloc (base64.c:140) ==6825== by 0x127AB0EF: secretSaveValue (secret_driver.c:302) ==6825== by 0x127AB30A: secretSetValue (secret_driver.c:861) ==6825== by 0x5130CF4: virSecretSetValue (libvirt.c:14457) ==6825== by 0x41CC8F: remoteDispatchSecretSetValueHelper (remote_dispatch.h:10962) ==6825== by 0x5174F84: virNetServerProgramDispatch (virnetserverprogram.c:416) ==6825== by 0x5171260: virNetServerHandleJob (virnetserver.c:161) ==6825== by 0x50990AD: virThreadPoolWorker (threadpool.c:143) ==6825== by 0x5098755: virThreadHelper (threads-pthread.c:161) ==6825== by 0x550AE99: start_thread (pthread_create.c:308) ==6825== by 0x58124BC: clone (clone.S:112) ==6825== Address 0xb7786c8 is 8 bytes inside a block of size 9 alloc'd ==6825== at 0x4C2B6CD: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==6825== by 0xA57E3D7: base64_encode (in /usr/lib/x86_64-linux-gnu/libroken.so.18.1.0) ==6825== by 0x10DDBC98: base64_encode_alloc (base64.c:140) ==6825== by 0x127AB0EF: secretSaveValue (secret_driver.c:302) ==6825== by 0x127AB30A: secretSetValue (secret_driver.c:861) ==6825== by 0x5130CF4: virSecretSetValue (libvirt.c:14457) ==6825== by 0x41CC8F: remoteDispatchSecretSetValueHelper (remote_dispatch.h:10962) ==6825== by 0x5174F84: virNetServerProgramDispatch (virnetserverprogram.c:416) ==6825== by 0x5171260: virNetServerHandleJob (virnetserver.c:161) ==6825== by 0x50990AD: virThreadPoolWorker (threadpool.c:143)
This one is very interesting. It shows that the 'base64_encode' function is doing an out-of-bounds read. More tellingly though is that it is reporting 'base64_encode' function is in a wierd library:
/usr/lib/x86_64-linux-gnu/libroken.so.18.1.
If this were normal, we should expect to see that function present in 'base64.c' since this function code is provided by gnulib itself.
So something else libvirt is linking to, directly or indirectly is using libroken.so which also has a 'base64_encode'symbol defined. This is overriding gnulib's symbol of the same name.
I'm willing to bet the API contract of this libroken.so base64_encode. differs from GNULIBS, with crashtastic results
Can you try and find how this libroken.so is getting linked to libvirt ?
Or indeed what this library does, or what package it is part of.
The library is libroken18-heimdal under Ubuntu 12.04: http://packages.ubuntu.com/precise/libroken18-heimdal
When installing ubuntu-virt-server libraries like gnutls depend on this library.
I'm not sure why libvirt gets linked to libroken, but on Ubuntu systems it's installed on most system which use libvirt.
I haven't found a header file which as the symbol 'base64_encode' in it for libroken, only gnutls and glib seem to have something that has 'base64_encode' in it, but those are prefixed with g_
I expect that this is an internal symbol from libroken.so which they leak into the public namespace.
For now I have no clues to why it gets linked to libroken, but it seems the problem has been found.
If gnutls links to libroken.so, then libvirt gets linked to it indirectly It sounds like we might need to have a workaround in gnulib to avoid this problem. With other cases where gnulib replaces existing symbols they use some magic such that the gnulib replacement gets prefixed with 'rpl_'. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|