[libvirt-users] Libvirtd dead, pid still exists. ( Problem might be with TLS interface of libvirtd )
Dear All, Please find few minutes from your time and guide us with some pointers if possible. We are facing a libvirtd crash when we are trying to connect to qemu by default TLS transport. # virsh -c qemu+tls://localhost/system version error: authentication failed: TLS handshake failed A TLS packet with unexpected length was received. error: failed to connect to the hypervisor I used my own CA and certificates (generated using http://libvirt.org/remote.html#Remote_libvirtd_configuration on Redhat PC) on both Kontron PC and our Board (with patched libvirt). Libvirtd.conf was modified so that libvirt is listening all IPs using default IP (so that it was possible to use same certificates on all machines) These directories and files created and used. /etc/pki/CA/cacert.pem /etc/pki/libvirt/private/serverkey.pem /etc/pki/libvirt/servercert.pem /etc/pki/libvirt/private/clientkey.pem /etc/pki/libvirt/clientcert.pem TLS connection worked fine with Kontron PC # virsh -c qemu+tls://localhost/system version Compiled against library: libvir 0.9.5 Using library: libvir 0.9.5 Using API: QEMU 0.9.5 Running hypervisor: QEMU 0.12.1 But libvirt crashed on our Board (using libvirt 0.10.2, gnutls-2.10.5-1_WR4.3.x86_64 and libudev-161-4 rpms ) # virsh -c qemu+tls://localhost/system version error: authentication failed: TLS handshake failed A TLS packet with unexpected length was received. error: failed to connect to the hypervisor GDB: Program received signal SIGABRT, Aborted. 0x00007f8591246005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f8591246005 in raise () from /lib64/libc.so.6 #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6 #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11 #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11 #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11 #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11 #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11 #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11 #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11 #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11 #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11 #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11 #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11 #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11 #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26 #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f8592cb7437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0 Please let us know if it is a known problem. If not, I will raise a new bug for the same. ( Atleast I coulnt find the the known issues ) Thanks and Regards, Shree Duth Awasthi.
On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
GDB:
Program received signal SIGABRT, Aborted. 0x00007f8591246005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f8591246005 in raise () from /lib64/libc.so.6 #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6 #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11 #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11 #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11 #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11 #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11 #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11 #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11 #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11 #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11 #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11 #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11 #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11 #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26 #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f8592cb7437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0
The backstrace shows problem lies in libgcrypt library. So unless libvirt is overwriting some random memory areas, it's a libgcrypt's bug. Michal
Hi Michal, Thanks for your time. Yes it seems so. Adding one more info : The issue is not reproducable on our board with libvirt-0.9.4-rc0_WR4.3.x86_64, gnutls-utils-1.4.1-2_WR4.3.x86_64, libgcrypt-1.4.0-3_WR4.3.x86_64. We are unable to debug more as GDB symbols are not completely resolved. So, can you please give us some pointers to debug the issue further. Thanks and Regards, Shree Duth Awasthi. On Fri, Apr 5, 2013 at 9:34 AM, Michal Privoznik <mprivozn@redhat.com>wrote:
On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
GDB:
Program received signal SIGABRT, Aborted. 0x00007f8591246005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f8591246005 in raise () from /lib64/libc.so.6 #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6 #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11 #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11 #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11 #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11 #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11 #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11 #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11 #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11 #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11 #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11 #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11 #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11 #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26 #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f8592cb7437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0
The backstrace shows problem lies in libgcrypt library. So unless libvirt is overwriting some random memory areas, it's a libgcrypt's bug.
Michal
Dear All, A Gentle Reminder !!! Any suggestions please on how to debug further Gist : libvirt crash when trying to inquiry libvirt version using curl with TLS ( virsh -c qemu+tls://localhost/system version ) Your inputs would be of great help ! Thanks and Regards, Shree Duth Awasthi. On Fri, Apr 5, 2013 at 2:41 PM, SHREE DUTH AWASTHI < shreeduth.awasthi@gmail.com> wrote:
Hi Michal,
Thanks for your time.
Yes it seems so. Adding one more info :
The issue is not reproducable on our board with libvirt-0.9.4-rc0_WR4.3.x86_64, gnutls-utils-1.4.1-2_WR4.3.x86_64, libgcrypt-1.4.0-3_WR4.3.x86_64.
We are unable to debug more as GDB symbols are not completely resolved.
So, can you please give us some pointers to debug the issue further.
Thanks and Regards, Shree Duth Awasthi. On Fri, Apr 5, 2013 at 9:34 AM, Michal Privoznik <mprivozn@redhat.com>wrote:
On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
GDB:
Program received signal SIGABRT, Aborted. 0x00007f8591246005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f8591246005 in raise () from /lib64/libc.so.6 #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6 #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11 #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11 #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11 #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11 #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11 #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11 #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11 #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11 #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11 #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11 #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11 #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11 #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26 #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f8592cb7437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0
The backstrace shows problem lies in libgcrypt library. So unless libvirt is overwriting some random memory areas, it's a libgcrypt's bug.
Michal
Hi Michal, Open source developers of libgcrypt are pointing out to be a problem with libvirt. It seems that virsh does not make proper use of libgcrypt or gnutls. In fact, Libgcrypt is telling us what has been done wrong. Please find the latest GDB and let us know comments for the same. GDB: Breakpoint 3, 0x00007f555bb07410 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 (gdb) c Continuing. Program received signal SIGABRT, Aborted. 0x00007f555a096005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f555a096005 in raise () from /lib64/libc.so.6 #1 0x00007f555a098e40 in abort () from /lib64/libc.so.6 #2 0x00007f555b87fdc5 in _gcry_logv (level=50, fmt=0x7f555b8c6170 "operation is not possible without initialized secure memory\n", arg_ptr=0x7fff546e1130) at misc.c:136 #3 0x00007f555b8803d5 in _gcry_log_bug (fmt=0x48e0 <Address 0x48e0 out of bounds>) at misc.c:220 #4 0x00007f555b885697 in _gcry_secmem_malloc_internal (size=<value optimized out>) at secmem.c:497 #5 0x00007f555b88579c in _gcry_secmem_malloc (size=136) at secmem.c:522 #6 0x00007f555b880a65 in do_malloc (n=18656, flags=<value optimized out>, mem=0x7fff546e1290) at global.c:553 #7 0x00007f555b880aa9 in _gcry_malloc_secure (n=18656) at global.c:592 #8 0x00007f555b880b19 in _gcry_xmalloc_secure (n=136) at global.c:746 #9 0x00007f555b8c35df in _gcry_mpi_alloc_limb_space (nlimbs=17, secure=18656) at mpiutil.c:92 #10 0x00007f555b8c365f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75 #11 0x00007f555b8b025a in secret (output=0x17cfa20, input=0x17d0480, skey=0x6) at rsa.c:365 #12 0x00007f555b8b045a in _gcry_rsa_sign (algo=<value optimized out>, resarr=0x17d0660, data=0x17d0480, skey=<value optimized out>) at rsa.c:608 #13 0x00007f555b88c1ef in pubkey_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:692 #14 _gcry_pk_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:1807 ---Type <return> to continue, or q <return> to quit--- #15 0x00007f555bb29d8c in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f555bb15e7a in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f555bb1ddd6 in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f555bb1e67f in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f555bb1edaf in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f555bb0af85 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f555bb06c55 in ?? () from /usr/lib64/libgnutls.so.26 #22 0x00007f555bb07437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #23 0x00007f555c8a961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0 #24 0x00007f555c89ea2b in virNetServerClientInit () from /usr/lib64/libvirt.so.0 #25 0x00007f555c89c821 in ?? () from /usr/lib64/libvirt.so.0 #26 0x00007f555c8a012a in ?? () from /usr/lib64/libvirt.so.0 #27 0x00007f555c79fbf5 in virEventPollRunOnce () from /usr/lib64/libvirt.so.0 #28 0x00007f555c79e825 in virEventRunDefaultImpl () from /usr/lib64/libvirt.so.0 #29 0x00007f555c89c20d in virNetServerRun () from /usr/lib64/libvirt.so.0 #30 0x000000000040c830 in ?? () Thanking you in anticipation. Thanks and Regards, Shree Duth Awasthi. On Fri, Apr 5, 2013 at 9:34 AM, Michal Privoznik <mprivozn@redhat.com>wrote:
On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
GDB:
Program received signal SIGABRT, Aborted. 0x00007f8591246005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f8591246005 in raise () from /lib64/libc.so.6 #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6 #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11 #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11 #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11 #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11 #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11 #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11 #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11 #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11 #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11 #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11 #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11 #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11 #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26 #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f8592cb7437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0
The backstrace shows problem lies in libgcrypt library. So unless libvirt is overwriting some random memory areas, it's a libgcrypt's bug.
Michal
On Fri, Apr 12, 2013 at 02:02:20PM +0200, SHREE DUTH AWASTHI wrote:
Hi Michal,
Open source developers of libgcrypt are pointing out to be a problem with libvirt.
It seems that virsh does not make proper use of libgcrypt or gnutls. In fact, Libgcrypt is telling us what has been done wrong. Please find the latest GDB and let us know comments for the same.
GDB:
Breakpoint 3, 0x00007f555bb07410 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 (gdb) c Continuing. Program received signal SIGABRT, Aborted. 0x00007f555a096005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f555a096005 in raise () from /lib64/libc.so.6 #1 0x00007f555a098e40 in abort () from /lib64/libc.so.6 #2 0x00007f555b87fdc5 in _gcry_logv (level=50, fmt=0x7f555b8c6170 "operation is not possible without initialized secure memory\n", arg_ptr=0x7fff546e1130) at misc.c:136 #3 0x00007f555b8803d5 in _gcry_log_bug (fmt=0x48e0 <Address 0x48e0 out of bounds>) at misc.c:220 #4 0x00007f555b885697 in _gcry_secmem_malloc_internal (size=<value optimized out>) at secmem.c:497 #5 0x00007f555b88579c in _gcry_secmem_malloc (size=136) at secmem.c:522 #6 0x00007f555b880a65 in do_malloc (n=18656, flags=<value optimized out>, mem=0x7fff546e1290) at global.c:553 #7 0x00007f555b880aa9 in _gcry_malloc_secure (n=18656) at global.c:592 #8 0x00007f555b880b19 in _gcry_xmalloc_secure (n=136) at global.c:746 #9 0x00007f555b8c35df in _gcry_mpi_alloc_limb_space (nlimbs=17, secure=18656) at mpiutil.c:92 #10 0x00007f555b8c365f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75 #11 0x00007f555b8b025a in secret (output=0x17cfa20, input=0x17d0480, skey=0x6) at rsa.c:365 #12 0x00007f555b8b045a in _gcry_rsa_sign (algo=<value optimized out>, resarr=0x17d0660, data=0x17d0480, skey=<value optimized out>) at rsa.c:608 #13 0x00007f555b88c1ef in pubkey_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:692 #14 _gcry_pk_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:1807 ---Type <return> to continue, or q <return> to quit--- #15 0x00007f555bb29d8c in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f555bb15e7a in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f555bb1ddd6 in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f555bb1e67f in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f555bb1edaf in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f555bb0af85 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f555bb06c55 in ?? () from /usr/lib64/libgnutls.so.26 #22 0x00007f555bb07437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #23 0x00007f555c8a961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0 #24 0x00007f555c89ea2b in virNetServerClientInit () from /usr/lib64/libvirt.so.0 #25 0x00007f555c89c821 in ?? () from /usr/lib64/libvirt.so.0 #26 0x00007f555c8a012a in ?? () from /usr/lib64/libvirt.so.0 #27 0x00007f555c79fbf5 in virEventPollRunOnce () from /usr/lib64/libvirt.so.0 #28 0x00007f555c79e825 in virEventRunDefaultImpl () from /usr/lib64/libvirt.so.0 #29 0x00007f555c89c20d in virNetServerRun () from /usr/lib64/libvirt.so.0 #30 0x000000000040c830 in ?? ()
What does 'ulimit -a' show for the user that you are running libvirtd as ? Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Hi Daniel, Thanks for your time. Please find the requested output. # ulimit -a core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63706 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited Please let us know if you want the internal log buffer when libvirt. crashed. Thanks and Regards, Shree Duth Awasthi. On Fri, Apr 12, 2013 at 2:24 PM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Fri, Apr 12, 2013 at 02:02:20PM +0200, SHREE DUTH AWASTHI wrote:
Hi Michal,
Open source developers of libgcrypt are pointing out to be a problem with libvirt.
It seems that virsh does not make proper use of libgcrypt or gnutls. In fact, Libgcrypt is telling us what has been done wrong. Please find the latest GDB and let us know comments for the same.
GDB:
Breakpoint 3, 0x00007f555bb07410 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 (gdb) c Continuing. Program received signal SIGABRT, Aborted. 0x00007f555a096005 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007f555a096005 in raise () from /lib64/libc.so.6 #1 0x00007f555a098e40 in abort () from /lib64/libc.so.6 #2 0x00007f555b87fdc5 in _gcry_logv (level=50, fmt=0x7f555b8c6170 "operation is not possible without initialized secure memory\n", arg_ptr=0x7fff546e1130) at misc.c:136 #3 0x00007f555b8803d5 in _gcry_log_bug (fmt=0x48e0 <Address 0x48e0 out of bounds>) at misc.c:220 #4 0x00007f555b885697 in _gcry_secmem_malloc_internal (size=<value optimized out>) at secmem.c:497 #5 0x00007f555b88579c in _gcry_secmem_malloc (size=136) at secmem.c:522 #6 0x00007f555b880a65 in do_malloc (n=18656, flags=<value optimized out>, mem=0x7fff546e1290) at global.c:553 #7 0x00007f555b880aa9 in _gcry_malloc_secure (n=18656) at global.c:592 #8 0x00007f555b880b19 in _gcry_xmalloc_secure (n=136) at global.c:746 #9 0x00007f555b8c35df in _gcry_mpi_alloc_limb_space (nlimbs=17, secure=18656) at mpiutil.c:92 #10 0x00007f555b8c365f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75 #11 0x00007f555b8b025a in secret (output=0x17cfa20, input=0x17d0480, skey=0x6) at rsa.c:365 #12 0x00007f555b8b045a in _gcry_rsa_sign (algo=<value optimized out>, resarr=0x17d0660, data=0x17d0480, skey=<value optimized out>) at rsa.c:608 #13 0x00007f555b88c1ef in pubkey_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:692 #14 _gcry_pk_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:1807 ---Type <return> to continue, or q <return> to quit--- #15 0x00007f555bb29d8c in ?? () from /usr/lib64/libgnutls.so.26 #16 0x00007f555bb15e7a in ?? () from /usr/lib64/libgnutls.so.26 #17 0x00007f555bb1ddd6 in ?? () from /usr/lib64/libgnutls.so.26 #18 0x00007f555bb1e67f in ?? () from /usr/lib64/libgnutls.so.26 #19 0x00007f555bb1edaf in ?? () from /usr/lib64/libgnutls.so.26 #20 0x00007f555bb0af85 in ?? () from /usr/lib64/libgnutls.so.26 #21 0x00007f555bb06c55 in ?? () from /usr/lib64/libgnutls.so.26 #22 0x00007f555bb07437 in gnutls_handshake () from /usr/lib64/libgnutls.so.26 #23 0x00007f555c8a961b in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0 #24 0x00007f555c89ea2b in virNetServerClientInit () from /usr/lib64/libvirt.so.0 #25 0x00007f555c89c821 in ?? () from /usr/lib64/libvirt.so.0 #26 0x00007f555c8a012a in ?? () from /usr/lib64/libvirt.so.0 #27 0x00007f555c79fbf5 in virEventPollRunOnce () from /usr/lib64/libvirt.so.0 #28 0x00007f555c79e825 in virEventRunDefaultImpl () from /usr/lib64/libvirt.so.0 #29 0x00007f555c89c20d in virNetServerRun () from /usr/lib64/libvirt.so.0 #30 0x000000000040c830 in ?? ()
What does 'ulimit -a' show for the user that you are running libvirtd as ?
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
On Fri, Apr 12, 2013 at 03:14:58PM +0200, SHREE DUTH AWASTHI wrote:
Hi Daniel,
Thanks for your time.
Please find the requested output.
# ulimit -a core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63706 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Ok, so ordinarily gnutls would initialize libgcrypt disabling secmem. Libvirt, however, needs to register thread callbacks with gcrypt. Doing this in turn disables gnutls' setup code. So secmem is left enabled. This is not an issue on most distros, since they allow users to mlock sufficient memory. Anyway we need to fix libvirt to disable secmem, since we've blocked gnutls' own setup from running Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
So secmem is left enabled.This is not an issue on most distros, since
Hi Daniel, Thanks for your explanation. they allow users to mlock sufficient memory. I could not understand your above statement. Can you please explain it a bit more. Please let us know the place where we need to look into for the corresponding source code. We will try to provide a fix for it. To add more info, This works fine on our board with libvirt version 0.9.4. This is not working from libvirt 0.9.10 Thanks and Regards, Shree Duth Awasthi. GDB FULL ( if needed ) (gdb) bt full #0 0x00007f5adad0b005 in raise () from /lib64/libc.so.6 No symbol table info available. #1 0x00007f5adad0de40 in abort () from /lib64/libc.so.6 No symbol table info available. #2 0x00007f5adc4f4dc5 in _gcry_logv (level=50, fmt=0x7f5adc53b170 "operation is not possible without initialized secure memory\n", arg_ptr=0x7fff04a2a770) at misc.c:136 No locals. #3 0x00007f5adc4f53d5 in _gcry_log_bug (fmt=0x67d6 <Address 0x67d6 out of bounds>) at misc.c:220 arg_ptr = {{gp_offset = 8, fp_offset = 48, overflow_arg_area = 0x7fff04a2a850, reg_save_area = 0x7fff04a2a790}} #4 0x00007f5adc4fa697 in _gcry_secmem_malloc_internal (size=<value optimized out>) at secmem.c:497 mb = <value optimized out> #5 0x00007f5adc4fa79c in _gcry_secmem_malloc (size=136) at secmem.c:522 p = <value optimized out> #6 0x00007f5adc4f5a65 in do_malloc (n=26582, flags=<value optimized out>, mem=0x7fff04a2a8d0) at global.c:553 m = <value optimized out> #7 0x00007f5adc4f5aa9 in _gcry_malloc_secure (n=26582) at global.c:592 ---Type <return> to continue, or q <return> to quit--- mem = 0x0 #8 0x00007f5adc4f5b19 in _gcry_xmalloc_secure (n=136) at global.c:746 No locals. #9 0x00007f5adc5385df in _gcry_mpi_alloc_limb_space (nlimbs=17, secure=26582) at mpiutil.c:92 len = 26582 #10 0x00007f5adc53865f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75 No locals. #11 0x00007f5adc52525a in secret (output=0x2297d80, input=0x228ce80, skey=0x6) at rsa.c:365 m1 = <value optimized out> m2 = <value optimized out> h = <value optimized out> #12 0x00007f5adc52545a in _gcry_rsa_sign (algo=<value optimized out>, resarr=0x228cfb0, data=0x228ce80, skey=<value optimized out>) at rsa.c:608 sk = {n = 0x231b790, e = 0x231ddc0, d = 0x23100e0, p = 0x230fb10, q = 0x231dd50, u = 0x228c690} #13 0x00007f5adc5011ef in pubkey_sign (r_sig=0x7fff04a2aac8, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:692 module = <value optimized out> i = 32767 ---Type <return> to continue, or q <return> to quit--- #14 _gcry_pk_sign (r_sig=0x7fff04a2aac8, s_hash=<value optimized out>, s_skey=<value optimized out>) at pubkey.c:1807 skey = 0x22991c0 hash = 0x228ce80 result = 0x228cfb0 pubkey = <value optimized out> module = 0x224b890 algo_name = 0x7f5adc547967 "rsa" algo_elems = 0x7f5adc547bd1 "s" i = <value optimized out> rc = <value optimized out> __PRETTY_FUNCTION__ = "_gcry_pk_sign" __FUNCTION__ = "_gcry_pk_sign" #15 0x00007f5adc79ef9c in _wrap_gcry_pk_sign (algo=GNUTLS_PK_RSA, signature=0x7fff04a2ab50, vdata=<value optimized out>, pk_params=0x7fff04a2ab70) at pk-libgcrypt.c:308 s_hash = 0x230f370 s_key = 0x2288680 ---Type <return> to continue, or q <return> to quit--- s_sig = 0x0 list = <value optimized out> rc = <value optimized out> ret = <value optimized out> hash = 0x22cfe30 res = {0x0, 0x0} #16 0x00007f5adc78b08a in _gnutls_pkcs1_rsa_encrypt (ciphertext=<value optimized out>, plaintext=<value optimized out>, params=<value optimized out>, params_len=6, btype=<value optimized out>) at gnutls_pk.c:150 i = <value optimized out> pad = <value optimized out> ret = <value optimized out> edata = 0x228c980 "" ps = 0x228c982 "\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377" k = <value optimized out> ---Type <return> to continue, or q <return> to quit--- psize = <value optimized out> mod_bits = <value optimized out> pk_params = {params = {0x224f360, 0x224ef40, 0x224f3d0, 0x224f140, 0x225dba0, 0x224ffc0}, params_nr = 6, flags = 32767} to_encrypt = {data = 0x228c980 "", size = 128} encrypted = {data = 0x7fff04a2ad90 "!\002", size = 3671393680} #17 0x00007f5adc792fe6 in _gnutls_sign (algo=<value optimized out>, params=<value optimized out>, params_size=<value optimized out>, data=0x7fff04a2acb0, signature=0x0) at gnutls_sig.c:251 ret = <value optimized out> #18 0x00007f5adc79388f in _gnutls_handshake_sign_data (session=0x22ceb70, cert=0x2278c20, pkey=<value optimized out>, params=<value optimized out>, signature=0x7fff04a2ad90, sign_algo=<value optimized out>) at gnutls_sig.c:226 dconcat = {data = 0x7fff04a2acd0 "0!0\t\006\005+\016\003\002\032\005", size = 35} ret = 0 td_sha = {registered = 0, hd = {gc = 0x2288680, rh = {cc = 0x2288680, ctx = 0x82}}, algorithm = GNUTLS_MAC_SHA1, key = 0x7fff04a2ad00, keysize = -623573616, active = 0} concat = "0!0\t\006\005+\016\003\002\032\005\000\004\024\213^q\253^G\342\062\256\263\310\060\230\030(2-d\212\300\004\377\177\000\000\020\255\242\004\377\177\000\000\200\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\200", '\000' <repeats 15 times>"\340, \306(" ---Type <return> to continue, or q <return> to quit--- ver = GNUTLS_TLS1_2 hash_algo = GNUTLS_DIG_SHA1 #19 0x00007f5adc793fbf in gen_dhe_server_kx (session=0x22ceb70, data=0x7fff04a2ae00) at auth_dhe.c:152 g = <value optimized out> p = <value optimized out> mpis = <value optimized out> ret = 263 data_size = <value optimized out> apr_cert_list = 0x2278c20 apr_pkey = 0x2278560 apr_cert_list_length = 1 signature = {data = 0x221 <Address 0x221 out of bounds>, size = 0} ddata = {data = 0x2280f70 "", size = 263} dh_params = <value optimized out> sign_algo = <value optimized out> ver = GNUTLS_TLS1_2 ---Type <return> to continue, or q <return> to quit--- #20 0x00007f5adc780195 in _gnutls_send_server_kx_message (session=0x67d6, again=<value optimized out>) at gnutls_kx.c:207 data = 0x2280f70 "" data_size = <value optimized out> ret = <value optimized out> #21 0x00007f5adc77bc55 in _gnutls_handshake_server (session=0x22ceb70) at gnutls_handshake.c:3047 ret = 545 #22 0x00007f5adc77c481 in gnutls_handshake (session=0x22ceb70) at gnutls_handshake.c:2709 ret = <value optimized out> #23 0x00007f5add51e744 in virNetTLSSessionHandshake () from /usr/lib64/libvirt.so.0 No symbol table info available. #24 0x00007f5add513a2b in virNetServerClientInit () from /usr/lib64/libvirt.so.0 No symbol table info available. #25 0x00007f5add511821 in ?? () from /usr/lib64/libvirt.so.0 No symbol table info available. #26 0x00007f5add51512a in ?? () from /usr/lib64/libvirt.so.0 No symbol table info available. On Fri, Apr 12, 2013 at 3:24 PM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Fri, Apr 12, 2013 at 03:14:58PM +0200, SHREE DUTH AWASTHI wrote:
Hi Daniel,
Thanks for your time.
Please find the requested output.
# ulimit -a core file size (blocks, -c) 1000000 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 63706 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 1024 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Ok, so ordinarily gnutls would initialize libgcrypt disabling secmem. Libvirt, however, needs to register thread callbacks with gcrypt. Doing this in turn disables gnutls' setup code. So secmem is left enabled. This is not an issue on most distros, since they allow users to mlock sufficient memory.
Anyway we need to fix libvirt to disable secmem, since we've blocked gnutls' own setup from running
Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
participants (3)
-
Daniel P. Berrange -
Michal Privoznik -
SHREE DUTH AWASTHI