[libvirt-users] Libvirtd dead, pid still exists. ( Problem might be with TLS interface of libvirtd )
2:04 a.m.
Dear All,
Please find few minutes from your time and guide us with some pointers if
possible.
We are facing a libvirtd crash when we are trying to connect to qemu by
default TLS transport.
# virsh -c qemu+tls://localhost/system version
error: authentication failed: TLS handshake failed A TLS packet with
unexpected length was received.
error: failed to connect to the hypervisor
I used my own CA and certificates (generated using
http://libvirt.org/remote.html#Remote_libvirtd_configuration on Redhat PC)
on both Kontron PC and our Board (with patched libvirt). Libvirtd.conf was
modified so that libvirt is listening all IPs using default IP (so that it
was possible to use same certificates on all machines)
These directories and files created and used.
/etc/pki/CA/cacert.pem
/etc/pki/libvirt/private/serverkey.pem
/etc/pki/libvirt/servercert.pem
/etc/pki/libvirt/private/clientkey.pem
/etc/pki/libvirt/clientcert.pem
TLS connection worked fine with Kontron PC
# virsh -c qemu+tls://localhost/system version
Compiled against library: libvir 0.9.5
Using library: libvir 0.9.5
Using API: QEMU 0.9.5
Running hypervisor: QEMU 0.12.1
But libvirt crashed on our Board (using libvirt 0.10.2,
gnutls-2.10.5-1_WR4.3.x86_64 and libudev-161-4 rpms )
# virsh -c qemu+tls://localhost/system version
error: authentication failed: TLS handshake failed A TLS packet with
unexpected length was received.
error: failed to connect to the hypervisor
GDB:
Program received signal SIGABRT, Aborted.
0x00007f8591246005 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f8591246005 in raise () from /lib64/libc.so.6
#1 0x00007f8591248e40 in abort () from /lib64/libc.so.6
#2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11
#3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11
#4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11
#5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11
#6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11
#7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11
#8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11
#9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11
#10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11
#11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11
#12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11
#13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11
#14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26
#15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26
#16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26
#17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26
#18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26
#19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26
#20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26
#21 0x00007f8592cb7437 in gnutls_handshake () from
/usr/lib64/libgnutls.so.26
#22 0x00007f8593a5961b in virNetTLSSessionHandshake () from
/usr/lib64/libvirt.so.0
Please let us know if it is a known problem. If not, I will raise a new bug
for the same. ( Atleast I coulnt find the the known issues )
Thanks and Regards,
Shree Duth Awasthi.
2:34 a.m.
On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
GDB:
Program received signal SIGABRT, Aborted.
0x00007f8591246005 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f8591246005 in raise () from /lib64/libc.so.6
#1 0x00007f8591248e40 in abort () from /lib64/libc.so.6
#2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11
#3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11
#4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11
#5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11
#6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11
#7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11
#8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11
#9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11
#10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11
#11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11
#12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11
#13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11
#14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26
#15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26
#16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26
#17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26
#18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26
#19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26
#20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26
#21 0x00007f8592cb7437 in gnutls_handshake () from
/usr/lib64/libgnutls.so.26
#22 0x00007f8593a5961b in virNetTLSSessionHandshake () from
/usr/lib64/libvirt.so.0
The backstrace shows problem lies in libgcrypt library. So unless
libvirt is overwriting some random memory areas, it's a libgcrypt's bug.
Michal
7:41 a.m.
Hi Michal,
Thanks for your time.
Yes it seems so. Adding one more info :
The issue is not reproducable on our board with
libvirt-0.9.4-rc0_WR4.3.x86_64, gnutls-utils-1.4.1-2_WR4.3.x86_64,
libgcrypt-1.4.0-3_WR4.3.x86_64.
We are unable to debug more as GDB symbols are not completely resolved.
So, can you please give us some pointers to debug the issue further.
Thanks and Regards,
Shree Duth Awasthi.
On Fri, Apr 5, 2013 at 9:34 AM, Michal Privoznik <mprivozn(a)redhat.com>wrote:
On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
> GDB:
>
> Program received signal SIGABRT, Aborted.
> 0x00007f8591246005 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0 0x00007f8591246005 in raise () from /lib64/libc.so.6
> #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6
> #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11
> #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11
> #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11
> #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11
> #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11
> #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11
> #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11
> #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11
> #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11
> #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11
> #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11
> #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11
> #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26
> #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26
> #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26
> #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26
> #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26
> #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26
> #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26
> #21 0x00007f8592cb7437 in gnutls_handshake () from
> /usr/lib64/libgnutls.so.26
> #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from
> /usr/lib64/libvirt.so.0
>
The backstrace shows problem lies in libgcrypt library. So unless
libvirt is overwriting some random memory areas, it's a libgcrypt's bug.
Michal
9:13 a.m.
Dear All,
A Gentle Reminder !!!
Any suggestions please on how to debug further
Gist : libvirt crash when trying to inquiry libvirt version using curl with
TLS ( virsh -c qemu+tls://localhost/system version )
Your inputs would be of great help !
Thanks and Regards,
Shree Duth Awasthi.
On Fri, Apr 5, 2013 at 2:41 PM, SHREE DUTH AWASTHI <
shreeduth.awasthi(a)gmail.com> wrote:
Hi Michal,
Thanks for your time.
Yes it seems so. Adding one more info :
The issue is not reproducable on our board with
libvirt-0.9.4-rc0_WR4.3.x86_64, gnutls-utils-1.4.1-2_WR4.3.x86_64,
libgcrypt-1.4.0-3_WR4.3.x86_64.
We are unable to debug more as GDB symbols are not completely resolved.
So, can you please give us some pointers to debug the issue further.
Thanks and Regards,
Shree Duth Awasthi.
On Fri, Apr 5, 2013 at 9:34 AM, Michal Privoznik <mprivozn(a)redhat.com>wrote:
> On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
>
> > GDB:
> >
> > Program received signal SIGABRT, Aborted.
> > 0x00007f8591246005 in raise () from /lib64/libc.so.6
> > (gdb) bt
> > #0 0x00007f8591246005 in raise () from /lib64/libc.so.6
> > #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6
> > #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11
> > #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11
> > #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11
> > #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11
> > #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11
> > #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11
> > #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11
> > #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11
> > #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11
> > #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11
> > #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11
> > #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11
> > #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26
> > #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26
> > #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26
> > #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26
> > #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26
> > #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26
> > #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26
> > #21 0x00007f8592cb7437 in gnutls_handshake () from
> > /usr/lib64/libgnutls.so.26
> > #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from
> > /usr/lib64/libvirt.so.0
> >
>
> The backstrace shows problem lies in libgcrypt library. So unless
> libvirt is overwriting some random memory areas, it's a libgcrypt's bug.
>
> Michal
>
7:02 a.m.
Hi Michal,
Open source developers of libgcrypt are pointing out to be a problem with
libvirt.
It seems that virsh does not make proper use of libgcrypt or gnutls. In
fact, Libgcrypt is telling us what has been done wrong. Please find the
latest GDB and let us know comments for the same.
GDB:
Breakpoint 3, 0x00007f555bb07410 in gnutls_handshake () from
/usr/lib64/libgnutls.so.26
(gdb) c
Continuing.
Program received signal SIGABRT, Aborted.
0x00007f555a096005 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f555a096005 in raise () from /lib64/libc.so.6
#1 0x00007f555a098e40 in abort () from /lib64/libc.so.6
#2 0x00007f555b87fdc5 in _gcry_logv (level=50, fmt=0x7f555b8c6170 "operation
is not possible without initialized secure memory\n",
arg_ptr=0x7fff546e1130) at misc.c:136
#3 0x00007f555b8803d5 in _gcry_log_bug (fmt=0x48e0 <Address 0x48e0 out of
bounds>) at misc.c:220
#4 0x00007f555b885697 in _gcry_secmem_malloc_internal (size=<value
optimized out>) at secmem.c:497
#5 0x00007f555b88579c in _gcry_secmem_malloc (size=136) at secmem.c:522
#6 0x00007f555b880a65 in do_malloc (n=18656, flags=<value optimized out>,
mem=0x7fff546e1290) at global.c:553
#7 0x00007f555b880aa9 in _gcry_malloc_secure (n=18656) at global.c:592
#8 0x00007f555b880b19 in _gcry_xmalloc_secure (n=136) at global.c:746
#9 0x00007f555b8c35df in _gcry_mpi_alloc_limb_space (nlimbs=17,
secure=18656) at mpiutil.c:92
#10 0x00007f555b8c365f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75
#11 0x00007f555b8b025a in secret (output=0x17cfa20, input=0x17d0480,
skey=0x6) at rsa.c:365
#12 0x00007f555b8b045a in _gcry_rsa_sign (algo=<value optimized out>,
resarr=0x17d0660, data=0x17d0480, skey=<value optimized out>) at rsa.c:608
#13 0x00007f555b88c1ef in pubkey_sign (r_sig=0x7fff546e1488, s_hash=<value
optimized out>, s_skey=<value optimized out>) at pubkey.c:692
#14 _gcry_pk_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>,
s_skey=<value optimized out>) at pubkey.c:1807
---Type <return> to continue, or q <return> to quit---
#15 0x00007f555bb29d8c in ?? () from /usr/lib64/libgnutls.so.26
#16 0x00007f555bb15e7a in ?? () from /usr/lib64/libgnutls.so.26
#17 0x00007f555bb1ddd6 in ?? () from /usr/lib64/libgnutls.so.26
#18 0x00007f555bb1e67f in ?? () from /usr/lib64/libgnutls.so.26
#19 0x00007f555bb1edaf in ?? () from /usr/lib64/libgnutls.so.26
#20 0x00007f555bb0af85 in ?? () from /usr/lib64/libgnutls.so.26
#21 0x00007f555bb06c55 in ?? () from /usr/lib64/libgnutls.so.26
#22 0x00007f555bb07437 in gnutls_handshake () from
/usr/lib64/libgnutls.so.26
#23 0x00007f555c8a961b in virNetTLSSessionHandshake () from
/usr/lib64/libvirt.so.0
#24 0x00007f555c89ea2b in virNetServerClientInit () from
/usr/lib64/libvirt.so.0
#25 0x00007f555c89c821 in ?? () from /usr/lib64/libvirt.so.0
#26 0x00007f555c8a012a in ?? () from /usr/lib64/libvirt.so.0
#27 0x00007f555c79fbf5 in virEventPollRunOnce () from
/usr/lib64/libvirt.so.0
#28 0x00007f555c79e825 in virEventRunDefaultImpl () from
/usr/lib64/libvirt.so.0
#29 0x00007f555c89c20d in virNetServerRun () from /usr/lib64/libvirt.so.0
#30 0x000000000040c830 in ?? ()
Thanking you in anticipation.
Thanks and Regards,
Shree Duth Awasthi.
On Fri, Apr 5, 2013 at 9:34 AM, Michal Privoznik <mprivozn(a)redhat.com>wrote:
On 05.04.2013 09:04, SHREE DUTH AWASTHI wrote:
> GDB:
>
> Program received signal SIGABRT, Aborted.
> 0x00007f8591246005 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0 0x00007f8591246005 in raise () from /lib64/libc.so.6
> #1 0x00007f8591248e40 in abort () from /lib64/libc.so.6
> #2 0x00007f8592a2fdc5 in ?? () from /lib64/libgcrypt.so.11
> #3 0x00007f8592a303d5 in ?? () from /lib64/libgcrypt.so.11
> #4 0x00007f8592a35697 in ?? () from /lib64/libgcrypt.so.11
> #5 0x00007f8592a3579c in ?? () from /lib64/libgcrypt.so.11
> #6 0x00007f8592a30a65 in ?? () from /lib64/libgcrypt.so.11
> #7 0x00007f8592a30aa9 in ?? () from /lib64/libgcrypt.so.11
> #8 0x00007f8592a30b19 in ?? () from /lib64/libgcrypt.so.11
> #9 0x00007f8592a735df in ?? () from /lib64/libgcrypt.so.11
> #10 0x00007f8592a7365f in ?? () from /lib64/libgcrypt.so.11
> #11 0x00007f8592a6025a in ?? () from /lib64/libgcrypt.so.11
> #12 0x00007f8592a6045a in ?? () from /lib64/libgcrypt.so.11
> #13 0x00007f8592a3c1ef in ?? () from /lib64/libgcrypt.so.11
> #14 0x00007f8592cd9d8c in ?? () from /usr/lib64/libgnutls.so.26
> #15 0x00007f8592cc5e7a in ?? () from /usr/lib64/libgnutls.so.26
> #16 0x00007f8592ccddd6 in ?? () from /usr/lib64/libgnutls.so.26
> #17 0x00007f8592cce67f in ?? () from /usr/lib64/libgnutls.so.26
> #18 0x00007f8592ccedaf in ?? () from /usr/lib64/libgnutls.so.26
> #19 0x00007f8592cbaf85 in ?? () from /usr/lib64/libgnutls.so.26
> #20 0x00007f8592cb6c55 in ?? () from /usr/lib64/libgnutls.so.26
> #21 0x00007f8592cb7437 in gnutls_handshake () from
> /usr/lib64/libgnutls.so.26
> #22 0x00007f8593a5961b in virNetTLSSessionHandshake () from
> /usr/lib64/libvirt.so.0
>
The backstrace shows problem lies in libgcrypt library. So unless
libvirt is overwriting some random memory areas, it's a libgcrypt's bug.
Michal
7:24 a.m.
On Fri, Apr 12, 2013 at 02:02:20PM +0200, SHREE DUTH AWASTHI wrote:
Hi Michal,
Open source developers of libgcrypt are pointing out to be a problem with
libvirt.
It seems that virsh does not make proper use of libgcrypt or gnutls. In
fact, Libgcrypt is telling us what has been done wrong. Please find the
latest GDB and let us know comments for the same.
GDB:
Breakpoint 3, 0x00007f555bb07410 in gnutls_handshake () from
/usr/lib64/libgnutls.so.26
(gdb) c
Continuing.
Program received signal SIGABRT, Aborted.
0x00007f555a096005 in raise () from /lib64/libc.so.6
(gdb) bt
#0 0x00007f555a096005 in raise () from /lib64/libc.so.6
#1 0x00007f555a098e40 in abort () from /lib64/libc.so.6
#2 0x00007f555b87fdc5 in _gcry_logv (level=50, fmt=0x7f555b8c6170 "operation
is not possible without initialized secure memory\n",
arg_ptr=0x7fff546e1130) at misc.c:136
#3 0x00007f555b8803d5 in _gcry_log_bug (fmt=0x48e0 <Address 0x48e0 out of
bounds>) at misc.c:220
#4 0x00007f555b885697 in _gcry_secmem_malloc_internal (size=<value
optimized out>) at secmem.c:497
#5 0x00007f555b88579c in _gcry_secmem_malloc (size=136) at secmem.c:522
#6 0x00007f555b880a65 in do_malloc (n=18656, flags=<value optimized out>,
mem=0x7fff546e1290) at global.c:553
#7 0x00007f555b880aa9 in _gcry_malloc_secure (n=18656) at global.c:592
#8 0x00007f555b880b19 in _gcry_xmalloc_secure (n=136) at global.c:746
#9 0x00007f555b8c35df in _gcry_mpi_alloc_limb_space (nlimbs=17,
secure=18656) at mpiutil.c:92
#10 0x00007f555b8c365f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75
#11 0x00007f555b8b025a in secret (output=0x17cfa20, input=0x17d0480,
skey=0x6) at rsa.c:365
#12 0x00007f555b8b045a in _gcry_rsa_sign (algo=<value optimized out>,
resarr=0x17d0660, data=0x17d0480, skey=<value optimized out>) at rsa.c:608
#13 0x00007f555b88c1ef in pubkey_sign (r_sig=0x7fff546e1488, s_hash=<value
optimized out>, s_skey=<value optimized out>) at pubkey.c:692
#14 _gcry_pk_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>,
s_skey=<value optimized out>) at pubkey.c:1807
---Type <return> to continue, or q <return> to quit---
#15 0x00007f555bb29d8c in ?? () from /usr/lib64/libgnutls.so.26
#16 0x00007f555bb15e7a in ?? () from /usr/lib64/libgnutls.so.26
#17 0x00007f555bb1ddd6 in ?? () from /usr/lib64/libgnutls.so.26
#18 0x00007f555bb1e67f in ?? () from /usr/lib64/libgnutls.so.26
#19 0x00007f555bb1edaf in ?? () from /usr/lib64/libgnutls.so.26
#20 0x00007f555bb0af85 in ?? () from /usr/lib64/libgnutls.so.26
#21 0x00007f555bb06c55 in ?? () from /usr/lib64/libgnutls.so.26
#22 0x00007f555bb07437 in gnutls_handshake () from
/usr/lib64/libgnutls.so.26
#23 0x00007f555c8a961b in virNetTLSSessionHandshake () from
/usr/lib64/libvirt.so.0
#24 0x00007f555c89ea2b in virNetServerClientInit () from
/usr/lib64/libvirt.so.0
#25 0x00007f555c89c821 in ?? () from /usr/lib64/libvirt.so.0
#26 0x00007f555c8a012a in ?? () from /usr/lib64/libvirt.so.0
#27 0x00007f555c79fbf5 in virEventPollRunOnce () from
/usr/lib64/libvirt.so.0
#28 0x00007f555c79e825 in virEventRunDefaultImpl () from
/usr/lib64/libvirt.so.0
#29 0x00007f555c89c20d in virNetServerRun () from /usr/lib64/libvirt.so.0
#30 0x000000000040c830 in ?? ()
What does 'ulimit -a' show for the user that you are running libvirtd
as ?
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
8:14 a.m.
Hi Daniel,
Thanks for your time.
Please find the requested output.
# ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63706
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Please let us know if you want the internal log buffer when libvirt.
crashed.
Thanks and Regards,
Shree Duth Awasthi.
On Fri, Apr 12, 2013 at 2:24 PM, Daniel P. Berrange <berrange(a)redhat.com>wrote:
On Fri, Apr 12, 2013 at 02:02:20PM +0200, SHREE DUTH AWASTHI wrote:
> Hi Michal,
>
> Open source developers of libgcrypt are pointing out to be a problem with
> libvirt.
>
> It seems that virsh does not make proper use of libgcrypt or gnutls. In
> fact, Libgcrypt is telling us what has been done wrong. Please find the
> latest GDB and let us know comments for the same.
>
> GDB:
>
> Breakpoint 3, 0x00007f555bb07410 in gnutls_handshake () from
> /usr/lib64/libgnutls.so.26
> (gdb) c
> Continuing.
> Program received signal SIGABRT, Aborted.
> 0x00007f555a096005 in raise () from /lib64/libc.so.6
> (gdb) bt
> #0 0x00007f555a096005 in raise () from /lib64/libc.so.6
> #1 0x00007f555a098e40 in abort () from /lib64/libc.so.6
> #2 0x00007f555b87fdc5 in _gcry_logv (level=50, fmt=0x7f555b8c6170
"operation
> is not possible without initialized secure memory\n",
> arg_ptr=0x7fff546e1130) at misc.c:136
> #3 0x00007f555b8803d5 in _gcry_log_bug (fmt=0x48e0 <Address 0x48e0 out
of
> bounds>) at misc.c:220
> #4 0x00007f555b885697 in _gcry_secmem_malloc_internal (size=<value
> optimized out>) at secmem.c:497
> #5 0x00007f555b88579c in _gcry_secmem_malloc (size=136) at secmem.c:522
> #6 0x00007f555b880a65 in do_malloc (n=18656, flags=<value optimized
out>,
> mem=0x7fff546e1290) at global.c:553
> #7 0x00007f555b880aa9 in _gcry_malloc_secure (n=18656) at global.c:592
> #8 0x00007f555b880b19 in _gcry_xmalloc_secure (n=136) at global.c:746
> #9 0x00007f555b8c35df in _gcry_mpi_alloc_limb_space (nlimbs=17,
> secure=18656) at mpiutil.c:92
> #10 0x00007f555b8c365f in _gcry_mpi_alloc_secure (nlimbs=17) at
mpiutil.c:75
> #11 0x00007f555b8b025a in secret (output=0x17cfa20, input=0x17d0480,
> skey=0x6) at rsa.c:365
> #12 0x00007f555b8b045a in _gcry_rsa_sign (algo=<value optimized out>,
> resarr=0x17d0660, data=0x17d0480, skey=<value optimized out>) at
rsa.c:608
> #13 0x00007f555b88c1ef in pubkey_sign (r_sig=0x7fff546e1488,
s_hash=<value
> optimized out>, s_skey=<value optimized out>) at pubkey.c:692
> #14 _gcry_pk_sign (r_sig=0x7fff546e1488, s_hash=<value optimized out>,
> s_skey=<value optimized out>) at pubkey.c:1807
> ---Type <return> to continue, or q <return> to quit---
> #15 0x00007f555bb29d8c in ?? () from /usr/lib64/libgnutls.so.26
> #16 0x00007f555bb15e7a in ?? () from /usr/lib64/libgnutls.so.26
> #17 0x00007f555bb1ddd6 in ?? () from /usr/lib64/libgnutls.so.26
> #18 0x00007f555bb1e67f in ?? () from /usr/lib64/libgnutls.so.26
> #19 0x00007f555bb1edaf in ?? () from /usr/lib64/libgnutls.so.26
> #20 0x00007f555bb0af85 in ?? () from /usr/lib64/libgnutls.so.26
> #21 0x00007f555bb06c55 in ?? () from /usr/lib64/libgnutls.so.26
> #22 0x00007f555bb07437 in gnutls_handshake () from
> /usr/lib64/libgnutls.so.26
> #23 0x00007f555c8a961b in virNetTLSSessionHandshake () from
> /usr/lib64/libvirt.so.0
> #24 0x00007f555c89ea2b in virNetServerClientInit () from
> /usr/lib64/libvirt.so.0
> #25 0x00007f555c89c821 in ?? () from /usr/lib64/libvirt.so.0
> #26 0x00007f555c8a012a in ?? () from /usr/lib64/libvirt.so.0
> #27 0x00007f555c79fbf5 in virEventPollRunOnce () from
> /usr/lib64/libvirt.so.0
> #28 0x00007f555c79e825 in virEventRunDefaultImpl () from
> /usr/lib64/libvirt.so.0
> #29 0x00007f555c89c20d in virNetServerRun () from /usr/lib64/libvirt.so.0
> #30 0x000000000040c830 in ?? ()
What does 'ulimit -a' show for the user that you are running libvirtd
as ?
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:|
|: http://libvirt.org -o- http://virt-manager.org:|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/:|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
8:24 a.m.
On Fri, Apr 12, 2013 at 03:14:58PM +0200, SHREE DUTH AWASTHI wrote:
Hi Daniel,
Thanks for your time.
Please find the requested output.
# ulimit -a
core file size (blocks, -c) 1000000
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 63706
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 1024
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
Ok, so ordinarily gnutls would initialize libgcrypt disabling secmem.
Libvirt, however, needs to register thread callbacks with gcrypt. Doing
this in turn disables gnutls' setup code. So secmem is left enabled.
This is not an issue on most distros, since they allow users to mlock
sufficient memory.
Anyway we need to fix libvirt to disable secmem, since we've blocked
gnutls' own setup from running
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
8:47 a.m.
Hi Daniel,
Thanks for your explanation.
>So secmem is left enabled.This is not an issue on most distros,
since
they allow users to mlock
sufficient memory.
I could not understand your above statement. Can you please explain it a
bit more.
Please let us know the place where we need to look into for the
corresponding source code. We will try to provide a fix for it.
To add more info, This works fine on our board with libvirt version 0.9.4.
This is not working from libvirt 0.9.10
Thanks and Regards,
Shree Duth Awasthi.
GDB FULL ( if needed )
(gdb) bt full
#0 0x00007f5adad0b005 in raise () from /lib64/libc.so.6
No symbol table info available.
#1 0x00007f5adad0de40 in abort () from /lib64/libc.so.6
No symbol table info available.
#2 0x00007f5adc4f4dc5 in _gcry_logv (level=50, fmt=0x7f5adc53b170
"operation is not possible without initialized secure memory\n",
arg_ptr=0x7fff04a2a770) at misc.c:136
No locals.
#3 0x00007f5adc4f53d5 in _gcry_log_bug (fmt=0x67d6 <Address 0x67d6 out of
bounds>) at misc.c:220
arg_ptr = {{gp_offset = 8, fp_offset = 48, overflow_arg_area =
0x7fff04a2a850, reg_save_area = 0x7fff04a2a790}}
#4 0x00007f5adc4fa697 in _gcry_secmem_malloc_internal (size=<value
optimized out>) at secmem.c:497
mb = <value optimized out>
#5 0x00007f5adc4fa79c in _gcry_secmem_malloc (size=136) at secmem.c:522
p = <value optimized out>
#6 0x00007f5adc4f5a65 in do_malloc (n=26582, flags=<value optimized out>,
mem=0x7fff04a2a8d0) at global.c:553
m = <value optimized out>
#7 0x00007f5adc4f5aa9 in _gcry_malloc_secure (n=26582) at global.c:592
---Type <return> to continue, or q <return> to quit---
mem = 0x0
#8 0x00007f5adc4f5b19 in _gcry_xmalloc_secure (n=136) at global.c:746
No locals.
#9 0x00007f5adc5385df in _gcry_mpi_alloc_limb_space (nlimbs=17,
secure=26582) at mpiutil.c:92
len = 26582
#10 0x00007f5adc53865f in _gcry_mpi_alloc_secure (nlimbs=17) at mpiutil.c:75
No locals.
#11 0x00007f5adc52525a in secret (output=0x2297d80, input=0x228ce80,
skey=0x6) at rsa.c:365
m1 = <value optimized out>
m2 = <value optimized out>
h = <value optimized out>
#12 0x00007f5adc52545a in _gcry_rsa_sign (algo=<value optimized out>,
resarr=0x228cfb0, data=0x228ce80, skey=<value optimized out>) at rsa.c:608
sk = {n = 0x231b790, e = 0x231ddc0, d = 0x23100e0, p = 0x230fb10, q
= 0x231dd50, u = 0x228c690}
#13 0x00007f5adc5011ef in pubkey_sign (r_sig=0x7fff04a2aac8, s_hash=<value
optimized out>, s_skey=<value optimized out>) at pubkey.c:692
module = <value optimized out>
i = 32767
---Type <return> to continue, or q <return> to quit---
#14 _gcry_pk_sign (r_sig=0x7fff04a2aac8, s_hash=<value optimized out>,
s_skey=<value optimized out>) at pubkey.c:1807
skey = 0x22991c0
hash = 0x228ce80
result = 0x228cfb0
pubkey = <value optimized out>
module = 0x224b890
algo_name = 0x7f5adc547967 "rsa"
algo_elems = 0x7f5adc547bd1 "s"
i = <value optimized out>
rc = <value optimized out>
__PRETTY_FUNCTION__ = "_gcry_pk_sign"
__FUNCTION__ = "_gcry_pk_sign"
#15 0x00007f5adc79ef9c in _wrap_gcry_pk_sign (algo=GNUTLS_PK_RSA,
signature=0x7fff04a2ab50, vdata=<value optimized out>,
pk_params=0x7fff04a2ab70)
at pk-libgcrypt.c:308
s_hash = 0x230f370
s_key = 0x2288680
---Type <return> to continue, or q <return> to quit---
s_sig = 0x0
list = <value optimized out>
rc = <value optimized out>
ret = <value optimized out>
hash = 0x22cfe30
res = {0x0, 0x0}
#16 0x00007f5adc78b08a in _gnutls_pkcs1_rsa_encrypt (ciphertext=<value
optimized out>, plaintext=<value optimized out>, params=<value optimized
out>,
params_len=6, btype=<value optimized out>) at gnutls_pk.c:150
i = <value optimized out>
pad = <value optimized out>
ret = <value optimized out>
edata = 0x228c980 ""
ps = 0x228c982
"\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"
k = <value optimized out>
---Type <return> to continue, or q <return> to quit---
psize = <value optimized out>
mod_bits = <value optimized out>
pk_params = {params = {0x224f360, 0x224ef40, 0x224f3d0, 0x224f140,
0x225dba0, 0x224ffc0}, params_nr = 6, flags = 32767}
to_encrypt = {data = 0x228c980 "", size = 128}
encrypted = {data = 0x7fff04a2ad90 "!\002", size = 3671393680}
#17 0x00007f5adc792fe6 in _gnutls_sign (algo=<value optimized out>,
params=<value optimized out>, params_size=<value optimized out>,
data=0x7fff04a2acb0, signature=0x0) at gnutls_sig.c:251
ret = <value optimized out>
#18 0x00007f5adc79388f in _gnutls_handshake_sign_data (session=0x22ceb70,
cert=0x2278c20, pkey=<value optimized out>, params=<value optimized out>,
signature=0x7fff04a2ad90, sign_algo=<value optimized out>) at
gnutls_sig.c:226
dconcat = {data = 0x7fff04a2acd0
"0!0\t\006\005+\016\003\002\032\005", size = 35}
ret = 0
td_sha = {registered = 0, hd = {gc = 0x2288680, rh = {cc =
0x2288680, ctx = 0x82}}, algorithm = GNUTLS_MAC_SHA1, key = 0x7fff04a2ad00,
keysize = -623573616, active = 0}
concat =
"0!0\t\006\005+\016\003\002\032\005\000\004\024\213^q\253^G\342\062\256\263\310\060\230\030(2-d\212\300\004\377\177\000\000\020\255\242\004\377\177\000\000\200\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\200",
'\000' <repeats 15 times>"\340, \306("
---Type <return> to continue, or q <return> to quit---
ver = GNUTLS_TLS1_2
hash_algo = GNUTLS_DIG_SHA1
#19 0x00007f5adc793fbf in gen_dhe_server_kx (session=0x22ceb70,
data=0x7fff04a2ae00) at auth_dhe.c:152
g = <value optimized out>
p = <value optimized out>
mpis = <value optimized out>
ret = 263
data_size = <value optimized out>
apr_cert_list = 0x2278c20
apr_pkey = 0x2278560
apr_cert_list_length = 1
signature = {data = 0x221 <Address 0x221 out of bounds>, size = 0}
ddata = {data = 0x2280f70 "", size = 263}
dh_params = <value optimized out>
sign_algo = <value optimized out>
ver = GNUTLS_TLS1_2
---Type <return> to continue, or q <return> to quit---
#20 0x00007f5adc780195 in _gnutls_send_server_kx_message (session=0x67d6,
again=<value optimized out>) at gnutls_kx.c:207
data = 0x2280f70 ""
data_size = <value optimized out>
ret = <value optimized out>
#21 0x00007f5adc77bc55 in _gnutls_handshake_server (session=0x22ceb70) at
gnutls_handshake.c:3047
ret = 545
#22 0x00007f5adc77c481 in gnutls_handshake (session=0x22ceb70) at
gnutls_handshake.c:2709
ret = <value optimized out>
#23 0x00007f5add51e744 in virNetTLSSessionHandshake () from
/usr/lib64/libvirt.so.0
No symbol table info available.
#24 0x00007f5add513a2b in virNetServerClientInit () from
/usr/lib64/libvirt.so.0
No symbol table info available.
#25 0x00007f5add511821 in ?? () from /usr/lib64/libvirt.so.0
No symbol table info available.
#26 0x00007f5add51512a in ?? () from /usr/lib64/libvirt.so.0
No symbol table info available.
On Fri, Apr 12, 2013 at 3:24 PM, Daniel P. Berrange <berrange(a)redhat.com>wrote:
On Fri, Apr 12, 2013 at 03:14:58PM +0200, SHREE DUTH AWASTHI wrote:
> Hi Daniel,
>
> Thanks for your time.
>
> Please find the requested output.
>
> # ulimit -a
> core file size (blocks, -c) 1000000
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 63706
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 1024
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) 1024
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
Ok, so ordinarily gnutls would initialize libgcrypt disabling secmem.
Libvirt, however, needs to register thread callbacks with gcrypt. Doing
this in turn disables gnutls' setup code. So secmem is left enabled.
This is not an issue on most distros, since they allow users to mlock
sufficient memory.
Anyway we need to fix libvirt to disable secmem, since we've blocked
gnutls' own setup from running
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:|
|: http://libvirt.org -o- http://virt-manager.org:|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/:|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
4242
days inactive
4249
days old
8 comments
3 participants
participants (3)
-
Daniel P. Berrange -
Michal Privoznik -
SHREE DUTH AWASTHI