2:53 a.m.
On Thu, Mar 26, 2015 at 02:55:04PM +0800, 梅磊 wrote:
Thanks for your reply, Martin, I have read the coredump, but I am not
familiar with libvirt, so could you help me to figure it out why it crash?
Attachment is the coredump gdb output and all the generate files when libvirtd stop.
I forgot to ask you to install debuginfo packages to further
investigate. But looking at the log only netcf-debuginfo is needed.
Could you install it and then run the "t a a bt f" again, please?
From the current logs I can only guess that there is some network XML
mangled and netcf can't read it. When you're power went out, as you
said, is it possible you were updating something in the meantime? Try
running "rpm -Va" to see if there's something broken in the system.
And Do you think this issue is the same issue with the below root
cause?
https://bugzilla.redhat.com/show_bug.cgi?id=1179981
It is not, that is caused by something else.
-Best Regards,
Andy
-----邮件原件-----
发件人: Martin Kletzander [mailto:mkletzan@redhat.com]
发送时间: 24 March 2015 21:20
收件人: 梅磊
抄送: libvirt-users(a)redhat.com
主题: Re: [libvirt-users] libvirtd can't start
On Tue, Mar 24, 2015 at 04:29:07PM +0800, 梅磊 wrote:
>Hi experts,
> The libvirtd can’t start on my server after the server interruption of
>power supply, the status is below:
>
>[root@openstack3 libvirt]# service libvirtd status Redirecting to
>/bin/systemctl status libvirtd.service libvirtd.service -
>Virtualization daemon
> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
> Active: failed (Result: signal) since Tue 2015-03-24 16:10:31 CST;
>7min ago
> Process: 27372 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
>(code=killed,
>signal=SEGV)
> Main PID: 27372 (code=killed, signal=SEGV)
>
>Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000:
>27383: debug : virStateInitialize:847 : Running global init for netcf
>state driver Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24
>08:10:31.413+0000:
>27383: debug : virObjectNew:199 : OBJECT_NEW: obj=0x7f73f0080d00
>classname=virNetcfDriverState Mar 24 16:10:31 openstack3
>libvirtd[27372]: ====== end of log ===== Mar 24 16:10:31 openstack3
>systemd[1]: libvirtd.service: main process exited, code=killed,
>status=11/SEGV Mar 24 16:10:31 openstack3 systemd[1]: Unit
>libvirtd.service entered failed state.
>
>And the attachment is the log after restart the libvirtd.
>
>After long time search on google, I didn’t get any useful information,
>so anyone can give me some hint on this issue?
>
What did you search for?
According to the log:
abrt-hook-ccpp: Saved core dump of pid 28928 (/usr/sbin/libvirtd) to
/var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928 (97218560 bytes)
libvirtd has segfaulted and you have the coredump. Use gdb to open
it:
# gdb -c /var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928
and then look at what happened, if you can't make anything of it, post it as an
attachment here or create a bug from it:
(gdb) bt
(gdb) t a a bt full
>-BR,
>Andy
>
>_______________________________________________
>libvirt-users mailing list
>libvirt-users(a)redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users
4:31 a.m.
New subject: [libvirt-users] 答复: 答复: libvirtd can't start
Hi Martin:
Actually, I have installed all the debuginfo for this coredump file. But the netcf
debuginfo report "mismatch" as below:
warning: the debug information found in
"/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match
"/lib64/libnetcf.so.1" (CRC mismatch).
So I ignore this warning before. You can check what I did on the attachment
"netcf_debuginfo_install.log".
Another attachment is the result of "rpm -Va" to check all the system config
files status, I didn't find anything wrong.
Thanks for your help!
-Best Regards,
Andy
-----邮件原件-----
发件人: Martin Kletzander [mailto:mkletzan@redhat.com]
发送时间: 26 March 2015 15:54
收件人: 梅磊
抄送: libvirt-users(a)redhat.com
主题: Re: 答复: [libvirt-users] libvirtd can't start
On Thu, Mar 26, 2015 at 02:55:04PM +0800, 梅磊 wrote:
Thanks for your reply, Martin, I have read the coredump, but I am not
familiar with libvirt, so could you help me to figure it out why it crash?
Attachment is the coredump gdb output and all the generate files when libvirtd stop.
I forgot to ask you to install debuginfo packages to further investigate. But looking at
the log only netcf-debuginfo is needed.
Could you install it and then run the "t a a bt f" again, please?
From the current logs I can only guess that there is some network XML
mangled and netcf can't read it. When you're power went out, as you said, is it
possible you were updating something in the meantime? Try running "rpm -Va" to
see if there's something broken in the system.
And Do you think this issue is the same issue with the below root
cause?
https://bugzilla.redhat.com/show_bug.cgi?id=1179981
It is not, that is caused by something else.
-Best Regards,
Andy
-----邮件原件-----
发件人: Martin Kletzander [mailto:mkletzan@redhat.com]
发送时间: 24 March 2015 21:20
收件人: 梅磊
抄送: libvirt-users(a)redhat.com
主题: Re: [libvirt-users] libvirtd can't start
On Tue, Mar 24, 2015 at 04:29:07PM +0800, 梅磊 wrote:
>Hi experts,
> The libvirtd can’t start on my server after the server interruption
>of power supply, the status is below:
>
>[root@openstack3 libvirt]# service libvirtd status Redirecting to
>/bin/systemctl status libvirtd.service libvirtd.service -
>Virtualization daemon
> Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled)
> Active: failed (Result: signal) since Tue 2015-03-24 16:10:31 CST;
>7min ago
> Process: 27372 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS
>(code=killed,
>signal=SEGV)
> Main PID: 27372 (code=killed, signal=SEGV)
>
>Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000:
>27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31
>openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000:
>27383: debug : virStateInitialize:847 : Running global init for netcf
>state driver Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24
>08:10:31.413+0000:
>27383: debug : virObjectNew:199 : OBJECT_NEW: obj=0x7f73f0080d00
>classname=virNetcfDriverState Mar 24 16:10:31 openstack3
>libvirtd[27372]: ====== end of log ===== Mar 24 16:10:31 openstack3
>systemd[1]: libvirtd.service: main process exited, code=killed,
>status=11/SEGV Mar 24 16:10:31 openstack3 systemd[1]: Unit
>libvirtd.service entered failed state.
>
>And the attachment is the log after restart the libvirtd.
>
>After long time search on google, I didn’t get any useful information,
>so anyone can give me some hint on this issue?
>
What did you search for?
According to the log:
abrt-hook-ccpp: Saved core dump of pid 28928 (/usr/sbin/libvirtd) to
/var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928 (97218560 bytes)
libvirtd has segfaulted and you have the coredump. Use gdb to open
it:
# gdb -c /var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928
and then look at what happened, if you can't make anything of it, post it as an
attachment here or create a bug from it:
(gdb) bt
(gdb) t a a bt full
>-BR,
>Andy
>
>_______________________________________________
>libvirt-users mailing list
>libvirt-users(a)redhat.com
>https://www.redhat.com/mailman/listinfo/libvirt-users
5:34 p.m.
New subject: [libvirt-users] 答复: 答复: libvirtd can't start
On 03/26/2015 05:31 AM, 梅磊 wrote:
Hi Martin:
Actually, I have installed all the debuginfo for this coredump file. But the netcf
debuginfo report "mismatch" as below:
warning: the debug information found in
"/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match
"/lib64/libnetcf.so.1" (CRC mismatch).
So I ignore this warning before. You can check what I did on the attachment
"netcf_debuginfo_install.log".
Core was generated by `/usr/sbin/libvirtd --listen'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1
Missing separate debuginfos, use: debuginfo-install netcf-libs-0.2.4-1.el6.x86_64
Notice it says that you need the debuginfo for netcf-0.2.4-1.el6
yum install netcf-debuginfo
Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks, priorities
Loading mirror speeds from cached hostfile
* epel: ftp.sjtu.edu.cn
* epel-debuginfo: ftp.sjtu.edu.cn
60 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed
--> Finished Dependency Resolution
And here it says that what you installed is netcf-debuginfo-0.2.3-8.el7
So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got
packages mixed and matched in a way that isn't going to work correctly
(I'm surprised that yum would even allow it; usually it doesn't).
Can you tell us the output of "uname -a; rpm -qa | grep
'\(libvirt\|netcf\|libnl\)' | sort" ?
(and also it would be useful to know how you got a mix of el6 and el7
packages on your system)
1:47 a.m.
New subject: [libvirt-users] 答复: 答复: 答复: libvirtd can't start
Laine, you are right, some packages mixed:
[root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort
libnl-1.1.4-3.el7.x86_64
libnl3-3.2.21-7.el6.x86_64
libnl3-cli-3.2.21-7.el6.x86_64
libnl3-debuginfo-3.2.21-7.el6.x86_64
libnl-debuginfo-1.1.4-3.el7.x86_64
libnl-devel-1.1.4-3.el7.x86_64
libvirt-client-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-config-network-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-interface-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-lxc-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-network-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-secret-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-storage-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-kvm-1.1.1-29.el7_0.3.x86_64
libvirt-debuginfo-1.1.1-29.el7_0.3.x86_64
libvirt-devel-1.1.1-29.el7_0.3.x86_64
libvirt-docs-1.1.1-29.el7_0.3.x86_64
libvirt-gconfig-0.1.7-3.el7.x86_64
libvirt-glib-0.1.7-3.el7.x86_64
libvirt-gobject-0.1.7-3.el7.x86_64
libvirt-java-0.4.9-4.el7.noarch
libvirt-java-devel-0.4.9-4.el7.noarch
libvirt-python-1.1.1-29.el7_0.3.x86_64
netcf-debuginfo-0.2.3-8.el7.x86_64
netcf-libs-0.2.4-1.el6.x86_64
[root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort | grep
el6
libnl3-3.2.21-7.el6.x86_64
libnl3-cli-3.2.21-7.el6.x86_64
libnl3-debuginfo-3.2.21-7.el6.x86_64
netcf-libs-0.2.4-1.el6.x86_64
[root@openstack3 ~]#
[root@openstack3 ~]# uname -a
Linux openstack3 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
The distribution is centos 7.
I tried to update the package to el7 version, but due to the package's el6 version is
newer than the el7 version, so I can't update it.
Then I want to remove these packages and reinstall it, but a lot of dependencies, do you
think I can do it?
-BR,
Andy
-----邮件原件-----
发件人: sendmail [mailto:justsendmailnothingelse@gmail.com] 代表 Laine Stump
发送时间: 27 March 2015 06:35
收件人: libvirt-users(a)redhat.com
抄送: 梅磊; 'Martin Kletzander'
主题: Re: [libvirt-users] 答复: 答复: libvirtd can't start
On 03/26/2015 05:31 AM, 梅磊 wrote:
Hi Martin:
Actually, I have installed all the debuginfo for this coredump file. But the netcf
debuginfo report "mismatch" as below:
warning: the debug information found in
"/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match
"/lib64/libnetcf.so.1" (CRC mismatch).
So I ignore this warning before. You can check what I did on the attachment
"netcf_debuginfo_install.log".
Core was generated by `/usr/sbin/libvirtd --listen'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1 Missing
separate debuginfos, use: debuginfo-install
netcf-libs-0.2.4-1.el6.x86_64
Notice it says that you need the debuginfo for netcf-0.2.4-1.el6
yum install netcf-debuginfo
Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks,
priorities Loading mirror speeds from cached hostfile
* epel: ftp.sjtu.edu.cn
* epel-debuginfo: ftp.sjtu.edu.cn
60 packages excluded due to repository priority protections Resolving
Dependencies
--> Running transaction check
---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed
--> Finished Dependency Resolution
And here it says that what you installed is netcf-debuginfo-0.2.3-8.el7
So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got packages
mixed and matched in a way that isn't going to work correctly (I'm surprised that
yum would even allow it; usually it doesn't).
Can you tell us the output of "uname -a; rpm -qa | grep
'\(libvirt\|netcf\|libnl\)' | sort" ?
(and also it would be useful to know how you got a mix of el6 and el7 packages on your
system)
7:17 a.m.
New subject: [libvirt-users] 答复: 答复: 答复: 答复: libvirtd can't start
After remove all the mixed packages and reinstall them, this issue has been fixed.
Thanks for your help!
-Best Regards,
Andy
-----邮件原件-----
发件人: libvirt-users-bounces(a)redhat.com [mailto:libvirt-users-bounces@redhat.com] 代表 梅磊
发送时间: 27 March 2015 14:47
收件人: 'Laine Stump'; libvirt-users(a)redhat.com
抄送: 'Martin Kletzander'
主题: [libvirt-users] 答复: 答复: 答复: libvirtd can't start
Laine, you are right, some packages mixed:
[root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort
libnl-1.1.4-3.el7.x86_64
libnl3-3.2.21-7.el6.x86_64
libnl3-cli-3.2.21-7.el6.x86_64
libnl3-debuginfo-3.2.21-7.el6.x86_64
libnl-debuginfo-1.1.4-3.el7.x86_64
libnl-devel-1.1.4-3.el7.x86_64
libvirt-client-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-config-network-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-interface-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-lxc-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-network-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-secret-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-driver-storage-1.1.1-29.el7_0.3.x86_64
libvirt-daemon-kvm-1.1.1-29.el7_0.3.x86_64
libvirt-debuginfo-1.1.1-29.el7_0.3.x86_64
libvirt-devel-1.1.1-29.el7_0.3.x86_64
libvirt-docs-1.1.1-29.el7_0.3.x86_64
libvirt-gconfig-0.1.7-3.el7.x86_64
libvirt-glib-0.1.7-3.el7.x86_64
libvirt-gobject-0.1.7-3.el7.x86_64
libvirt-java-0.4.9-4.el7.noarch
libvirt-java-devel-0.4.9-4.el7.noarch
libvirt-python-1.1.1-29.el7_0.3.x86_64
netcf-debuginfo-0.2.3-8.el7.x86_64
netcf-libs-0.2.4-1.el6.x86_64
[root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort | grep
el6
libnl3-3.2.21-7.el6.x86_64
libnl3-cli-3.2.21-7.el6.x86_64
libnl3-debuginfo-3.2.21-7.el6.x86_64
netcf-libs-0.2.4-1.el6.x86_64
[root@openstack3 ~]#
[root@openstack3 ~]# uname -a
Linux openstack3 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64
x86_64 x86_64 GNU/Linux
The distribution is centos 7.
I tried to update the package to el7 version, but due to the package's el6 version is
newer than the el7 version, so I can't update it.
Then I want to remove these packages and reinstall it, but a lot of dependencies, do you
think I can do it?
-BR,
Andy
-----邮件原件-----
发件人: sendmail [mailto:justsendmailnothingelse@gmail.com] 代表 Laine Stump
发送时间: 27 March 2015 06:35
收件人: libvirt-users(a)redhat.com
抄送: 梅磊; 'Martin Kletzander'
主题: Re: [libvirt-users] 答复: 答复: libvirtd can't start
On 03/26/2015 05:31 AM, 梅磊 wrote:
Hi Martin:
Actually, I have installed all the debuginfo for this coredump file. But the netcf
debuginfo report "mismatch" as below:
warning: the debug information found in
"/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match
"/lib64/libnetcf.so.1" (CRC mismatch).
So I ignore this warning before. You can check what I did on the attachment
"netcf_debuginfo_install.log".
Core was generated by `/usr/sbin/libvirtd --listen'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1 Missing
separate debuginfos, use: debuginfo-install
netcf-libs-0.2.4-1.el6.x86_64
Notice it says that you need the debuginfo for netcf-0.2.4-1.el6
yum install netcf-debuginfo
Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks,
priorities Loading mirror speeds from cached hostfile
* epel: ftp.sjtu.edu.cn
* epel-debuginfo: ftp.sjtu.edu.cn
60 packages excluded due to repository priority protections Resolving
Dependencies
--> Running transaction check
---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed
--> Finished Dependency Resolution
And here it says that what you installed is netcf-debuginfo-0.2.3-8.el7
So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got packages
mixed and matched in a way that isn't going to work correctly (I'm surprised that
yum would even allow it; usually it doesn't).
Can you tell us the output of "uname -a; rpm -qa | grep
'\(libvirt\|netcf\|libnl\)' | sort" ?
(and also it would be useful to know how you got a mix of el6 and el7 packages on your
system)
_______________________________________________
libvirt-users mailing list
libvirt-users(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users
3528
days inactive
3529
days old
4 comments
3 participants
participants (3)
-
Laine Stump -
Martin Kletzander -
梅磊