Re: [libvirt-users] 答复: libvirtd can't start
On Thu, Mar 26, 2015 at 02:55:04PM +0800, 梅磊 wrote:
Thanks for your reply, Martin, I have read the coredump, but I am not familiar with libvirt, so could you help me to figure it out why it crash? Attachment is the coredump gdb output and all the generate files when libvirtd stop.
I forgot to ask you to install debuginfo packages to further investigate. But looking at the log only netcf-debuginfo is needed. Could you install it and then run the "t a a bt f" again, please? From the current logs I can only guess that there is some network XML mangled and netcf can't read it. When you're power went out, as you said, is it possible you were updating something in the meantime? Try running "rpm -Va" to see if there's something broken in the system.
And Do you think this issue is the same issue with the below root cause? https://bugzilla.redhat.com/show_bug.cgi?id=1179981
It is not, that is caused by something else.
-Best Regards, Andy
-----邮件原件----- 发件人: Martin Kletzander [mailto:mkletzan@redhat.com] 发送时间: 24 March 2015 21:20 收件人: 梅磊 抄送: libvirt-users@redhat.com 主题: Re: [libvirt-users] libvirtd can't start
On Tue, Mar 24, 2015 at 04:29:07PM +0800, 梅磊 wrote:
Hi experts, The libvirtd can’t start on my server after the server interruption of power supply, the status is below:
[root@openstack3 libvirt]# service libvirtd status Redirecting to /bin/systemctl status libvirtd.service libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled) Active: failed (Result: signal) since Tue 2015-03-24 16:10:31 CST; 7min ago Process: 27372 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=killed, signal=SEGV) Main PID: 27372 (code=killed, signal=SEGV)
Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: 27383: debug : virStateInitialize:847 : Running global init for netcf state driver Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: 27383: debug : virObjectNew:199 : OBJECT_NEW: obj=0x7f73f0080d00 classname=virNetcfDriverState Mar 24 16:10:31 openstack3 libvirtd[27372]: ====== end of log ===== Mar 24 16:10:31 openstack3 systemd[1]: libvirtd.service: main process exited, code=killed, status=11/SEGV Mar 24 16:10:31 openstack3 systemd[1]: Unit libvirtd.service entered failed state.
And the attachment is the log after restart the libvirtd.
After long time search on google, I didn’t get any useful information, so anyone can give me some hint on this issue?
What did you search for?
According to the log: abrt-hook-ccpp: Saved core dump of pid 28928 (/usr/sbin/libvirtd) to /var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928 (97218560 bytes)
libvirtd has segfaulted and you have the coredump. Use gdb to open it: # gdb -c /var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928
and then look at what happened, if you can't make anything of it, post it as an attachment here or create a bug from it:
(gdb) bt (gdb) t a a bt full
-BR, Andy
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Hi Martin: Actually, I have installed all the debuginfo for this coredump file. But the netcf debuginfo report "mismatch" as below: warning: the debug information found in "/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match "/lib64/libnetcf.so.1" (CRC mismatch). So I ignore this warning before. You can check what I did on the attachment "netcf_debuginfo_install.log". Another attachment is the result of "rpm -Va" to check all the system config files status, I didn't find anything wrong. Thanks for your help! -Best Regards, Andy -----邮件原件----- 发件人: Martin Kletzander [mailto:mkletzan@redhat.com] 发送时间: 26 March 2015 15:54 收件人: 梅磊 抄送: libvirt-users@redhat.com 主题: Re: 答复: [libvirt-users] libvirtd can't start On Thu, Mar 26, 2015 at 02:55:04PM +0800, 梅磊 wrote:
Thanks for your reply, Martin, I have read the coredump, but I am not familiar with libvirt, so could you help me to figure it out why it crash? Attachment is the coredump gdb output and all the generate files when libvirtd stop.
I forgot to ask you to install debuginfo packages to further investigate. But looking at the log only netcf-debuginfo is needed. Could you install it and then run the "t a a bt f" again, please?
From the current logs I can only guess that there is some network XML mangled and netcf can't read it. When you're power went out, as you said, is it possible you were updating something in the meantime? Try running "rpm -Va" to see if there's something broken in the system.
And Do you think this issue is the same issue with the below root cause? https://bugzilla.redhat.com/show_bug.cgi?id=1179981
It is not, that is caused by something else.
-Best Regards, Andy
-----邮件原件----- 发件人: Martin Kletzander [mailto:mkletzan@redhat.com] 发送时间: 24 March 2015 21:20 收件人: 梅磊 抄送: libvirt-users@redhat.com 主题: Re: [libvirt-users] libvirtd can't start
On Tue, Mar 24, 2015 at 04:29:07PM +0800, 梅磊 wrote:
Hi experts, The libvirtd can’t start on my server after the server interruption of power supply, the status is below:
[root@openstack3 libvirt]# service libvirtd status Redirecting to /bin/systemctl status libvirtd.service libvirtd.service - Virtualization daemon Loaded: loaded (/usr/lib/systemd/system/libvirtd.service; enabled) Active: failed (Result: signal) since Tue 2015-03-24 16:10:31 CST; 7min ago Process: 27372 ExecStart=/usr/sbin/libvirtd $LIBVIRTD_ARGS (code=killed, signal=SEGV) Main PID: 27372 (code=killed, signal=SEGV)
Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.412+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: 27383: debug : virFileClose:90 : Closed fd 22 Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: 27383: debug : virStateInitialize:847 : Running global init for netcf state driver Mar 24 16:10:31 openstack3 libvirtd[27372]: 2015-03-24 08:10:31.413+0000: 27383: debug : virObjectNew:199 : OBJECT_NEW: obj=0x7f73f0080d00 classname=virNetcfDriverState Mar 24 16:10:31 openstack3 libvirtd[27372]: ====== end of log ===== Mar 24 16:10:31 openstack3 systemd[1]: libvirtd.service: main process exited, code=killed, status=11/SEGV Mar 24 16:10:31 openstack3 systemd[1]: Unit libvirtd.service entered failed state.
And the attachment is the log after restart the libvirtd.
After long time search on google, I didn’t get any useful information, so anyone can give me some hint on this issue?
What did you search for?
According to the log: abrt-hook-ccpp: Saved core dump of pid 28928 (/usr/sbin/libvirtd) to /var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928 (97218560 bytes)
libvirtd has segfaulted and you have the coredump. Use gdb to open it: # gdb -c /var/tmp/abrt/ccpp-2015-03-24-16:19:22-28928
and then look at what happened, if you can't make anything of it, post it as an attachment here or create a bug from it:
(gdb) bt (gdb) t a a bt full
-BR, Andy
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
On 03/26/2015 05:31 AM, 梅磊 wrote:
Hi Martin: Actually, I have installed all the debuginfo for this coredump file. But the netcf debuginfo report "mismatch" as below: warning: the debug information found in "/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match "/lib64/libnetcf.so.1" (CRC mismatch).
So I ignore this warning before. You can check what I did on the attachment "netcf_debuginfo_install.log".
Core was generated by `/usr/sbin/libvirtd --listen'. Program terminated with signal 11, Segmentation fault. #0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1 Missing separate debuginfos, use: debuginfo-install netcf-libs-0.2.4-1.el6.x86_64
Notice it says that you need the debuginfo for netcf-0.2.4-1.el6
yum install netcf-debuginfo Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks, priorities Loading mirror speeds from cached hostfile * epel: ftp.sjtu.edu.cn * epel-debuginfo: ftp.sjtu.edu.cn 60 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed --> Finished Dependency Resolution
And here it says that what you installed is netcf-debuginfo-0.2.3-8.el7 So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got packages mixed and matched in a way that isn't going to work correctly (I'm surprised that yum would even allow it; usually it doesn't). Can you tell us the output of "uname -a; rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort" ? (and also it would be useful to know how you got a mix of el6 and el7 packages on your system)
Laine, you are right, some packages mixed: [root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort libnl-1.1.4-3.el7.x86_64 libnl3-3.2.21-7.el6.x86_64 libnl3-cli-3.2.21-7.el6.x86_64 libnl3-debuginfo-3.2.21-7.el6.x86_64 libnl-debuginfo-1.1.4-3.el7.x86_64 libnl-devel-1.1.4-3.el7.x86_64 libvirt-client-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-config-network-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-interface-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-lxc-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-network-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-secret-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-storage-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-kvm-1.1.1-29.el7_0.3.x86_64 libvirt-debuginfo-1.1.1-29.el7_0.3.x86_64 libvirt-devel-1.1.1-29.el7_0.3.x86_64 libvirt-docs-1.1.1-29.el7_0.3.x86_64 libvirt-gconfig-0.1.7-3.el7.x86_64 libvirt-glib-0.1.7-3.el7.x86_64 libvirt-gobject-0.1.7-3.el7.x86_64 libvirt-java-0.4.9-4.el7.noarch libvirt-java-devel-0.4.9-4.el7.noarch libvirt-python-1.1.1-29.el7_0.3.x86_64 netcf-debuginfo-0.2.3-8.el7.x86_64 netcf-libs-0.2.4-1.el6.x86_64 [root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort | grep el6 libnl3-3.2.21-7.el6.x86_64 libnl3-cli-3.2.21-7.el6.x86_64 libnl3-debuginfo-3.2.21-7.el6.x86_64 netcf-libs-0.2.4-1.el6.x86_64 [root@openstack3 ~]# [root@openstack3 ~]# uname -a Linux openstack3 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux The distribution is centos 7. I tried to update the package to el7 version, but due to the package's el6 version is newer than the el7 version, so I can't update it. Then I want to remove these packages and reinstall it, but a lot of dependencies, do you think I can do it? -BR, Andy -----邮件原件----- 发件人: sendmail [mailto:justsendmailnothingelse@gmail.com] 代表 Laine Stump 发送时间: 27 March 2015 06:35 收件人: libvirt-users@redhat.com 抄送: 梅磊; 'Martin Kletzander' 主题: Re: [libvirt-users] 答复: 答复: libvirtd can't start On 03/26/2015 05:31 AM, 梅磊 wrote:
Hi Martin: Actually, I have installed all the debuginfo for this coredump file. But the netcf debuginfo report "mismatch" as below: warning: the debug information found in "/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match "/lib64/libnetcf.so.1" (CRC mismatch).
So I ignore this warning before. You can check what I did on the attachment "netcf_debuginfo_install.log".
Core was generated by `/usr/sbin/libvirtd --listen'. Program terminated with signal 11, Segmentation fault. #0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1 Missing separate debuginfos, use: debuginfo-install netcf-libs-0.2.4-1.el6.x86_64
Notice it says that you need the debuginfo for netcf-0.2.4-1.el6
yum install netcf-debuginfo Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks, priorities Loading mirror speeds from cached hostfile * epel: ftp.sjtu.edu.cn * epel-debuginfo: ftp.sjtu.edu.cn 60 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed --> Finished Dependency Resolution
And here it says that what you installed is netcf-debuginfo-0.2.3-8.el7 So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got packages mixed and matched in a way that isn't going to work correctly (I'm surprised that yum would even allow it; usually it doesn't). Can you tell us the output of "uname -a; rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort" ? (and also it would be useful to know how you got a mix of el6 and el7 packages on your system)
After remove all the mixed packages and reinstall them, this issue has been fixed. Thanks for your help! -Best Regards, Andy -----邮件原件----- 发件人: libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@redhat.com] 代表 梅磊 发送时间: 27 March 2015 14:47 收件人: 'Laine Stump'; libvirt-users@redhat.com 抄送: 'Martin Kletzander' 主题: [libvirt-users] 答复: 答复: 答复: libvirtd can't start Laine, you are right, some packages mixed: [root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort libnl-1.1.4-3.el7.x86_64 libnl3-3.2.21-7.el6.x86_64 libnl3-cli-3.2.21-7.el6.x86_64 libnl3-debuginfo-3.2.21-7.el6.x86_64 libnl-debuginfo-1.1.4-3.el7.x86_64 libnl-devel-1.1.4-3.el7.x86_64 libvirt-client-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-config-network-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-config-nwfilter-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-interface-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-lxc-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-network-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-nodedev-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-nwfilter-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-qemu-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-secret-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-driver-storage-1.1.1-29.el7_0.3.x86_64 libvirt-daemon-kvm-1.1.1-29.el7_0.3.x86_64 libvirt-debuginfo-1.1.1-29.el7_0.3.x86_64 libvirt-devel-1.1.1-29.el7_0.3.x86_64 libvirt-docs-1.1.1-29.el7_0.3.x86_64 libvirt-gconfig-0.1.7-3.el7.x86_64 libvirt-glib-0.1.7-3.el7.x86_64 libvirt-gobject-0.1.7-3.el7.x86_64 libvirt-java-0.4.9-4.el7.noarch libvirt-java-devel-0.4.9-4.el7.noarch libvirt-python-1.1.1-29.el7_0.3.x86_64 netcf-debuginfo-0.2.3-8.el7.x86_64 netcf-libs-0.2.4-1.el6.x86_64 [root@openstack3 ~]# rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort | grep el6 libnl3-3.2.21-7.el6.x86_64 libnl3-cli-3.2.21-7.el6.x86_64 libnl3-debuginfo-3.2.21-7.el6.x86_64 netcf-libs-0.2.4-1.el6.x86_64 [root@openstack3 ~]# [root@openstack3 ~]# uname -a Linux openstack3 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux The distribution is centos 7. I tried to update the package to el7 version, but due to the package's el6 version is newer than the el7 version, so I can't update it. Then I want to remove these packages and reinstall it, but a lot of dependencies, do you think I can do it? -BR, Andy -----邮件原件----- 发件人: sendmail [mailto:justsendmailnothingelse@gmail.com] 代表 Laine Stump 发送时间: 27 March 2015 06:35 收件人: libvirt-users@redhat.com 抄送: 梅磊; 'Martin Kletzander' 主题: Re: [libvirt-users] 答复: 答复: libvirtd can't start On 03/26/2015 05:31 AM, 梅磊 wrote:
Hi Martin: Actually, I have installed all the debuginfo for this coredump file. But the netcf debuginfo report "mismatch" as below: warning: the debug information found in "/usr/lib/debug/usr/lib64/libnetcf.so.1.4.0.debug" does not match "/lib64/libnetcf.so.1" (CRC mismatch).
So I ignore this warning before. You can check what I did on the attachment "netcf_debuginfo_install.log".
Core was generated by `/usr/sbin/libvirtd --listen'. Program terminated with signal 11, Segmentation fault. #0 0x00007fded87138db in ?? () from /lib64/libnetcf.so.1 Missing separate debuginfos, use: debuginfo-install netcf-libs-0.2.4-1.el6.x86_64
Notice it says that you need the debuginfo for netcf-0.2.4-1.el6
yum install netcf-debuginfo Loaded plugins: auto-update-debuginfo, fastestmirror, langpacks, priorities Loading mirror speeds from cached hostfile * epel: ftp.sjtu.edu.cn * epel-debuginfo: ftp.sjtu.edu.cn 60 packages excluded due to repository priority protections Resolving Dependencies --> Running transaction check ---> Package netcf-debuginfo.x86_64 0:0.2.3-8.el7 will be installed --> Finished Dependency Resolution
And here it says that what you installed is netcf-debuginfo-0.2.3-8.el7 So are you running CentOS 6 or CentOS 7? It sounds like maybe you've got packages mixed and matched in a way that isn't going to work correctly (I'm surprised that yum would even allow it; usually it doesn't). Can you tell us the output of "uname -a; rpm -qa | grep '\(libvirt\|netcf\|libnl\)' | sort" ? (and also it would be useful to know how you got a mix of el6 and el7 packages on your system) _______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
participants (3)
-
Laine Stump -
Martin Kletzander -
梅磊