[libvirt-users] libvirtd segfault when using oVirt 4.1 with graphic console - CentOS7.3
hello, I am getting such error: libvirtd[27218]: segfault at 0 ip 00007f4940725721 sp 00007f4930711740 error 4 in libvirt.so.0.2000.0[7f4940678000+353000] when I am trying to start VM with graphic spice/vnc console - in headless mode(without graphic console) it is running I noticed this after update oVirt 4.0 to oVirt 4.1 however I noticed that also libvirtd and related packages were upgraded from 1.2.7 to 2.0.0: libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64 I am running up to date CentOS7.3 with 3.10.0-514.16.1.el7.x86_64 I have same issue with SELinux and without SELinux(checked after reboot with permissive mode) I tried to get information about this issue from oVirt team, but after mail conversation both me and oVirt team thinks that it might be issue in libvirtd in below link I am putting xmls generated by vdsm which are passing to the libvirtd to run the VMs first one is from vdsm log and second one is extracted from dump after segfault of libvirtd https://paste.fedoraproject.org/paste/eqpe8Byu2l-3SRdXc6LTLl5M1UNdIGYhyRLivL... Regards, Rafal Wojciechowski
On Thu, Apr 20, 2017 at 07:36:42AM +0200, Rafał Wojciechowski wrote:
hello,
I am getting such error: libvirtd[27218]: segfault at 0 ip 00007f4940725721 sp 00007f4930711740 error 4 in libvirt.so.0.2000.0[7f4940678000+353000]
when I am trying to start VM with graphic spice/vnc console - in headless mode(without graphic console) it is running I noticed this after update oVirt 4.0 to oVirt 4.1 however I noticed that also libvirtd and related packages were upgraded from 1.2.7 to 2.0.0: libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
I am running up to date CentOS7.3 with 3.10.0-514.16.1.el7.x86_64 I have same issue with SELinux and without SELinux(checked after reboot with permissive mode)
I tried to get information about this issue from oVirt team, but after mail conversation both me and oVirt team thinks that it might be issue in libvirtd
in below link I am putting xmls generated by vdsm which are passing to the libvirtd to run the VMs first one is from vdsm log and second one is extracted from dump after segfault of libvirtd
https://paste.fedoraproject.org/paste/eqpe8Byu2l-3SRdXc6LTLl5M1UNdIGYhyRLivL...
Hi, could you please provide core backtrace? I've tried to reproduce it on CentOS 7.3 with the same graphics configuration but the guest started and there was no segfault. Thanks, Pavel
hello, I attached core dump - not sure if it was what you have asked for I am rather just admin not developer :) Regards, Rafal Wojciechowski W dniu 20.04.2017 o 16:44, Pavel Hrdina pisze:
On Thu, Apr 20, 2017 at 07:36:42AM +0200, Rafał Wojciechowski wrote:
hello,
I am getting such error: libvirtd[27218]: segfault at 0 ip 00007f4940725721 sp 00007f4930711740 error 4 in libvirt.so.0.2000.0[7f4940678000+353000]
when I am trying to start VM with graphic spice/vnc console - in headless mode(without graphic console) it is running I noticed this after update oVirt 4.0 to oVirt 4.1 however I noticed that also libvirtd and related packages were upgraded from 1.2.7 to 2.0.0: libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
I am running up to date CentOS7.3 with 3.10.0-514.16.1.el7.x86_64 I have same issue with SELinux and without SELinux(checked after reboot with permissive mode)
I tried to get information about this issue from oVirt team, but after mail conversation both me and oVirt team thinks that it might be issue in libvirtd
in below link I am putting xmls generated by vdsm which are passing to the libvirtd to run the VMs first one is from vdsm log and second one is extracted from dump after segfault of libvirtd
https://paste.fedoraproject.org/paste/eqpe8Byu2l-3SRdXc6LTLl5M1UNdIGYhyRLivL... Hi,
could you please provide core backtrace? I've tried to reproduce it on CentOS 7.3 with the same graphics configuration but the guest started and there was no segfault.
Thanks,
Pavel
On Thu, Apr 20, 2017 at 06:10:04PM +0200, Rafał Wojciechowski wrote:
hello,
I attached core dump - not sure if it was what you have asked for I am rather just admin not developer :)
Regards,
Rafal Wojciechowski
The attached core send in private mail helped to figure out where the crash happened. Backtrace: Thread 1 (Thread 0x7f194b99d700 (LWP 5631)): #0 virNetDevGetifaddrsAddress (addr=0x7f194b99c7c0, ifname=0x7f193400e2b0 "ovirtmgmt") at util/virnetdevip.c:738 #1 virNetDevIPAddrGet (ifname=0x7f193400e2b0 "ovirtmgmt", addr=addr@entry=0x7f194b99c7c0) at util/virnetdevip.c:795 #2 0x00007f19467800d6 in networkGetNetworkAddress (netname=<optimized out>, netaddr=netaddr@entry=0x7f1924013f18) at network/bridge_driver.c:4780 #3 0x00007f193e43a33c in qemuProcessGraphicsSetupNetworkAddress (listenAddr=0x7f19340f7650 "127.0.0.1", glisten=0x7f1924013f10) at qemu/qemu_process.c:4062 #4 qemuProcessGraphicsSetupListen (vm=<optimized out>, graphics=0x7f1924014f10, cfg=0x7f1934119f00) at qemu/qemu_process.c:4133 #5 qemuProcessSetupGraphics (flags=17, vm=0x7f19240155d0, driver=0x7f193411f1d0) at qemu/qemu_process.c:4196 #6 qemuProcessPrepareDomain (conn=conn@entry=0x7f192c00ab50, driver=driver@entry=0x7f193411f1d0, vm=vm@entry=0x7f19240155d0, flags=flags@entry=17) at qemu/qemu_process.c:4969 #7 0x00007f193e4417c0 in qemuProcessStart (conn=conn@entry=0x7f192c00ab50, driver=driver@entry=0x7f193411f1d0, vm=0x7f19240155d0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_START, migrateFrom=migrateFrom@entry=0x0, migrateFd=migrateFd@entry=-1, migratePath=migratePath@entry=0x0, snapshot=snapshot@entry=0x0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=17, flags@entry=1) at qemu/qemu_process.c:5553 #8 0x00007f193e490030 in qemuDomainCreateXML (conn=0x7f192c00ab50, xml=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:1774 #9 0x00007f195aa6af81 in virDomainCreateXML (conn=0x7f192c00ab50, xmlDesc=0x7f1924003f00 "<?xml version='1.0' encoding='UTF-8'?>\n<domain xmlns:ovirt=\"http://ovirt.org/vm/tune/1.0\" type=\"kvm\">\n <name>zzzzz</name>\n <uuid>7d14f6e2-978b-47a5-875d-be5d6b28af2c</uuid>\n <memory>1048576</"..., flags=0) at libvirt-domain.c:180 #10 0x00007f195b6e8dfa in remoteDispatchDomainCreateXML (server=0x7f195bc4eb40, msg=0x7f195bc73ee0, ret=0x7f19240038b0, args=0x7f1924003a70, rerr=0x7f194b99cc50, client=0x7f195bc71e90) at remote_dispatch.h:4257 #11 remoteDispatchDomainCreateXMLHelper (server=0x7f195bc4eb40, client=0x7f195bc71e90, msg=0x7f195bc73ee0, rerr=0x7f194b99cc50, args=0x7f1924003a70, ret=0x7f19240038b0) at remote_dispatch.h:4235 #12 0x00007f195aaea002 in virNetServerProgramDispatchCall (msg=0x7f195bc73ee0, client=0x7f195bc71e90, server=0x7f195bc4eb40, prog=0x7f195bc62fa0) at rpc/virnetserverprogram.c:437 #13 virNetServerProgramDispatch (prog=0x7f195bc62fa0, server=server@entry=0x7f195bc4eb40, client=0x7f195bc71e90, msg=0x7f195bc73ee0) at rpc/virnetserverprogram.c:307 #14 0x00007f195b6f9c6d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f195bc4eb40) at rpc/virnetserver.c:148 #15 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f195bc4eb40) at rpc/virnetserver.c:169 #16 0x00007f195a9d6d41 in virThreadPoolWorker (opaque=opaque@entry=0x7f195bc44160) at util/virthreadpool.c:167 #17 0x00007f195a9d60c8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #18 0x00007f1957ff9dc5 in start_thread (arg=0x7f194b99d700) at pthread_create.c:308 #19 0x00007f1957d2873d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 I'll send a patch to upstream libvirt to fix this crash. However it can take a while to get it back to CentOS/RHEL. The source of this crash is that you have a "tun0" network interface without IP address and that interface is checked before "ovirtmgmt" and it causes the crash. You can workaround it by removing the "tun0" interface if it doesn't have any IP address. Pavel
W dniu 20.04.2017 o 16:44, Pavel Hrdina pisze:
On Thu, Apr 20, 2017 at 07:36:42AM +0200, Rafał Wojciechowski wrote:
hello,
I am getting such error: libvirtd[27218]: segfault at 0 ip 00007f4940725721 sp 00007f4930711740 error 4 in libvirt.so.0.2000.0[7f4940678000+353000]
when I am trying to start VM with graphic spice/vnc console - in headless mode(without graphic console) it is running I noticed this after update oVirt 4.0 to oVirt 4.1 however I noticed that also libvirtd and related packages were upgraded from 1.2.7 to 2.0.0: libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
I am running up to date CentOS7.3 with 3.10.0-514.16.1.el7.x86_64 I have same issue with SELinux and without SELinux(checked after reboot with permissive mode)
I tried to get information about this issue from oVirt team, but after mail conversation both me and oVirt team thinks that it might be issue in libvirtd
in below link I am putting xmls generated by vdsm which are passing to the libvirtd to run the VMs first one is from vdsm log and second one is extracted from dump after segfault of libvirtd
https://paste.fedoraproject.org/paste/eqpe8Byu2l-3SRdXc6LTLl5M1UNdIGYhyRLivL... Hi,
could you please provide core backtrace? I've tried to reproduce it on CentOS 7.3 with the same graphics configuration but the guest started and there was no segfault.
Thanks,
Pavel
Hi, I can confirm. without tun0 after reboot it is working fine. it was not so obvious. thanks Regards, Rafal Wojciechowski W dniu 21.04.2017 o 10:53, Pavel Hrdina pisze:
On Thu, Apr 20, 2017 at 06:10:04PM +0200, Rafał Wojciechowski wrote:
hello,
I attached core dump - not sure if it was what you have asked for I am rather just admin not developer :)
Regards,
Rafal Wojciechowski The attached core send in private mail helped to figure out where the crash happened. Backtrace:
Thread 1 (Thread 0x7f194b99d700 (LWP 5631)): #0 virNetDevGetifaddrsAddress (addr=0x7f194b99c7c0, ifname=0x7f193400e2b0 "ovirtmgmt") at util/virnetdevip.c:738 #1 virNetDevIPAddrGet (ifname=0x7f193400e2b0 "ovirtmgmt", addr=addr@entry=0x7f194b99c7c0) at util/virnetdevip.c:795 #2 0x00007f19467800d6 in networkGetNetworkAddress (netname=<optimized out>, netaddr=netaddr@entry=0x7f1924013f18) at network/bridge_driver.c:4780 #3 0x00007f193e43a33c in qemuProcessGraphicsSetupNetworkAddress (listenAddr=0x7f19340f7650 "127.0.0.1", glisten=0x7f1924013f10) at qemu/qemu_process.c:4062 #4 qemuProcessGraphicsSetupListen (vm=<optimized out>, graphics=0x7f1924014f10, cfg=0x7f1934119f00) at qemu/qemu_process.c:4133 #5 qemuProcessSetupGraphics (flags=17, vm=0x7f19240155d0, driver=0x7f193411f1d0) at qemu/qemu_process.c:4196 #6 qemuProcessPrepareDomain (conn=conn@entry=0x7f192c00ab50, driver=driver@entry=0x7f193411f1d0, vm=vm@entry=0x7f19240155d0, flags=flags@entry=17) at qemu/qemu_process.c:4969 #7 0x00007f193e4417c0 in qemuProcessStart (conn=conn@entry=0x7f192c00ab50, driver=driver@entry=0x7f193411f1d0, vm=0x7f19240155d0, asyncJob=asyncJob@entry=QEMU_ASYNC_JOB_START, migrateFrom=migrateFrom@entry=0x0, migrateFd=migrateFd@entry=-1, migratePath=migratePath@entry=0x0, snapshot=snapshot@entry=0x0, vmop=vmop@entry=VIR_NETDEV_VPORT_PROFILE_OP_CREATE, flags=17, flags@entry=1) at qemu/qemu_process.c:5553 #8 0x00007f193e490030 in qemuDomainCreateXML (conn=0x7f192c00ab50, xml=<optimized out>, flags=<optimized out>) at qemu/qemu_driver.c:1774 #9 0x00007f195aa6af81 in virDomainCreateXML (conn=0x7f192c00ab50, xmlDesc=0x7f1924003f00 "<?xml version='1.0' encoding='UTF-8'?>\n<domain xmlns:ovirt=\"http://ovirt.org/vm/tune/1.0\" type=\"kvm\">\n <name>zzzzz</name>\n <uuid>7d14f6e2-978b-47a5-875d-be5d6b28af2c</uuid>\n <memory>1048576</"..., flags=0) at libvirt-domain.c:180 #10 0x00007f195b6e8dfa in remoteDispatchDomainCreateXML (server=0x7f195bc4eb40, msg=0x7f195bc73ee0, ret=0x7f19240038b0, args=0x7f1924003a70, rerr=0x7f194b99cc50, client=0x7f195bc71e90) at remote_dispatch.h:4257 #11 remoteDispatchDomainCreateXMLHelper (server=0x7f195bc4eb40, client=0x7f195bc71e90, msg=0x7f195bc73ee0, rerr=0x7f194b99cc50, args=0x7f1924003a70, ret=0x7f19240038b0) at remote_dispatch.h:4235 #12 0x00007f195aaea002 in virNetServerProgramDispatchCall (msg=0x7f195bc73ee0, client=0x7f195bc71e90, server=0x7f195bc4eb40, prog=0x7f195bc62fa0) at rpc/virnetserverprogram.c:437 #13 virNetServerProgramDispatch (prog=0x7f195bc62fa0, server=server@entry=0x7f195bc4eb40, client=0x7f195bc71e90, msg=0x7f195bc73ee0) at rpc/virnetserverprogram.c:307 #14 0x00007f195b6f9c6d in virNetServerProcessMsg (msg=<optimized out>, prog=<optimized out>, client=<optimized out>, srv=0x7f195bc4eb40) at rpc/virnetserver.c:148 #15 virNetServerHandleJob (jobOpaque=<optimized out>, opaque=0x7f195bc4eb40) at rpc/virnetserver.c:169 #16 0x00007f195a9d6d41 in virThreadPoolWorker (opaque=opaque@entry=0x7f195bc44160) at util/virthreadpool.c:167 #17 0x00007f195a9d60c8 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #18 0x00007f1957ff9dc5 in start_thread (arg=0x7f194b99d700) at pthread_create.c:308 #19 0x00007f1957d2873d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
I'll send a patch to upstream libvirt to fix this crash. However it can take a while to get it back to CentOS/RHEL. The source of this crash is that you have a "tun0" network interface without IP address and that interface is checked before "ovirtmgmt" and it causes the crash. You can workaround it by removing the "tun0" interface if it doesn't have any IP address.
Pavel
W dniu 20.04.2017 o 16:44, Pavel Hrdina pisze:
On Thu, Apr 20, 2017 at 07:36:42AM +0200, Rafał Wojciechowski wrote:
hello,
I am getting such error: libvirtd[27218]: segfault at 0 ip 00007f4940725721 sp 00007f4930711740 error 4 in libvirt.so.0.2000.0[7f4940678000+353000]
when I am trying to start VM with graphic spice/vnc console - in headless mode(without graphic console) it is running I noticed this after update oVirt 4.0 to oVirt 4.1 however I noticed that also libvirtd and related packages were upgraded from 1.2.7 to 2.0.0: libvirt-daemon-kvm-2.0.0-10.el7_3.5.x86_64 libvirt-daemon-driver-qemu-2.0.0-10.el7_3.5.x86_64
I am running up to date CentOS7.3 with 3.10.0-514.16.1.el7.x86_64 I have same issue with SELinux and without SELinux(checked after reboot with permissive mode)
I tried to get information about this issue from oVirt team, but after mail conversation both me and oVirt team thinks that it might be issue in libvirtd
in below link I am putting xmls generated by vdsm which are passing to the libvirtd to run the VMs first one is from vdsm log and second one is extracted from dump after segfault of libvirtd
https://paste.fedoraproject.org/paste/eqpe8Byu2l-3SRdXc6LTLl5M1UNdIGYhyRLivL... Hi,
could you please provide core backtrace? I've tried to reproduce it on CentOS 7.3 with the same graphics configuration but the guest started and there was no segfault.
Thanks,
Pavel
participants (2)
-
Pavel Hrdina -
Rafał Wojciechowski