[libvirt-users] libvirtd hangs
Hello everybody! We have a cluster of servers managed by VMmanager 5 KVM (by ispsystem). A typical node: # cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) # uname -r 3.10.0-693.11.6.el7.x86_64 # rpm -qa |grep libvirt libvirt-daemon-driver-qemu-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-disk-3.7.0-1.el7.centos.x86_64 libvirt-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-core-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-nodedev-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-lxc-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-iscsi-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-gluster-3.7.0-1.el7.centos.x86_64 libvirt-daemon-kvm-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-network-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-interface-3.7.0-1.el7.centos.x86_64 libvirt-daemon-config-network-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-rbd-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-3.7.0-1.el7.centos.x86_64 libvirt-libs-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-nwfilter-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-secret-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-mpath-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-scsi-3.7.0-1.el7.centos.x86_64 libvirt-client-3.7.0-1.el7.centos.x86_64 libvirt-daemon-3.7.0-1.el7.centos.x86_64 libvirt-daemon-config-nwfilter-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-logical-3.7.0-1.el7.centos.x86_64 # rpm -qa |grep qemu qemu-kvm-common-rhev-2.6.0-27.1.el7.centos.maros.x86_64 ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch qemu-img-rhev-2.6.0-27.1.el7.centos.maros.x86_64 qemu-kvm-rhev-2.6.0-27.1.el7.centos.maros.x86_64 # rpm -qa |grep ebtables ebtables-2.0.10-15.el7.centos.marosnet.x86_64 ebtables build with patch https://marc.info/?l=netfilter-devel&m=150728694430435 (described at https://bugzilla.redhat.com/show_bug.cgi?id=1495893) Sometimes libvirtd just hangs and stops answering for virsh requests (like `virsh list --all`). At those moments: # strace -p 5786 read(53, "\0\0\0\34", 4) = 4 read(53, "keep\0\0\0\1\0\0\0\2\0\0\0\2\0\0\0\0\0\0\0\0", 24) = 24 poll([{fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}, {fd=19, events=POLLIN}, {fd=23, events=POLLIN|POLLERR|POLLHUP}, {fd=21, events=POLLIN|POLLERR|POLLHUP}, {fd=27, events=POLLIN|POLLERR|POLLHUP}, {fd=25, events=POLLIN|POLLERR|POLLHUP}, {fd=22, events=POLLIN|POLLERR|POLLHUP}, {fd=24, events=POLLIN|POLLERR|POLLHUP}, {fd=26, events=POLLIN|POLLERR|POLLHUP}, {fd=29, events=POLLIN|POLLERR|POLLHUP}, {fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=31, events=POLLIN|POLLERR|POLLHUP}, {fd=33, events=POLLIN|POLLERR|POLLHUP}, {fd=32, events=POLLIN|POLLERR|POLLHUP}, {fd=36, events=POLLIN|POLLERR|POLLHUP}, {fd=35, events=POLLIN|POLLERR|POLLHUP}, {fd=39, events=POLLIN|POLLERR|POLLHUP}, {fd=40, events=POLLIN|POLLERR|POLLHUP}, {fd=41, events=POLLIN|POLLERR|POLLHUP}, {fd=44, events=POLLIN|POLLERR|POLLHUP}, {fd=42, events=POLLIN|POLLERR|POLLHUP}, {fd=43, events=POLLIN|POLLERR|POLLHUP}, {fd=48, events=POLLIN|POLLERR|POLLHUP}, {fd=49, events=POLLIN|POLLERR|POLLHUP}, {fd=59, events=POLLIN|POLLERR|POLLHUP}, {fd=46, events=POLLIN|POLLERR|POLLHUP}, {fd=50, events=POLLIN|POLLERR|POLLHUP}, ...], 43, 5000 # gdb -p 5786 (gdb) thread apply all bt Thread 17 (Thread 0x7f411a9d7700 (LWP 5788)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183ab0) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a9d7700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 16 (Thread 0x7f411a1d6700 (LWP 5789)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183a00) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a1d6700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 15 (Thread 0x7f41199d5700 (LWP 5790)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183950) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f41199d5700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 14 (Thread 0x7f41191d4700 (LWP 5791)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b1838a0) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f41191d4700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 13 (Thread 0x7f41189d3700 (LWP 5792)): ---Type <return> to continue, or q <return> to quit--- log_level = 3 at /etc/libvirt/libvirtd.conf doesn't help to detect the problem. Actually, libvirtd continues acting, but is not responding. It's like waiting for something... may be an answer. No zombieing, no cpu loading. This fixes the issue: rm -f /run/ebtables.lock ; killall -9 virsh; systemctl restart systemd-{journald,udevd,logind,machined} ; systemctl restart libvirtd The same situation appears with libvirt-3.2.0-14.el7_4.7.x86_64. Could anybody help to resolve this situation?
On Thu, Jan 18, 2018 at 11:30:16AM +0700, Artem Likhachev wrote:
Hello everybody!
We have a cluster of servers managed by VMmanager 5 KVM (by ispsystem).
A typical node:
# cat /etc/redhat-release CentOS Linux release 7.3.1611 (Core) # uname -r 3.10.0-693.11.6.el7.x86_64
# rpm -qa |grep libvirt libvirt-daemon-driver-qemu-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-disk-3.7.0-1.el7.centos.x86_64 libvirt-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-core-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-nodedev-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-lxc-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-iscsi-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-gluster-3.7.0-1.el7.centos.x86_64 libvirt-daemon-kvm-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-network-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-interface-3.7.0-1.el7.centos.x86_64 libvirt-daemon-config-network-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-rbd-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-3.7.0-1.el7.centos.x86_64 libvirt-libs-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-nwfilter-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-secret-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-mpath-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-scsi-3.7.0-1.el7.centos.x86_64 libvirt-client-3.7.0-1.el7.centos.x86_64 libvirt-daemon-3.7.0-1.el7.centos.x86_64 libvirt-daemon-config-nwfilter-3.7.0-1.el7.centos.x86_64 libvirt-daemon-driver-storage-logical-3.7.0-1.el7.centos.x86_64
# rpm -qa |grep qemu qemu-kvm-common-rhev-2.6.0-27.1.el7.centos.maros.x86_64 ipxe-roms-qemu-20160127-5.git6366fa7a.el7.noarch qemu-img-rhev-2.6.0-27.1.el7.centos.maros.x86_64 qemu-kvm-rhev-2.6.0-27.1.el7.centos.maros.x86_64
# rpm -qa |grep ebtables ebtables-2.0.10-15.el7.centos.marosnet.x86_64
ebtables build with patch https://marc.info/?l=netfilter-devel&m=150728694430435 (described at https://bugzilla.redhat.com/show_bug.cgi?id=1495893)
Sometimes libvirtd just hangs and stops answering for virsh requests (like `virsh list --all`).
At those moments:
# strace -p 5786 read(53, "\0\0\0\34", 4) = 4 read(53, "keep\0\0\0\1\0\0\0\2\0\0\0\2\0\0\0\0\0\0\0\0", 24) = 24 poll([{fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}, {fd=19, events=POLLIN}, {fd=23, events=POLLIN|POLLERR|POLLHUP}, {fd=21, events=POLLIN|POLLERR|POLLHUP}, {fd=27, events=POLLIN|POLLERR|POLLHUP}, {fd=25, events=POLLIN|POLLERR|POLLHUP}, {fd=22, events=POLLIN|POLLERR|POLLHUP}, {fd=24, events=POLLIN|POLLERR|POLLHUP}, {fd=26, events=POLLIN|POLLERR|POLLHUP}, {fd=29, events=POLLIN|POLLERR|POLLHUP}, {fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=31, events=POLLIN|POLLERR|POLLHUP}, {fd=33, events=POLLIN|POLLERR|POLLHUP}, {fd=32, events=POLLIN|POLLERR|POLLHUP}, {fd=36, events=POLLIN|POLLERR|POLLHUP}, {fd=35, events=POLLIN|POLLERR|POLLHUP}, {fd=39, events=POLLIN|POLLERR|POLLHUP}, {fd=40, events=POLLIN|POLLERR|POLLHUP}, {fd=41, events=POLLIN|POLLERR|POLLHUP}, {fd=44, events=POLLIN|POLLERR|POLLHUP}, {fd=42, events=POLLIN|POLLERR|POLLHUP}, {fd=43, events=POLLIN|POLLERR|POLLHUP}, {fd=48, events=POLLIN|POLLERR|POLLHUP}, {fd=49, events=POLLIN|POLLERR|POLLHUP}, {fd=59, events=POLLIN|POLLERR|POLLHUP}, {fd=46, events=POLLIN|POLLERR|POLLHUP}, {fd=50, events=POLLIN|POLLERR|POLLHUP}, ...], 43, 5000
# gdb -p 5786 (gdb) thread apply all bt
Thread 17 (Thread 0x7f411a9d7700 (LWP 5788)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183ab0) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a9d7700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 16 (Thread 0x7f411a1d6700 (LWP 5789)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183a00) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a1d6700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 15 (Thread 0x7f41199d5700 (LWP 5790)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183950) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f41199d5700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 14 (Thread 0x7f41191d4700 (LWP 5791)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b1838a0) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f41191d4700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
These are all idle workers.
Thread 13 (Thread 0x7f41189d3700 (LWP 5792)): ---Type <return> to continue, or q <return> to quit---
You omitted the backtrace of the other threads. Jan
log_level = 3 at /etc/libvirt/libvirtd.conf doesn't help to detect the problem. Actually, libvirtd continues acting, but is not responding. It's like waiting for something... may be an answer. No zombieing, no cpu loading.
This fixes the issue: rm -f /run/ebtables.lock ; killall -9 virsh; systemctl restart systemd-{journald,udevd,logind,machined} ; systemctl restart libvirtd
The same situation appears with libvirt-3.2.0-14.el7_4.7.x86_64.
Could anybody help to resolve this situation?
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Ján Tomko wrote 2018-01-18 14:27:
On Thu, Jan 18, 2018 at 11:30:16AM +0700, Artem Likhachev wrote:
# strace -p 5786 read(53, "\0\0\0\34", 4) = 4 read(53, "keep\0\0\0\1\0\0\0\2\0\0\0\2\0\0\0\0\0\0\0\0", 24) = 24 poll([{fd=5, events=POLLIN}, {fd=7, events=POLLIN}, {fd=12, events=POLLIN}, {fd=13, events=POLLIN}, {fd=14, events=POLLIN}, {fd=15, events=POLLIN}, {fd=19, events=POLLIN}, {fd=23, events=POLLIN|POLLERR|POLLHUP}, {fd=21, events=POLLIN|POLLERR|POLLHUP}, {fd=27, events=POLLIN|POLLERR|POLLHUP}, {fd=25, events=POLLIN|POLLERR|POLLHUP}, {fd=22, events=POLLIN|POLLERR|POLLHUP}, {fd=24, events=POLLIN|POLLERR|POLLHUP}, {fd=26, events=POLLIN|POLLERR|POLLHUP}, {fd=29, events=POLLIN|POLLERR|POLLHUP}, {fd=30, events=POLLIN|POLLERR|POLLHUP}, {fd=31, events=POLLIN|POLLERR|POLLHUP}, {fd=33, events=POLLIN|POLLERR|POLLHUP}, {fd=32, events=POLLIN|POLLERR|POLLHUP}, {fd=36, events=POLLIN|POLLERR|POLLHUP}, {fd=35, events=POLLIN|POLLERR|POLLHUP}, {fd=39, events=POLLIN|POLLERR|POLLHUP}, {fd=40, events=POLLIN|POLLERR|POLLHUP}, {fd=41, events=POLLIN|POLLERR|POLLHUP}, {fd=44, events=POLLIN|POLLERR|POLLHUP}, {fd=42, events=POLLIN|POLLERR|POLLHUP}, {fd=43, events=POLLIN|POLLERR|POLLHUP}, {fd=48, events=POLLIN|POLLERR|POLLHUP}, {fd=49, events=POLLIN|POLLERR|POLLHUP}, {fd=59, events=POLLIN|POLLERR|POLLHUP}, {fd=46, events=POLLIN|POLLERR|POLLHUP}, {fd=50, events=POLLIN|POLLERR|POLLHUP}, ...], 43, 5000
# gdb -p 5786 (gdb) thread apply all bt
Thread 17 (Thread 0x7f411a9d7700 (LWP 5788)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183ab0) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a9d7700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 16 (Thread 0x7f411a1d6700 (LWP 5789)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183a00) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f411a1d6700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 15 (Thread 0x7f41199d5700 (LWP 5790)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b183950) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f41199d5700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
Thread 14 (Thread 0x7f41191d4700 (LWP 5791)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f4129c2a2e6 in virCondWait (c=c@entry=0x7f412b18ebb8, m=m@entry=0x7f412b18eb90) at util/virthread.c:154 #2 0x00007f4129c2ada3 in virThreadPoolWorker (opaque=opaque@entry=0x7f412b1838a0) at util/virthreadpool.c:124 #3 0x00007f4129c2a078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f4127033dc5 in start_thread (arg=0x7f41191d4700) at pthread_create.c:308 #5 0x00007f4126d6273d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113
These are all idle workers.
Thread 13 (Thread 0x7f41189d3700 (LWP 5792)): ---Type <return> to continue, or q <return> to quit---
You omitted the backtrace of the other threads.
Jan
Please, specify what do you mean? What should I do? I'm not familiar with gdb at good level. Here is strace output on virsh: read(6, "", 7645) = 0 close(6) = 0 getuid() = 0 geteuid() = 0 geteuid() = 0 getuid() = 0 geteuid() = 0 socket(AF_LOCAL, SOCK_STREAM, 0) = 6 connect(6, {sa_family=AF_LOCAL, sun_path="/var/run/libvirt/libvirt-sock"}, 110) = 0 getsockname(6, {sa_family=AF_LOCAL, NULL}, [2]) = 0 futex(0x7fd1b09dcfe8, FUTEX_WAKE_PRIVATE, 2147483647) = 0 fcntl(6, F_GETFD) = 0 fcntl(6, F_SETFD, FD_CLOEXEC) = 0 fcntl(6, F_GETFL) = 0x2 (flags O_RDWR) fcntl(6, F_SETFL, O_RDWR|O_NONBLOCK) = 0 futex(0x7fd1b09dcee8, FUTEX_WAKE_PRIVATE, 2147483647) = 0 pipe2([7, 8], O_CLOEXEC) = 0 write(5, "\0", 1) = 1 futex(0x7fd1b09dc280, FUTEX_WAKE_PRIVATE, 1) = 1 futex(0x7fd1b09dcfb0, FUTEX_WAKE_PRIVATE, 2147483647) = 0 futex(0x7fd1b09dcf00, FUTEX_WAKE_PRIVATE, 2147483647) = 0 write(5, "\0", 1) = 1 futex(0x7fd1b09dc280, FUTEX_WAKE_PRIVATE, 1) = 1 rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0 poll([{fd=6, events=POLLOUT}, {fd=7, events=POLLIN}], 2, -1) = 1 ([{fd=6, revents=POLLOUT}]) rt_sigprocmask(SIG_SETMASK, [], NULL, 8) = 0 write(6, "\0\0\0\34 \0\200\206\0\0\0\1\0\0\0B\0\0\0\0\0\0\0\0\0\0\0\0", 28) = 28 rt_sigprocmask(SIG_BLOCK, [PIPE CHLD WINCH], [], 8) = 0 poll([{fd=6, events=POLLIN}, {fd=7, events=POLLIN}], 2, -1 ^Cstrace: Process 16202 detached And some errors at libvirtd log sometimes: янв 22 03:10:03 hostname.net libvirtd[22256]: 2018-01-22 00:10:03.639+0000: 22256: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error янв 22 03:15:03 hostname.net libvirtd[22256]: 2018-01-22 00:15:03.802+0000: 22256: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error янв 22 04:45:03 hostname.net libvirtd[22256]: 2018-01-22 01:45:03.210+0000: 22256: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error янв 22 09:10:03 hostname.net libvirtd[22256]: 2018-01-22 06:10:03.829+0000: 22256: error : virNetSocketReadWire:1808 : End of file while reading data: Input/output error --- Artem Likhachev
On Thu, Jan 25, 2018 at 11:17:54AM +0700, Artem Likhachev wrote:
Ján Tomko wrote 2018-01-18 14:27:
On Thu, Jan 18, 2018 at 11:30:16AM +0700, Artem Likhachev wrote:
# gdb -p 5786 (gdb) thread apply all bt
[...]
Thread 13 (Thread 0x7f41189d3700 (LWP 5792)): ---Type <return> to continue, or q <return> to quit---
You omitted the backtrace of the other threads.
Jan
Please, specify what do you mean? What should I do? I'm not familiar with gdb at good level.
When gdb asks: '---Type <return> to continue, or q <return> to quit---' keep pressing Enter until it displays the backtrace of all the threads, not just the ones that fit on the first screen. Jan
Ján Tomko писал 2018-01-25 16:04:
On Thu, Jan 25, 2018 at 11:17:54AM +0700, Artem Likhachev wrote:
Ján Tomko wrote 2018-01-18 14:27:
On Thu, Jan 18, 2018 at 11:30:16AM +0700, Artem Likhachev wrote:
# gdb -p 5786 (gdb) thread apply all bt
[...]
Thread 13 (Thread 0x7f41189d3700 (LWP 5792)): ---Type <return> to continue, or q <return> to quit---
You omitted the backtrace of the other threads.
Jan
Please, specify what do you mean? What should I do? I'm not familiar with gdb at good level.
When gdb asks: '---Type <return> to continue, or q <return> to quit---' keep pressing Enter until it displays the backtrace of all the threads, not just the ones that fit on the first screen.
Jan
!DSPAM:5a699db46321070757529!
Sorry, here is is: (gdb) thread apply all bt Thread 17 (Thread 0x7f360fc20700 (LWP 993)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9bb8, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621adeab0) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360fc20700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 16 (Thread 0x7f360f41f700 (LWP 994)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9bb8, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621adea00) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360f41f700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 15 (Thread 0x7f360ec1e700 (LWP 995)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9bb8, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621ade950) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360ec1e700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 14 (Thread 0x7f360e41d700 (LWP 996)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9bb8, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621ade8a0) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360e41d700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 13 (Thread 0x7f360dc1c700 (LWP 997)): ---Type <return> to continue, or q <return> to quit--- #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9bb8, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621adea00) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360dc1c700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 12 (Thread 0x7f360d41b700 (LWP 998)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9c58, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeed3b in virThreadPoolWorker (opaque=opaque@entry=0x7f3621ade8a0) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360d41b700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 11 (Thread 0x7f360cc1a700 (LWP 999)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9c58, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeed3b in virThreadPoolWorker (opaque=opaque@entry=0x7f3621ade950) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360cc1a700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 10 (Thread 0x7f360c419700 (LWP 1000)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9c58, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeed3b in virThreadPoolWorker (opaque=opaque@entry=0x7f3621adea00) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360c419700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 9 (Thread 0x7f360bc18700 (LWP 1001)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9c58, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00007f361eeeed3b in virThreadPoolWorker (opaque=opaque@entry=0x7f3621ade8a0) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360bc18700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 8 (Thread 0x7f360b417700 (LWP 1002)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621ae9c58, m=m@entry=0x7f3621ae9b90) at util/virthread.c:154 #2 0x00007f361eeeed3b in virThreadPoolWorker (opaque=opaque@entry=0x7f3621adea00) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360b417700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 7 (Thread 0x7f360136f700 (LWP 1003)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621b01f68, m=m@entry=0x7f3621b01f40) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621add6c0) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360136f700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 6 (Thread 0x7f3600b6e700 (LWP 1004)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621b01f68, m=m@entry=0x7f3621b01f40) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621add610) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f3600b6e700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 5 (Thread 0x7f360036d700 (LWP 1005)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621b01f68, m=m@entry=0x7f3621b01f40) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621add6c0) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f360036d700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 4 (Thread 0x7f35ffb6c700 (LWP 1007)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621b01f68, m=m@entry=0x7f3621b01f40) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621add610) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f35ffb6c700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 3 (Thread 0x7f35ff36b700 (LWP 1010)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f3621b01f68, m=m@entry=0x7f3621b01f40) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621add6c0) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f35ff36b700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 2 (Thread 0x7f35abfff700 (LWP 13299)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x00007f361eeee2e6 in virCondWait (c=c@entry=0x7f35f82d3d68, m=m@entry=0x7f35f82d3d40) at util/virthread.c:154 #2 0x00007f361eeeeda3 in virThreadPoolWorker (opaque=opaque@entry=0x7f3621ad8890) at util/virthreadpool.c:124 #3 0x00007f361eeee078 in virThreadHelper (data=<optimized out>) at util/virthread.c:206 #4 0x00007f361c2f7dc5 in start_thread (arg=0x7f35abfff700) at pthread_create.c:308 #5 0x00007f361c02673d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:113 Thread 1 (Thread 0x7f361fb94880 (LWP 850)): #0 0x00007f361c01bdfd in poll () at ../sysdeps/unix/syscall-template.S:81 #1 0x00007f361ee97f96 in poll (__timeout=5000, __nfds=39, __fds=<optimized out>) at /usr/include/bits/poll2.h:46 #2 virEventPollRunOnce () at util/vireventpoll.c:641 #3 0x00007f361ee96a72 in virEventRunDefaultImpl () at util/virevent.c:314 #4 0x00007f361f0053ad in virNetDaemonRun (dmn=dmn@entry=0x7f3621aeb770) at rpc/virnetdaemon.c:818 #5 0x00007f361fbf2e0c in main (argc=<optimized out>, argv=<optimized out>) at libvirtd.c:1541 (gdb) (gdb) -- Artem Likhachev
participants (2)
-
Artem Likhachev -
Ján Tomko