[libvirt-users] Libvirt 4.2.0 hang on destination on live migration cancel
I'm trying the following command on the source machine: virsh migrate --live --copy-storage-all --verbose TEST qemu+ssh:// 10.30.76.66/system If I ssh into the destination machine when this command is running, I can see NBD copying data as expected, and if I wait long enough it completes and succeeds. However if I ctrl+c this command above before it completes, it causes virsh commands (like virsh list) to just hang seemingly forever on the destination machine. Here's the libvirt debug entries when I run virsh list in this state: [root@host ~]# export LIBVIRT_DEBUG=1 [root@host ~]# virsh list 2018-10-02 05:14:05.053+0000: 16207: debug : virGlobalInit:359 : register drivers 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddbb60 name=Test 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering Test as driver 0 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddc660 name=ESX 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering ESX as driver 12018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddd560 name=remote 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering remote as driver 2 2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterDefaultImpl:280 : registering default event implementation 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddHandle:115 : Used 0 handle slots, adding at least 10 more 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0 2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f70c18791c0 opaque=(nil) ff=(nil) 2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterImpl:241 : addHandle=0x7f70c187a8a0 updateHandle=0x7f70c1879500 removeHandle=0x7f70c1879360 addTimeout=0x7f70c187a660 updateTimeout=0x7f70c1879690 removeTimeout=0x7f70c1879220 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddTimeout:230 : Used 0 timeout slots, adding at least 10 more2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0 2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddTimeout:253 : EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x5559f8dba380 opaque=0x7ffe5b15e700 ff=(nil) 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenAuth:1218 : name=<null>, auth=0x7f70c1ddea00, flags=0x0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a3f0 classname=virConnect 2018-10-02 05:14:05.054+0000: 16208: debug : virThreadJobSet:99 : Thread 16208 is now running job vshEventLoop 2018-10-02 05:14:05.054+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.054+0000: 16207: debug : virConfLoadConfig:1576 : Loading config file '/etc/libvirt/libvirt.conf' 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 1 2018-10-02 05:14:05.054+0000: 16207: debug : virConfReadFile:752 : filename=/etc/libvirt/libvirt.conf 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.054+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1 2018-10-02 05:14:05.054+0000: 16207: debug : virFileClose:110 : Closed fd 6 2018-10-02 05:14:05.054+0000: 16207: debug : virConfGetValueString:897 : Get value string (nil) 0 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1013 : no name, allowing driver auto-select 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 0 (Test) ... 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 0 Test returned DECLINED 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 1 (ESX) ... 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 1 ESX returned DECLINED 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 2 (remote) ... 2018-10-02 05:14:05.054+0000: 16207: debug : remoteConnectOpen:1350 : Auto-probe remote URI 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:916 : proceeding with name = 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:925 : Connecting with transport 1 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:1060 : Proceeding with sockname /var/run/libvirt/libvirt-sock 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:641 : path=/var/run/libvirt/libvirt-sock spawnDaemon=0 binary=<null> 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:705 : connect() succeeded 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNew:237 : localAddr=0x7ffe5b15dd80 remoteAddr=0x7ffe5b15dcf0 fd=6 errfd=-1 pid=0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83aa20 classname=virNetSocket 2018-10-02 05:14:05.054+0000: 16207: info : virNetSocketNew:293 : RPC_SOCKET_NEW: sock=0x5559fa83aa20 fd=6 errfd=-1 pid=0 localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ae20 classname=virNetClient 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientNew:328 : RPC_CLIENT_NEW: client=0x5559fa83ae20 sock=0x5559fa83aa20 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83aa20 2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=2 fd=6 events=1 cb=0x7f70c19b4df0 opaque=0x5559fa83aa20 ff=0x7f70c19b5320 2018-10-02 05:14:05.055+0000: 16207: debug : virKeepAliveNew:196 : client=0x5559fa83ae20, interval=-1, count=0 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b120 classname=virKeepAlive 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s) 2018-10-02 05:14:05.055+0000: 16207: info : virKeepAliveNew:215 : RPC_KEEPALIVE_NEW: ka=0x5559fa83b120 client=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b340 classname=virConnectCloseCallbackData 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b340 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b040 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ac00 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a9b0 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b040 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ac00 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83a9b0 2018-10-02 05:14:05.055+0000: 16207: debug : doRemoteOpen:1179 : Trying authentication 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=1 d=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageNew:46 : msg=0x5559fa83b720 tracked=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageEncodePayload:389 : Encode length as 28 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=2 timeout=-1 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientSendInternal:2121 : RPC_CLIENT_MSG_TX_QUEUE: client=0x5559fa83ae20 len=28 prog=536903814 vers=1 proc=66 type=0 status=0 serial=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientCallNew:2074 : New call 0x5559fa83b6a0: msg=0x5559fa83b720, expectReply=1, nonBlock=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1880 : Outgoing message prog=536903814 version=1 serial=0 proc=66 type=0 length=28 dispatch=(nil) 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1939 : We have the buck head=0x5559fa83b6a0 call=0x5559fa83b6a0 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=2 events=0 2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s) 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=0 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1 And here's some information from gbd when its in this state: [root@host ~]# ps axf | grep libvirtd 17010 pts/2 S+ 0:00 \_ grep libvirtd 15555 ? Sl 0:00 libvirtd --daemon [root@host ~]# gdb -p $(pgrep libvirtd) (gdb) t a a bt Thread 17 (Thread 0x2ae648a10700 (LWP 15559)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x2ae648c11700 (LWP 15560)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x2ae648e12700 (LWP 15561)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x2ae649013700 (LWP 15562)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x2ae649214700 (LWP 15563)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x2ae649415700 (LWP 15564)): ---Type <return> to continue, or q <return> to quit--- #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x2ae649616700 (LWP 15565)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 10 (Thread 0x2ae649817700 (LWP 15566)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x2ae649a18700 (LWP 15567)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x2ae649c19700 (LWP 15568)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x2ae64d0aa700 (LWP 15588)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x2ae64d2ab700 (LWP 15589)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x2ae64d4ac700 (LWP 15590)): ---Type <return> to continue, or q <return> to quit--- #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x2ae64d6ad700 (LWP 15591)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 3 (Thread 0x2ae64d8ae700 (LWP 15592)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x2ae64dec8700 (LWP 15632)): #0 0x00002ae63d97ed0d in recvmsg () from /lib64/libpthread.so.0 #1 0x00002ae63e9caef0 in udev_monitor_receive_device () from /lib64/libudev.so.0 #2 0x00002ae64be26864 in udevEventHandleThread (opaque=<value optimized out>) at node_device/node_device_udev.c:1623 #3 0x00002ae63b2c0075 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x2ae6429759e0 (LWP 15555)): #0 0x00002ae63d97e334 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00002ae63d9795d8 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00002ae63d9794a7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ae64be2364b in udevEventHandleCallback (watch=<value optimized out>, fd=19, events=<value optimized out>, data=<value optimized out>) at node_device/node_device_udev.c:1666 #4 0x00002ae63b257564 in virEventPollDispatchHandles () at util/vireventpoll.c:508 #5 virEventPollRunOnce () at util/vireventpoll.c:657 #6 0x00002ae63b255eb0 in virEventRunDefaultImpl () at util/virevent.c:327 #7 0x00002ae63b3a338d in virNetDaemonRun (dmn=0x55b1d39818d0) at rpc/virnetdaemon.c:858 #8 0x000055b1d2a4d60b in main (argc=<value optimized out>, argv=<value optimized out>) at remote/remote_daemon.c:1475 (gdb) This issue I know is fixed in 4.7.0 because I run this version on my CentOS 7 machines and don't have this issue. However on my CentOS 6 machines I have 4.2.0, and do experience this issue. Upgrading to 4.7.0 on CentOS 6 is troublesome though, because of the requirement of gnutls >= 3.2. I'm hoping with the above output someone might be able to point me in the right direction to a commit that resolved this between 4.2.0 and 4.7.0, or a possible way to prevent this hang in 4.2.0 as described above ? Thanks in advance
For anyone who might find this thread from online searching, I was able to determine this commit fixes the above issue: https://libvirt.org/git/?p=libvirt.git;a=commitdiff;h=dddcb601ebf97ef222a03b... On Tue, Oct 2, 2018 at 8:42 AM Scott Sullivan < scottgregorysullivan@gmail.com> wrote:
I'm trying the following command on the source machine:
virsh migrate --live --copy-storage-all --verbose TEST qemu+ssh:// 10.30.76.66/system
If I ssh into the destination machine when this command is running, I can see NBD copying data as expected, and if I wait long enough it completes and succeeds.
However if I ctrl+c this command above before it completes, it causes virsh commands (like virsh list) to just hang seemingly forever on the destination machine.
Here's the libvirt debug entries when I run virsh list in this state:
[root@host ~]# export LIBVIRT_DEBUG=1 [root@host ~]# virsh list 2018-10-02 05:14:05.053+0000: 16207: debug : virGlobalInit:359 : register drivers 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddbb60 name=Test 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering Test as driver 0 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddc660 name=ESX 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering ESX as driver 12018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddd560 name=remote
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering remote as driver 2 2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterDefaultImpl:280 : registering default event implementation 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddHandle:115 : Used 0 handle slots, adding at least 10 more 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0
2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f70c18791c0 opaque=(nil) ff=(nil) 2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterImpl:241 : addHandle=0x7f70c187a8a0 updateHandle=0x7f70c1879500 removeHandle=0x7f70c1879360 addTimeout=0x7f70c187a660 updateTimeout=0x7f70c1879690 removeTimeout=0x7f70c1879220 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddTimeout:230 : Used 0 timeout slots, adding at least 10 more2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0 2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddTimeout:253 : EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x5559f8dba380 opaque=0x7ffe5b15e700 ff=(nil) 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenAuth:1218 : name=<null>, auth=0x7f70c1ddea00, flags=0x0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a3f0 classname=virConnect 2018-10-02 05:14:05.054+0000: 16208: debug : virThreadJobSet:99 : Thread 16208 is now running job vshEventLoop 2018-10-02 05:14:05.054+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.054+0000: 16207: debug : virConfLoadConfig:1576 : Loading config file '/etc/libvirt/libvirt.conf' 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 1 2018-10-02 05:14:05.054+0000: 16207: debug : virConfReadFile:752 : filename=/etc/libvirt/libvirt.conf 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.054+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1 2018-10-02 05:14:05.054+0000: 16207: debug : virFileClose:110 : Closed fd 6 2018-10-02 05:14:05.054+0000: 16207: debug : virConfGetValueString:897 : Get value string (nil) 0 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1013 : no name, allowing driver auto-select 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 0 (Test) ... 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 0 Test returned DECLINED 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 1 (ESX) ... 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 1 ESX returned DECLINED 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 2 (remote) ... 2018-10-02 05:14:05.054+0000: 16207: debug : remoteConnectOpen:1350 : Auto-probe remote URI 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:916 : proceeding with name = 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:925 : Connecting with transport 1 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:1060 : Proceeding with sockname /var/run/libvirt/libvirt-sock 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:641 : path=/var/run/libvirt/libvirt-sock spawnDaemon=0 binary=<null> 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:705 : connect() succeeded 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNew:237 : localAddr=0x7ffe5b15dd80 remoteAddr=0x7ffe5b15dcf0 fd=6 errfd=-1 pid=0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83aa20 classname=virNetSocket 2018-10-02 05:14:05.054+0000: 16207: info : virNetSocketNew:293 : RPC_SOCKET_NEW: sock=0x5559fa83aa20 fd=6 errfd=-1 pid=0 localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ae20 classname=virNetClient 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientNew:328 : RPC_CLIENT_NEW: client=0x5559fa83ae20 sock=0x5559fa83aa20 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83aa20 2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=2 fd=6 events=1 cb=0x7f70c19b4df0 opaque=0x5559fa83aa20 ff=0x7f70c19b5320 2018-10-02 05:14:05.055+0000: 16207: debug : virKeepAliveNew:196 : client=0x5559fa83ae20, interval=-1, count=0 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b120 classname=virKeepAlive 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s) 2018-10-02 05:14:05.055+0000: 16207: info : virKeepAliveNew:215 : RPC_KEEPALIVE_NEW: ka=0x5559fa83b120 client=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b340 classname=virConnectCloseCallbackData 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b340 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b040 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ac00 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a9b0 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b040 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ac00 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83a9b0 2018-10-02 05:14:05.055+0000: 16207: debug : doRemoteOpen:1179 : Trying authentication 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=1 d=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageNew:46 : msg=0x5559fa83b720 tracked=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageEncodePayload:389 : Encode length as 28 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=2 timeout=-1 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientSendInternal:2121 : RPC_CLIENT_MSG_TX_QUEUE: client=0x5559fa83ae20 len=28 prog=536903814 vers=1 proc=66 type=0 status=0 serial=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientCallNew:2074 : New call 0x5559fa83b6a0: msg=0x5559fa83b720, expectReply=1, nonBlock=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1880 : Outgoing message prog=536903814 version=1 serial=0 proc=66 type=0 length=28 dispatch=(nil) 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1939 : We have the buck head=0x5559fa83b6a0 call=0x5559fa83b6a0 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=2 events=0 2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s) 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=0 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1
And here's some information from gbd when its in this state:
[root@host ~]# ps axf | grep libvirtd 17010 pts/2 S+ 0:00 \_ grep libvirtd 15555 ? Sl 0:00 libvirtd --daemon [root@host ~]# gdb -p $(pgrep libvirtd)
(gdb) t a a bt
Thread 17 (Thread 0x2ae648a10700 (LWP 15559)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x2ae648c11700 (LWP 15560)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x2ae648e12700 (LWP 15561)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x2ae649013700 (LWP 15562)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x2ae649214700 (LWP 15563)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x2ae649415700 (LWP 15564)): ---Type <return> to continue, or q <return> to quit---
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x2ae649616700 (LWP 15565)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 10 (Thread 0x2ae649817700 (LWP 15566)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x2ae649a18700 (LWP 15567)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit---
#4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x2ae649c19700 (LWP 15568)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 7 (Thread 0x2ae64d0aa700 (LWP 15588)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x2ae64d2ab700 (LWP 15589)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x2ae64d4ac700 (LWP 15590)): ---Type <return> to continue, or q <return> to quit---
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x2ae64d6ad700 (LWP 15591)):
#0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 3 (Thread 0x2ae64d8ae700 (LWP 15592)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x2ae64dec8700 (LWP 15632)): #0 0x00002ae63d97ed0d in recvmsg () from /lib64/libpthread.so.0 #1 0x00002ae63e9caef0 in udev_monitor_receive_device () from /lib64/libudev.so.0 #2 0x00002ae64be26864 in udevEventHandleThread (opaque=<value optimized out>) at node_device/node_device_udev.c:1623 #3 0x00002ae63b2c0075 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x2ae6429759e0 (LWP 15555)): #0 0x00002ae63d97e334 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00002ae63d9795d8 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00002ae63d9794a7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ae64be2364b in udevEventHandleCallback (watch=<value optimized out>, fd=19, events=<value optimized out>, data=<value optimized out>) at node_device/node_device_udev.c:1666 #4 0x00002ae63b257564 in virEventPollDispatchHandles () at util/vireventpoll.c:508 #5 virEventPollRunOnce () at util/vireventpoll.c:657 #6 0x00002ae63b255eb0 in virEventRunDefaultImpl () at util/virevent.c:327 #7 0x00002ae63b3a338d in virNetDaemonRun (dmn=0x55b1d39818d0) at rpc/virnetdaemon.c:858 #8 0x000055b1d2a4d60b in main (argc=<value optimized out>, argv=<value optimized out>) at remote/remote_daemon.c:1475 (gdb)
This issue I know is fixed in 4.7.0 because I run this version on my CentOS 7 machines and don't have this issue. However on my CentOS 6 machines I have 4.2.0, and do experience this issue. Upgrading to 4.7.0 on CentOS 6 is troublesome though, because of the requirement of gnutls >= 3.2.
I'm hoping with the above output someone might be able to point me in the right direction to a commit that resolved this between 4.2.0 and 4.7.0, or a possible way to prevent this hang in 4.2.0 as described above ?
Thanks in advance
participants (1)
-
Scott Sullivan