[libvirt-users] Libvirt 4.2.0 hang on destination on live migration cancel

Tuesday, 2 October 2018

I'm trying the following command on the source machine:

virsh migrate --live --copy-storage-all  --verbose TEST qemu+ssh://
10.30.76.66/system

If I ssh into the destination machine when this command is running, I can
see NBD copying data as expected, and if I wait long enough it completes
and succeeds.

However if I ctrl+c this command above before it completes, it causes virsh
commands (like virsh list) to just hang seemingly forever on the
destination machine.

Here's the libvirt debug entries when I run virsh list in this state:

[root@host ~]# export LIBVIRT_DEBUG=1
[root@host ~]# virsh list
2018-10-02 05:14:05.053+0000: 16207: debug : virGlobalInit:359 : register
drivers
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 :
driver=0x7f70c1ddbb60 name=Test
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 :
registering Test as driver 0
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 :
driver=0x7f70c1ddc660 name=ESX
2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 :
registering ESX as driver 12018-10-02 05:14:05.054+0000: 16207: debug :
virRegisterConnectDriver:657 : driver=0x7f70c1ddd560 name=remote

2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 :
registering remote as driver 2
2018-10-02 05:14:05.054+0000: 16207: debug :
virEventRegisterDefaultImpl:280 : registering default event implementation
2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddHandle:115 :
Used 0 handle slots, adding at least 10 more
2018-10-02 05:14:05.054+0000: 16207: debug :
virEventPollInterruptLocked:722 : Skip interrupt, 0 0

2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddHandle:140 :
EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f70c18791c0 opaque=(nil)
ff=(nil)
2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterImpl:241 :
addHandle=0x7f70c187a8a0 updateHandle=0x7f70c1879500
removeHandle=0x7f70c1879360 addTimeout=0x7f70c187a660
updateTimeout=0x7f70c1879690 removeTimeout=0x7f70c1879220
2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddTimeout:230 :
Used 0 timeout slots, adding at least 10 more2018-10-02 05:14:05.054+0000:
16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0
2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddTimeout:253 :
EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x5559f8dba380
opaque=0x7ffe5b15e700 ff=(nil)
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenAuth:1218 :
name=<null>, auth=0x7f70c1ddea00, flags=0x0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83a3f0 classname=virConnect
                          2018-10-02 05:14:05.054+0000: 16208: debug :
virThreadJobSet:99 : Thread 16208 is now running job vshEventLoop
2018-10-02 05:14:05.054+0000: 16208: debug : virEventRunDefaultImpl:324 :
running default event implementation
2018-10-02 05:14:05.054+0000: 16207: debug : virConfLoadConfig:1576 :
Loading config file '/etc/libvirt/libvirt.conf'
2018-10-02 05:14:05.054+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 1
2018-10-02 05:14:05.054+0000: 16207: debug : virConfReadFile:752 :
filename=/etc/libvirt/libvirt.conf
2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.054+0000: 16208: debug :
virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.054+0000: 16208: debug :
virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.054+0000: 16208: info : virEventPollRunOnce:640 :
EVENT_POLL_RUN: nhandles=1 timeout=-1
2018-10-02 05:14:05.054+0000: 16207: debug : virFileClose:110 : Closed fd 6
2018-10-02 05:14:05.054+0000: 16207: debug : virConfGetValueString:897 :
Get value string (nil) 0
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1013 :
no name, allowing driver auto-select
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 :
trying driver 0 (Test) ...
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 :
driver 0 Test returned DECLINED
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 :
trying driver 1 (ESX) ...
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 :
driver 1 ESX returned DECLINED
2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 :
trying driver 2 (remote) ...
2018-10-02 05:14:05.054+0000: 16207: debug : remoteConnectOpen:1350 :
Auto-probe remote URI
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:916 : proceeding
with name =
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:925 : Connecting
with transport 1
2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:1060 : Proceeding
with sockname /var/run/libvirt/libvirt-sock
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:641
: path=/var/run/libvirt/libvirt-sock spawnDaemon=0 binary=<null>
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:705
: connect() succeeded
2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNew:237 :
localAddr=0x7ffe5b15dd80 remoteAddr=0x7ffe5b15dcf0 fd=6 errfd=-1 pid=0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83aa20 classname=virNetSocket
2018-10-02 05:14:05.054+0000: 16207: info : virNetSocketNew:293 :
RPC_SOCKET_NEW: sock=0x5559fa83aa20 fd=6 errfd=-1 pid=0
localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0
2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83ae20 classname=virNetClient
2018-10-02 05:14:05.055+0000: 16207: info : virNetClientNew:328 :
RPC_CLIENT_NEW: client=0x5559fa83ae20 sock=0x5559fa83aa20
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83aa20
2018-10-02 05:14:05.055+0000: 16207: debug :
virEventPollInterruptLocked:726 : Interrupting
2018-10-02 05:14:05.055+0000: 16207: info : virEventPollAddHandle:140 :
EVENT_POLL_ADD_HANDLE: watch=2 fd=6 events=1 cb=0x7f70c19b4df0
opaque=0x5559fa83aa20 ff=0x7f70c19b5320
2018-10-02 05:14:05.055+0000: 16207: debug : virKeepAliveNew:196 :
client=0x5559fa83ae20, interval=-1, count=0
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83b120 classname=virKeepAlive
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll
got 1 event(s)
2018-10-02 05:14:05.055+0000: 16207: info : virKeepAliveNew:215 :
RPC_KEEPALIVE_NEW: ka=0x5559fa83b120 client=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchTimeouts:432 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83ae20
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:478 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83b340 classname=virConnectCloseCallbackData
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:492 : i=0 w=1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83b340
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506
: EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83b040 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83ac00 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW:
obj=0x5559fa83a9b0 classname=virNetClientProgram
2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 :
running default event implementation
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83b040
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83ac00
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF:
obj=0x5559fa83a9b0
2018-10-02 05:14:05.055+0000: 16207: debug : doRemoteOpen:1179 : Trying
authentication
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=1 w=2, f=6 e=1 d=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageNew:46 :
msg=0x5559fa83b720 tracked=0
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageEncodePayload:389
: Encode length as 28
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 :
EVENT_POLL_RUN: nhandles=2 timeout=-1
2018-10-02 05:14:05.055+0000: 16207: info : virNetClientSendInternal:2121 :
RPC_CLIENT_MSG_TX_QUEUE: client=0x5559fa83ae20 len=28 prog=536903814 vers=1
proc=66 type=0 status=0 serial=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientCallNew:2074 : New
call 0x5559fa83b6a0: msg=0x5559fa83b720, expectReply=1, nonBlock=0
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1880 : Outgoing
message prog=536903814 version=1 serial=0 proc=66 type=0 length=28
dispatch=(nil)
2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1939 : We have
the buck head=0x5559fa83b6a0 call=0x5559fa83b6a0
2018-10-02 05:14:05.055+0000: 16207: info : virEventPollUpdateHandle:152 :
EVENT_POLL_UPDATE_HANDLE: watch=2 events=0
2018-10-02 05:14:05.055+0000: 16207: debug :
virEventPollInterruptLocked:726 : Interrupting
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll
got 1 event(s)
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchTimeouts:432 : Dispatch 1
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:478 : Dispatch 2
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollDispatchHandles:492 : i=0 w=1
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506
: EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 :
running default event implementation
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCleanupTimeouts:525 : Cleanup 1
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574
: Cleanup 2
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=0 w=1, f=4 e=1 d=0
2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 :
Prepare n=1 w=2, f=6 e=0 d=0
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers
2018-10-02 05:14:05.055+0000: 16208: debug :
virEventPollCalculateTimeout:371 : No timeout is pending
2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 :
EVENT_POLL_RUN: nhandles=1 timeout=-1

And here's some information from gbd when its in this state:

[root@host ~]# ps axf | grep libvirtd
17010 pts/2    S+     0:00          \_ grep libvirtd
15555 ?        Sl     0:00 libvirtd --daemon
[root@host ~]# gdb -p $(pgrep libvirtd)

(gdb) t a a bt

Thread 17 (Thread 0x2ae648a10700 (LWP 15559)):

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x2ae648c11700 (LWP 15560)):

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
---Type <return> to continue, or q <return> to quit---
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0

#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x2ae648e12700 (LWP 15561)):
#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x2ae649013700 (LWP 15562)):
#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit---
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x2ae649214700 (LWP 15563)):
#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0

#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x2ae649415700 (LWP 15564)):
---Type <return> to continue, or q <return> to quit---

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x2ae649616700 (LWP 15565)):

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---

Thread 10 (Thread 0x2ae649817700 (LWP 15566)):
#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0

#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x2ae649a18700 (LWP 15567)):
#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
---Type <return> to continue, or q <return> to quit---

#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x2ae649c19700 (LWP 15568)):

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x2ae64d0aa700 (LWP 15588)):

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
---Type <return> to continue, or q <return> to quit---
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0

#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x2ae64d2ab700 (LWP 15589)):
#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0

#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x2ae64d4ac700 (LWP 15590)):
---Type <return> to continue, or q <return> to quit---

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x2ae64d6ad700 (LWP 15591)):

#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---

Thread 3 (Thread 0x2ae64d8ae700 (LWP 15592)):
#0  0x00002ae63d97b68c in pthread_cond_wait@(a)GLIBC_2.3.2 () from
/lib64/libpthread.so.0
#1  0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value
optimized out>) at util/virthread.c:154
#2  0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized
out>) at util/virthreadpool.c:124
#3  0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x2ae64dec8700 (LWP 15632)):
#0  0x00002ae63d97ed0d in recvmsg () from /lib64/libpthread.so.0
#1  0x00002ae63e9caef0 in udev_monitor_receive_device () from
/lib64/libudev.so.0
#2  0x00002ae64be26864 in udevEventHandleThread (opaque=<value optimized
out>) at node_device/node_device_udev.c:1623
#3  0x00002ae63b2c0075 in virThreadHelper (data=<value optimized out>) at
util/virthread.c:206
---Type <return> to continue, or q <return> to quit---
#4  0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0
#5  0x00002ae63e4d893d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x2ae6429759e0 (LWP 15555)):
#0  0x00002ae63d97e334 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00002ae63d9795d8 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x00002ae63d9794a7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00002ae64be2364b in udevEventHandleCallback (watch=<value optimized
out>, fd=19, events=<value optimized out>, data=<value optimized out>) at
node_device/node_device_udev.c:1666
#4  0x00002ae63b257564 in virEventPollDispatchHandles () at
util/vireventpoll.c:508
#5  virEventPollRunOnce () at util/vireventpoll.c:657
#6  0x00002ae63b255eb0 in virEventRunDefaultImpl () at util/virevent.c:327
#7  0x00002ae63b3a338d in virNetDaemonRun (dmn=0x55b1d39818d0) at
rpc/virnetdaemon.c:858
#8  0x000055b1d2a4d60b in main (argc=<value optimized out>, argv=<value
optimized out>) at remote/remote_daemon.c:1475
(gdb)

This issue I know is fixed in 4.7.0 because I run this version on my CentOS
7 machines and don't have this issue. However on my CentOS 6 machines I
have 4.2.0, and do experience this issue. Upgrading to 4.7.0 on CentOS 6 is
troublesome though, because of the requirement of gnutls >= 3.2.

I'm hoping with the above output someone might be able to point me in the
right direction to a commit that resolved this between 4.2.0 and 4.7.0, or
a possible way to prevent this hang in 4.2.0 as described above ?

Thanks in advance

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

[libvirt-users] Libvirt 4.2.0 hang on destination on live migration cancel