- Users - libvirt List Archives

[libvirt-users] Libvirt 4.2.0 hang on destination on live migration cancel
by Scott Sullivan 03 Oct '18

03 Oct '18

I'm trying the following command on the source machine: virsh migrate --live --copy-storage-all --verbose TEST qemu+ssh:// 10.30.76.66/system If I ssh into the destination machine when this command is running, I can see NBD copying data as expected, and if I wait long enough it completes and succeeds. However if I ctrl+c this command above before it completes, it causes virsh commands (like virsh list) to just hang seemingly forever on the destination machine. Here's the libvirt debug entries when I run virsh list in this state: [root@host ~]# export LIBVIRT_DEBUG=1 [root@host ~]# virsh list 2018-10-02 05:14:05.053+0000: 16207: debug : virGlobalInit:359 : register drivers 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddbb60 name=Test 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering Test as driver 0 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddc660 name=ESX 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering ESX as driver 12018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:657 : driver=0x7f70c1ddd560 name=remote 2018-10-02 05:14:05.054+0000: 16207: debug : virRegisterConnectDriver:668 : registering remote as driver 2 2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterDefaultImpl:280 : registering default event implementation 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddHandle:115 : Used 0 handle slots, adding at least 10 more 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0 2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=1 fd=4 events=1 cb=0x7f70c18791c0 opaque=(nil) ff=(nil) 2018-10-02 05:14:05.054+0000: 16207: debug : virEventRegisterImpl:241 : addHandle=0x7f70c187a8a0 updateHandle=0x7f70c1879500 removeHandle=0x7f70c1879360 addTimeout=0x7f70c187a660 updateTimeout=0x7f70c1879690 removeTimeout=0x7f70c1879220 2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollAddTimeout:230 : Used 0 timeout slots, adding at least 10 more2018-10-02 05:14:05.054+0000: 16207: debug : virEventPollInterruptLocked:722 : Skip interrupt, 0 0 2018-10-02 05:14:05.054+0000: 16207: info : virEventPollAddTimeout:253 : EVENT_POLL_ADD_TIMEOUT: timer=1 frequency=-1 cb=0x5559f8dba380 opaque=0x7ffe5b15e700 ff=(nil) 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenAuth:1218 : name=<null>, auth=0x7f70c1ddea00, flags=0x0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a3f0 classname=virConnect 2018-10-02 05:14:05.054+0000: 16208: debug : virThreadJobSet:99 : Thread 16208 is now running job vshEventLoop 2018-10-02 05:14:05.054+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.054+0000: 16207: debug : virConfLoadConfig:1576 : Loading config file '/etc/libvirt/libvirt.conf' 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 1 2018-10-02 05:14:05.054+0000: 16207: debug : virConfReadFile:752 : filename=/etc/libvirt/libvirt.conf 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.054+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.054+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1 2018-10-02 05:14:05.054+0000: 16207: debug : virFileClose:110 : Closed fd 6 2018-10-02 05:14:05.054+0000: 16207: debug : virConfGetValueString:897 : Get value string (nil) 0 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1013 : no name, allowing driver auto-select 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 0 (Test) ... 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 0 Test returned DECLINED 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 1 (ESX) ... 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1071 : driver 1 ESX returned DECLINED 2018-10-02 05:14:05.054+0000: 16207: debug : virConnectOpenInternal:1056 : trying driver 2 (remote) ... 2018-10-02 05:14:05.054+0000: 16207: debug : remoteConnectOpen:1350 : Auto-probe remote URI 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:916 : proceeding with name = 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:925 : Connecting with transport 1 2018-10-02 05:14:05.054+0000: 16207: debug : doRemoteOpen:1060 : Proceeding with sockname /var/run/libvirt/libvirt-sock 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:641 : path=/var/run/libvirt/libvirt-sock spawnDaemon=0 binary=<null> 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNewConnectUNIX:705 : connect() succeeded 2018-10-02 05:14:05.054+0000: 16207: debug : virNetSocketNew:237 : localAddr=0x7ffe5b15dd80 remoteAddr=0x7ffe5b15dcf0 fd=6 errfd=-1 pid=0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83aa20 classname=virNetSocket 2018-10-02 05:14:05.054+0000: 16207: info : virNetSocketNew:293 : RPC_SOCKET_NEW: sock=0x5559fa83aa20 fd=6 errfd=-1 pid=0 localAddr=127.0.0.1;0, remoteAddr=127.0.0.1;0 2018-10-02 05:14:05.054+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ae20 classname=virNetClient 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientNew:328 : RPC_CLIENT_NEW: client=0x5559fa83ae20 sock=0x5559fa83aa20 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83aa20 2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollAddHandle:140 : EVENT_POLL_ADD_HANDLE: watch=2 fd=6 events=1 cb=0x7f70c19b4df0 opaque=0x5559fa83aa20 ff=0x7f70c19b5320 2018-10-02 05:14:05.055+0000: 16207: debug : virKeepAliveNew:196 : client=0x5559fa83ae20, interval=-1, count=0 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b120 classname=virKeepAlive 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s) 2018-10-02 05:14:05.055+0000: 16207: info : virKeepAliveNew:215 : RPC_KEEPALIVE_NEW: ka=0x5559fa83b120 client=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ae20 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b340 classname=virConnectCloseCallbackData 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b340 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83b040 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83ac00 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16207: info : virObjectNew:254 : OBJECT_NEW: obj=0x5559fa83a9b0 classname=virNetClientProgram 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83b040 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83ac00 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16207: info : virObjectRef:388 : OBJECT_REF: obj=0x5559fa83a9b0 2018-10-02 05:14:05.055+0000: 16207: debug : doRemoteOpen:1179 : Trying authentication 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=1 d=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageNew:46 : msg=0x5559fa83b720 tracked=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.055+0000: 16207: debug : virNetMessageEncodePayload:389 : Encode length as 28 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=2 timeout=-1 2018-10-02 05:14:05.055+0000: 16207: info : virNetClientSendInternal:2121 : RPC_CLIENT_MSG_TX_QUEUE: client=0x5559fa83ae20 len=28 prog=536903814 vers=1 proc=66 type=0 status=0 serial=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientCallNew:2074 : New call 0x5559fa83b6a0: msg=0x5559fa83b720, expectReply=1, nonBlock=0 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1880 : Outgoing message prog=536903814 version=1 serial=0 proc=66 type=0 length=28 dispatch=(nil) 2018-10-02 05:14:05.055+0000: 16207: debug : virNetClientIO:1939 : We have the buck head=0x5559fa83b6a0 call=0x5559fa83b6a0 2018-10-02 05:14:05.055+0000: 16207: info : virEventPollUpdateHandle:152 : EVENT_POLL_UPDATE_HANDLE: watch=2 events=0 2018-10-02 05:14:05.055+0000: 16207: debug : virEventPollInterruptLocked:726 : Interrupting 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollRunOnce:650 : Poll got 1 event(s) 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchTimeouts:432 : Dispatch 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:478 : Dispatch 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollDispatchHandles:492 : i=0 w=1 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollDispatchHandles:506 : EVENT_POLL_DISPATCH_HANDLE: watch=1 events=1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventRunDefaultImpl:324 : running default event implementation 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupTimeouts:525 : Cleanup 1 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCleanupHandles:574 : Cleanup 2 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=0 w=1, f=4 e=1 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollMakePollFDs:401 : Prepare n=1 w=2, f=6 e=0 d=0 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:338 : Calculate expiry of 1 timers 2018-10-02 05:14:05.055+0000: 16208: debug : virEventPollCalculateTimeout:371 : No timeout is pending 2018-10-02 05:14:05.055+0000: 16208: info : virEventPollRunOnce:640 : EVENT_POLL_RUN: nhandles=1 timeout=-1 And here's some information from gbd when its in this state: [root@host ~]# ps axf | grep libvirtd 17010 pts/2 S+ 0:00 \_ grep libvirtd 15555 ? Sl 0:00 libvirtd --daemon [root@host ~]# gdb -p $(pgrep libvirtd) (gdb) t a a bt Thread 17 (Thread 0x2ae648a10700 (LWP 15559)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 16 (Thread 0x2ae648c11700 (LWP 15560)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 15 (Thread 0x2ae648e12700 (LWP 15561)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 14 (Thread 0x2ae649013700 (LWP 15562)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 13 (Thread 0x2ae649214700 (LWP 15563)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 12 (Thread 0x2ae649415700 (LWP 15564)): ---Type <return> to continue, or q <return> to quit--- #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 11 (Thread 0x2ae649616700 (LWP 15565)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 10 (Thread 0x2ae649817700 (LWP 15566)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 9 (Thread 0x2ae649a18700 (LWP 15567)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 8 (Thread 0x2ae649c19700 (LWP 15568)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13a8 in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 7 (Thread 0x2ae64d0aa700 (LWP 15588)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 ---Type <return> to continue, or q <return> to quit--- #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 6 (Thread 0x2ae64d2ab700 (LWP 15589)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 5 (Thread 0x2ae64d4ac700 (LWP 15590)): ---Type <return> to continue, or q <return> to quit--- #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 4 (Thread 0x2ae64d6ad700 (LWP 15591)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 ---Type <return> to continue, or q <return> to quit--- Thread 3 (Thread 0x2ae64d8ae700 (LWP 15592)): #0 0x00002ae63d97b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x00002ae63b2c01ba in virCondWait (c=<value optimized out>, m=<value optimized out>) at util/virthread.c:154 #2 0x00002ae63b2c13fb in virThreadPoolWorker (opaque=<value optimized out>) at util/virthreadpool.c:124 #3 0x00002ae63b2c0036 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 2 (Thread 0x2ae64dec8700 (LWP 15632)): #0 0x00002ae63d97ed0d in recvmsg () from /lib64/libpthread.so.0 #1 0x00002ae63e9caef0 in udev_monitor_receive_device () from /lib64/libudev.so.0 #2 0x00002ae64be26864 in udevEventHandleThread (opaque=<value optimized out>) at node_device/node_device_udev.c:1623 #3 0x00002ae63b2c0075 in virThreadHelper (data=<value optimized out>) at util/virthread.c:206 ---Type <return> to continue, or q <return> to quit--- #4 0x00002ae63d977aa1 in start_thread () from /lib64/libpthread.so.0 #5 0x00002ae63e4d893d in clone () from /lib64/libc.so.6 Thread 1 (Thread 0x2ae6429759e0 (LWP 15555)): #0 0x00002ae63d97e334 in __lll_lock_wait () from /lib64/libpthread.so.0 #1 0x00002ae63d9795d8 in _L_lock_854 () from /lib64/libpthread.so.0 #2 0x00002ae63d9794a7 in pthread_mutex_lock () from /lib64/libpthread.so.0 #3 0x00002ae64be2364b in udevEventHandleCallback (watch=<value optimized out>, fd=19, events=<value optimized out>, data=<value optimized out>) at node_device/node_device_udev.c:1666 #4 0x00002ae63b257564 in virEventPollDispatchHandles () at util/vireventpoll.c:508 #5 virEventPollRunOnce () at util/vireventpoll.c:657 #6 0x00002ae63b255eb0 in virEventRunDefaultImpl () at util/virevent.c:327 #7 0x00002ae63b3a338d in virNetDaemonRun (dmn=0x55b1d39818d0) at rpc/virnetdaemon.c:858 #8 0x000055b1d2a4d60b in main (argc=<value optimized out>, argv=<value optimized out>) at remote/remote_daemon.c:1475 (gdb) This issue I know is fixed in 4.7.0 because I run this version on my CentOS 7 machines and don't have this issue. However on my CentOS 6 machines I have 4.2.0, and do experience this issue. Upgrading to 4.7.0 on CentOS 6 is troublesome though, because of the requirement of gnutls >= 3.2. I'm hoping with the above output someone might be able to point me in the right direction to a commit that resolved this between 4.2.0 and 4.7.0, or a possible way to prevent this hang in 4.2.0 as described above ? Thanks in advance

1 1

Re: [libvirt-users] [virt-tools-list] Virt-manager: Restricted networks
by Cole Robinson 02 Oct '18

02 Oct '18

On 10/02/2018 08:50 AM, Olivier Léobal wrote: > Hello, > > 'Isolated' networks created in VMM (I’m running 1.4.3) still connect to > the host (as gateway). Is it possible to avoid this? It is my > understanding that QEMU provides a 'restrict' option for that, but I > don't understand it well, and can’t find it in VMM. > ccing libvirt-users That's expected of isolated mode, according to this: https://wiki.libvirt.org/page/VirtualNetworking#Isolated_mode I'm not sure if/how to go about creating a network that only VMs can communicate on - Cole

2 1

[libvirt-users] This QEMU doesn't support the LSI 53C895A SCSI controller
by Gionatan Danti 02 Oct '18

02 Oct '18

Hi all, trying to edit a domain xml to enable an LSI SCSI controller I get the following error: error: unsupported configuration: This QEMU doesn't support the LSI 53C895A SCSI controller It is my understanding that the error is raised due to RedHat disabling this controller in its own qemu-kvm builds. This seems an unfortunate decision, as it makes harder to migrate from VMWare (which uses LSI SCSI and SAS adapters) to RH+KVM. Can anyone elaborate on this decision? It is possible to enable LSI controller support in RedHat 7.x? Should I open a BZ against it? Thanks. -- Danti Gionatan Supporto Tecnico Assyoma S.r.l. - www.assyoma.it email: g.danti(a)assyoma.it - info(a)assyoma.it GPG public key ID: FF5F32A8

2 3

[libvirt-users] libvirt reported capabilities doesn't match /proc/cpuinfo while the model does match
by Braiam Peguero 01 Oct '18

01 Oct '18

Hi, According to virsh capabilities I only have the following cpu features: <cpu> <arch>x86_64</arch> <model>IvyBridge-IBRS</model> <vendor>Intel</vendor> <microcode version='32'/> <topology sockets='1' cores='4' threads='1'/> <feature name='ds'/> <feature name='acpi'/> <feature name='ss'/> <feature name='ht'/> <feature name='tm'/> <feature name='pbe'/> <feature name='dtes64'/> <feature name='monitor'/> <feature name='ds_cpl'/> <feature name='vmx'/> <feature name='smx'/> <feature name='est'/> <feature name='tm2'/> <feature name='xtpr'/> <feature name='pdcm'/> <feature name='pcid'/> <feature name='osxsave'/> <feature name='arat'/> <feature name='ssbd'/> <feature name='xsaveopt'/> <feature name='invtsc'/> <pages unit='KiB' size='4'/> <pages unit='KiB' size='2048'/> </cpu> Meanwhile, cpuinfo reports the following: flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm cpuid_fault epb pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms xsaveopt dtherm ida arat pln pts flush_l1d bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf This results on my cpu being detected, if I allow host copy, as AMD chip and the guest becomes unbearably slow. The model of the host cpu is correct. I'm using Debian testing/unstable. Compiled against library: libvirt 4.7.0 Using library: libvirt 4.7.0 Using API: QEMU 4.7.0 Running hypervisor: QEMU 2.12.0 -- Braiam

2 1

[libvirt-users] Libvirt TLS with Short Lived Certificates
by Charles Urquiola 01 Oct '18

01 Oct '18

I want to use short lived certificates with libvirtd to provided TLS access to the daemon. New certificates are generated on a daily basis and delivered to the host. Does libvirtd re-read TLS certificates with a reload of the service, systemctl reload libvirtd, or with a SIGHUP or is a full restart of the daemon required? --charlie

2 1

[libvirt-users] NUMA issues on virtualized hosts
by Lukas Hejtmanek 20 Sep '18

20 Sep '18

Hello, I have cluster with AMD EPYC 7351 cpu. Two CPUs per node. I have performance 8-NUMA configuration: This is from hypervizor: [root@hde10 ~]# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 64 On-line CPU(s) list: 0-63 Thread(s) per core: 2 Core(s) per socket: 16 Socket(s): 2 NUMA node(s): 8 Vendor ID: AuthenticAMD CPU family: 23 Model: 1 Model name: AMD EPYC 7351 16-Core Processor Stepping: 2 CPU MHz: 1800.000 CPU max MHz: 2400.0000 CPU min MHz: 1200.0000 BogoMIPS: 4800.05 Virtualization: AMD-V L1d cache: 32K L1i cache: 64K L2 cache: 512K L3 cache: 8192K NUMA node0 CPU(s): 0-3,32-35 NUMA node1 CPU(s): 4-7,36-39 NUMA node2 CPU(s): 8-11,40-43 NUMA node3 CPU(s): 12-15,44-47 NUMA node4 CPU(s): 16-19,48-51 NUMA node5 CPU(s): 20-23,52-55 NUMA node6 CPU(s): 24-27,56-59 NUMA node7 CPU(s): 28-31,60-63 I'm running one big virtual on this hypervizor - almost whole memory + all physical CPUs. This is what I'm seeing inside: root@zenon10:~# lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 32 On-line CPU(s) list: 0-31 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 8 NUMA node(s): 8 Vendor ID: AuthenticAMD CPU family: 23 Model: 1 Model name: AMD EPYC 7351 16-Core Processor Stepping: 2 CPU MHz: 2400.000 BogoMIPS: 4800.00 Virtualization: AMD-V Hypervisor vendor: KVM Virtualization type: full L1d cache: 64K L1i cache: 64K L2 cache: 512K NUMA node0 CPU(s): 0-3 NUMA node1 CPU(s): 4-7 NUMA node2 CPU(s): 8-11 NUMA node3 CPU(s): 12-15 NUMA node4 CPU(s): 16-19 NUMA node5 CPU(s): 20-23 NUMA node6 CPU(s): 24-27 NUMA node7 CPU(s): 28-31 This is virtual node configuration: (i tried different numatune settings but it was still the same) <domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>one-55782</name> <vcpu><![CDATA[32]]></vcpu> <cputune> <shares>32768</shares> </cputune> <memory>507904000</memory> <os> <type arch='x86_64'>hvm</type> </os> <devices> <emulator><![CDATA[/usr/bin/kvm]]></emulator> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.0'/> <target dev='vda'/> <driver name='qemu' type='qcow2' cache='unsafe'/> </disk> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.1'/> <target dev='vdc'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.2'/> <target dev='vdd'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.3'/> <target dev='vde'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> <disk type='file' device='cdrom'> <source file='/opt/opennebula/var/datastores/108/55782/disk.4'/> <target dev='vdb'/> <readonly/> <driver name='qemu' type='raw'/> </disk> <interface type='bridge'> <source bridge='br0'/> <mac address='02:00:93:fb:3b:78'/> <target dev='one-55782-0'/> <model type='virtio'/> <filterref filter='no-arp-mac-spoofing'> <parameter name='IP' value='147.251.59.120'/> </filterref> </interface> </devices> <features> <pae/> <acpi/> </features>  <cpu mode='host-passthrough'><topology sockets='8' cores='4' threads='1'/><numa><cell cpus='0-3' memory='62000000' /><cell cpus='4-7' memory='62000000' /><cell cpus='8-11' memory='62000000' /><cell cpus='12-15' memory='62000000' /><cell cpus='16-19' memory='62000000' /><cell cpus='20-23' memory='62000000' /><cell cpus='24-27' memory='62000000' /><cell cpus='28-31' memory='62000000' /></numa></cpu> <cputune><vcpupin vcpu='0' cpuset='0' /><vcpupin vcpu='1' cpuset='2' /><vcpupin vcpu='2' cpuset='4' /><vcpupin vcpu='3' cpuset='6' /><vcpupin vcpu='4' cpuset='8' /><vcpupin vcpu='5' cpuset='10' /><vcpupin vcpu='6' cpuset='12' /><vcpupin vcpu='7' cpuset='14' /><vcpupin vcpu='8' cpuset='16' /><vcpupin vcpu='9' cpuset='18' /><vcpupin vcpu='10' cpuset='20' /><vcpupin vcpu='11' cpuset='22' /><vcpupin vcpu='12' cpuset='24' /><vcpupin vcpu='13' cpuset='26' /><vcpupin vcpu='14' cpuset='28' /><vcpupin vcpu='15' cpuset='30' /><vcpupin vcpu='16' cpuset='1' /><vcpupin vcpu='17' cpuset='3' /><vcpupin vcpu='18' cpuset='5' /><vcpupin vcpu='19' cpuset='7' /><vcpupin vcpu='20' cpuset='9' /><vcpupin vcpu='21' cpuset='11' /><vcpupin vcpu='22' cpuset='13' /><vcpupin vcpu='23' cpuset='15' /><vcpupin vcpu='24' cpuset='17' /><vcpupin vcpu='25' cpuset='19' /><vcpupin vcpu='26' cpuset='21' /><vcpupin vcpu='27' cpuset='23' /><vcpupin vcpu='28' cpuset='25' /><vcpupin vcpu='29' cpuset='27' /><vcpupin vcpu='30' cpuset='29' /><vcpupin vcpu='31' cpuset='31' /></cputune> <numatune><memory mode='preferred' nodeset='0'/></numatune>) <devices><serial type='pty'><target port='0'/></serial><console type='pty'><target type='serial' port='0'/></console><channel type='pty'><target type='virtio' name='org.qemu.guest_agent.0'/></channel></devices> <devices><hostdev mode='subsystem' type='pci' managed='yes'><source><address domain='0x0' bus='0x11' slot='0x0' function='0x1'/></source></hostdev></devices> <devices><controller type='pci' index='1' model='pci-bridge'/><controller type='pci' index='2' model='pci-bridge'/><controller type='pci' index='3' model='pci-bridge'/><controller type='pci' index='4' model='pci-bridge'/><controller type='pci' index='5' model='pci-bridge'/></devices> <metadata> <system_datastore><![CDATA[/opt/opennebula/var/datastores/108/55782]]> </system_datastore> </metadata> </domain> If I run e.g., spec2017 on the virtual, I can see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1350 root 20 0 843136 830068 2524 R 78.1 0.2 513:16.16 bwaves_r_base.m 2456 root 20 0 804608 791264 2524 R 76.6 0.2 491:39.92 bwaves_r_base.m 4631 root 20 0 843136 829892 2344 R 75.8 0.2 450:16.04 bwaves_r_base.m 6441 root 20 0 802580 790212 2532 R 75.0 0.2 120:37.54 bwaves_r_base.m 7991 root 20 0 784676 772092 2576 R 75.0 0.2 387:15.39 bwaves_r_base.m 8142 root 20 0 843136 830044 2496 R 75.0 0.2 384:39.02 bwaves_r_base.m 8234 root 20 0 843136 830064 2524 R 75.0 0.2 99:04.48 bwaves_r_base.m 8578 root 20 0 749240 736604 2468 R 73.4 0.2 375:45.66 bwaves_r_base.m 9974 root 20 0 784676 771984 2468 R 73.4 0.2 348:01.36 bwaves_r_base.m 10396 root 20 0 802580 790264 2576 R 73.4 0.2 340:08.40 bwaves_r_base.m 12932 root 20 0 843136 830024 2480 R 73.4 0.2 288:39.76 bwaves_r_base.m 13113 root 20 0 784676 771864 2348 R 71.9 0.2 284:47.34 bwaves_r_base.m 13518 root 20 0 784676 762816 2540 R 71.9 0.2 276:31.58 bwaves_r_base.m 14443 root 20 0 784676 771984 2468 R 71.9 0.2 260:01.82 bwaves_r_base.m 12791 root 20 0 784676 772060 2544 R 70.3 0.2 291:43.96 bwaves_r_base.m 10544 root 20 0 843136 830068 2520 R 68.8 0.2 336:47.43 bwaves_r_base.m 15464 root 20 0 784676 762880 2608 R 60.9 0.2 239:19.14 bwaves_r_base.m 15487 root 20 0 784676 772048 2532 R 60.2 0.2 238:37.07 bwaves_r_base.m 16824 root 20 0 784676 772120 2604 R 55.5 0.2 212:10.92 bwaves_r_base.m 17255 root 20 0 843136 830012 2468 R 54.7 0.2 203:22.89 bwaves_r_base.m 17962 root 20 0 784676 772004 2488 R 54.7 0.2 188:26.07 bwaves_r_base.m 17505 root 20 0 843136 830068 2520 R 53.1 0.2 198:04.25 bwaves_r_base.m 27767 root 20 0 784676 771860 2344 R 52.3 0.2 592:25.95 bwaves_r_base.m 24458 root 20 0 843136 829888 2344 R 50.8 0.2 658:23.70 bwaves_r_base.m 30746 root 20 0 747376 735160 2604 R 43.0 0.2 556:47.67 bwaves_r_base.m The CPU TIME should be roughly the same but huge differences are obvious. This is what I see on the hypervizor: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 18201 oneadmin 20 0 474.0g 473.3g 1732 S 2459 94.0 33332:54 kvm 369 root 20 0 0 0 0 R 100.0 0.0 768:12.85 kswapd1 368 root 20 0 0 0 0 R 94.1 0.0 869:05.61 kswapd0 i.e., kswapd is eating whole CPU. Swap is turned off. [root@hde10 ~]# free total used free shared buff/cache available Mem: 528151432 503432580 1214048 34740 23504804 21907800 Swap: 0 0 0 Hypervisor is [root@hde10 ~]# cat /etc/redhat-release CentOS Linux release 7.5.1804 (Core) qemu-kvm-1.5.3-156.el7_5.5.x86_64 Virtual is Debian 9. Moreover, I'm using this type of disks for virtuals: <disk type='file' device='disk'> <source file='/opt/opennebula/var/datastores/108/55782/disk.3'/> <target dev='vde'/> <driver name='qemu' type='raw' cache='unsafe'/> </disk> If I keep cache='unsafe' and if I run iozone test on really big files (e.g., 8x 100GB), I can see huge cache pressure on the hypervizor - all 8 kswapd are running on 100 % percent and slowing things down. The disk under datastore is NVME SSD Intel 4500. If I set cache='none', kswaps are on idle, disk writes are pretty fast, however, with 8-NUMA configuration, writes slow down to less than 10MB/s as soon as the size of written data is roughly the same as memory size in the virtual node. iozone has 100 % CPU usage thereafter and it seems that it is traversing page lists. If I do the same with 1-NUMA configuration, everything is ok except performance penalty about 25 %. -- Lukáš Hejtmánek Linux Administrator only because Full Time Multitasking Ninja is not an official job title

2 7

[libvirt-users] Which objects does dynamic_ownership apply to?
by Milan Zamazal 20 Sep '18

20 Sep '18

Hi, I'm playing with dynamic ownership and not all objects have their owners changed. Is dynamic_ownership and its scope documented somewhere, besides the comment in qemu.conf? And what kinds of objects are handled by dynamic ownership? While some objects seem to be handled, other objects are apparently unaffected. For instance /dev/hwrng or a USB host device keep their root owners and are inaccessible to the VM. Is that expected or do I have anything wrong? Thanks, Milan

3 4

[libvirt-users] How to assign IP via virt-install
by Devis Reagan Ponraj -ERS, HCL Tech 17 Sep '18

17 Sep '18

Hi Team , I am new to virt-install , need your help pls . With below command I able to create a VM & I can see the mac address as well . virt-install --virt-type=kvm --name newclicentos --ram 6144 --vcpus=2 --import --network=bridge=br0,model=virtio --graphics vnc --disk path=centos7.0.qcow2,size=60,bus=virtio,format=qcow2 --os-variant centos7.0 --mac=00:00:00:00:01:01 With below command I am trying to assign an IP address , however it throws error . I dont want to give --location as I already have the vm build in centos7.0.qcow2 Is there a was I can assign static IP to my VM ? [root@localhost tmp]# virt-install --virt-type=kvm --name newclicentos --ram 6144 --vcpus=2 --import --network=bridge=br0,model=virtio --extra-args "ip=10.254.173.99::10.254.172.1::255.255.254.0::newclicentos:eth0:none" --mac=00:00:00:00:01:01 --graphics vnc --disk path=centos7.0.qcow2,size=60,bus=virtio,format=qcow2 ERROR --extra-args only work if specified with --location. Let me know if you need any info from me Thanks, Devis Reagan Regards, Devis Reagan HCL Technologies Ltd. www.hcl.com<http://www.hcl.com/> ::DISCLAIMER:: -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects. --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

2 2

[libvirt-users] Save VGA console log
by Naruto Nguyen 16 Sep '18

16 Sep '18

Hi everyone, Currently I boot my guest through VGA as LABEL vga KERNEL vmlinuz APPEND initrd=initrd console=tty0 rootfstype=ramfs If I boot through VGA tty0, is there any way to modify libvirt xml config that when a user type command "reboot" in VGA console, I can log all messages types from user in "console", could you please help? Thanks a lot Brs, Naruto

1 0

[libvirt-users] Secondary disk unattached if guest unresponsive and killed with 'virsh destroy'
by Mathew Crane 11 Sep '18

11 Sep '18

I have a very strange issue that has plagued our KVM-based hypervisor infrastructure. Environment: - All EL7-based libvirt releases since EL7.0 - Fully patched EL7-based KVM guest on a fully patched EL7-based KVM host. - Secondary disk attached via 'virsh attach-device guest /vm/guest-with-secondary-disk-01/disk2.xml --persistent' (note --persistent flag here) Problem Description: - Guest becomes unresponsive/hangs due to issue on guest. Typically the scenario is that PID1 hangs or goes into a bad state due to a misbehaving application. - Before virsh destroy, 'virsh dumpxml guest | grep vd[ab]' returns the expected filesystems. - Guest is destroyed from the hypervisor via 'virsh destroy guest'. - With guest still powered off, 'virsh dumpxml guest | grep vd[ab]' only shows 'vda'. 'vdb' now unattached. - To fix, requires re-run of 'virsh attach-device guest guest-with-secondary-disk-01/disk2.xml --persistent' Expected Behavior: - 'virsh destroy guest' does not drop the secondary disk attached to the guest if the node is in a hung state. Other notes: - If guest is not in a hung state, 'virsh destroy' does not drop the secondary disk. - 'virsh reset guest' does not exhibit the same behavior when guest is hung - vdb remains attached. - No logs produced in /var/log/libvirt indicating an issue, even with debug logging enabled. Is there any obvious I'm missing? Is this expected behavior? Guest XML via 'virsh dumpxml guest': <domain type='kvm' id='17'> <name>guest-with-secondary-disk-01</name> <uuid>ab578b51-3afd-4d01-86ab-a16b1b953919</uuid> <memory unit='KiB'>4194304</memory> <currentMemory unit='KiB'>4194304</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type> <boot dev='hd'/> <bootmenu enable='yes'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01.img'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/> <backingStore/> <target dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </disk> <controller type='usb' index='0' model='piix3-uhci'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </controller> <interface type='bridge'> <mac address='01:02:03:99:99:99'/> <source bridge='bridge_99'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='eth0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </interface> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/guest-with-secondary-disk-01.agent'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1' keymap='en-us'> <listen type='address' address='127.0.0.1'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1' primary='yes'/> <alias name='video0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <memballoon model='virtio'> <alias name='balloon0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </memballoon> <rng model='virtio'> <backend model='random'>/dev/random</backend> <alias name='rng0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </rng> </devices> <seclabel type='dynamic' model='selinux' relabel='yes'> <label>system_u:system_r:svirt_t:s0:c259,c495</label> <imagelabel>system_u:object_r:svirt_image_t:s0:c259,c495</imagelabel> </seclabel> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+107:+107</label> <imagelabel>+107:+107</imagelabel> </seclabel> </domain> Secondary disk XML: <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/vm/guest-with-secondary-disk-01/guest-with-secondary-disk-01_2.img'/> <target dev='vdb'/> </disk>

1 0