Re: [libvirt] [PATCH] Fix bug libvirt daemon segfault when new force console vm session break down existed console session. When force console vm command arrived, libvirtd will break down existed console session, and in this procedure, it will call daemonStreamEvent() to release resources , so daemonStreamFilter() need to check if stream filter existed when get client object lock and client->privData's lock, if not existed, just return -1

On 11/25/19 11:53 AM, Lance Liu wrote: > We first produce this bug in rhel7.4's libvir daemon。For easily > produce the bug, the step can be as follows: 1. add sleep(3) in > daemonStreamFilter() pre virMutexLock(&stream->priv->lock), then > build libvirtd bin executable, then restart libvirtd 2. use virsh > console open one vm's console, for this console（the vm's kernel need > console=ttyS0 boot parameter,then just input from keyboard on and on > 3. use virsh console --force to break the previous console session, > than you will get libvirt daemon deadlock。 > > And for the problem client->privData to be released problem, only > virNetServerClientClose() will free client->privData and client, I > think this can be fixed by remove virNetServerClientClose() from > daemonStreamEvent(), and replace with > virNetServerClientImmediateClose(), so virNetServerProcessClients() > will test the session would be closed。 RHEL-7.4 you say? That's libvirt-3.2.0 which is far from what we have on the master. Also, RHEL-7.4 itself is kind of old as RHEL-7.7 was released during summer. At least that explains why your original patch was against older libvirt. When I revert your patch I can see the deadlock still happening: Thread 5 (Thread 0x7feac8936700 (LWP 162328)): #0 0x00007feacd1d6f6c in __lll_lock_wait () at /lib64/libpthread.so.0 #1 0x00007feacd1cf194 in pthread_mutex_lock () at /lib64/libpthread.so.0 #2 0x00007feacdc78a9e in virMutexLock (m=0x565030303700) at ../../src/util/virthread.c:79 #3 0x00007feacdc488a2 in virObjectLock (anyobj=0x5650303036f0) at ../../src/util/virobject.c:433 #4 0x00007feacdd9f8bc in virNetServerClientSendMessage (client=0x5650303036f0, msg=0x7feab40281e0) at ../../src/rpc/virnetserverclient.c:1532 #5 0x00007feacdd9ac27 in virNetServerProgramSendError (program=536903814, version=1, client=0x5650303036f0, msg=0x7feab40281e0, rerr=0x7feac8935730, procedure=201, type=3, serial=16) at ../../src/rpc/virnetserverprogram.c:168 #6 0x00007feacdd9ad52 in virNetServerProgramSendStreamError (prog=0x5650302ccb70, client=0x5650303036f0, msg=0x7feab40281e0, rerr=0x7feac8935730, procedure=201, serial=16) at ../../src/rpc/virnetserverprogram.c:217 #7 0x000056502e9bbd24 in daemonStreamEvent (st=0x7feac40022a0, events=4, opaque=0x5650303036f0) at ../../src/remote/remote_daemon_stream.c:256 #8 0x00007feacdc01648 in virFDStreamCloseInt (st=0x7feac40022a0, streamAbort=true) at ../../src/util/virfdstream.c:715 #9 0x00007feacdc017d1 in virFDStreamAbort (st=0x7feac40022a0) at ../../src/util/virfdstream.c:759 #10 0x00007feacded9e1d in virStreamAbort (stream=0x7feac40022a0) at ../../src/libvirt-stream.c:1237 #11 0x00007feacdd456e3 in virChrdevOpen (devs=0x7fea740cae00, source=0x7fea740d26a0, st=0x7feab402f220, force=true) at ../../src/conf/virchrdev.c:387 #12 0x00007feab3f722e7 in qemuDomainOpenConsole (dom=0x7feab40329b0, dev_name=0x0, st=0x7feab402f220, flags=1) at ../../src/qemu/qemu_driver.c:17309 #13 0x00007feacdeb375f in virDomainOpenConsole (dom=0x7feab40329b0, dev_name=0x0, st=0x7feab402f220, flags=1) at ../../src/libvirt-domain.c:9662 #14 0x000056502e999894 in remoteDispatchDomainOpenConsole (server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0, rerr=0x7feac8935ad0, args=0x7feab402cd70) at ./remote/remote_daemon_dispatch_stubs.h:9211 #15 0x000056502e99976f in remoteDispatchDomainOpenConsoleHelper (server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0, rerr=0x7feac8935ad0, args=0x7feab402cd70, ret=0x0) at ./remote/remote_daemon_dispatch_stubs.h:9178 #16 0x00007feacdd9b4b1 in virNetServerProgramDispatchCall (prog=0x5650302ccb70, server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0) at ../../src/rpc/virnetserverprogram.c:430 #17 0x00007feacdd9b026 in virNetServerProgramDispatch (prog=0x5650302ccb70, server=0x56503027fc30, client=0x56503031ad70, msg=0x565030304eb0) at ../../src/rpc/virnetserverprogram.c:302 #18 0x00007feacdda1e34 in virNetServerProcessMsg (srv=0x56503027fc30, client=0x56503031ad70, prog=0x5650302ccb70, msg=0x565030304eb0) at ../../src/rpc/virnetserver.c:136 #19 0x00007feacdda1ef4 in virNetServerHandleJob (jobOpaque=0x565030336ea0, opaque=0x56503027fc30) at ../../src/rpc/virnetserver.c:153 #20 0x00007feacdc79800 in virThreadPoolWorker (opaque=0x56503028a1f0) at ../../src/util/virthreadpool.c:163 #21 0x00007feacdc78dad in virThreadHelper (data=0x56503028afd0) at ../../src/util/virthread.c:196 #22 0x00007feacd1cc458 in start_thread () at /lib64/libpthread.so.0 #23 0x00007feacd0fa6ef in clone () at /lib64/libc.so.6 Thread 1 (Thread 0x7feaca8a7dc0 (LWP 162311)): #0 0x00007feacd1d6f6c in __lll_lock_wait () at /lib64/libpthread.so.0 #1 0x00007feacd1cf194 in pthread_mutex_lock () at /lib64/libpthread.so.0 #2 0x00007feacdc78a9e in virMutexLock (m=0x5650302ff110) at ../../src/util/virthread.c:79 #3 0x000056502e9bbde3 in daemonStreamFilter (client=0x5650303036f0, msg=0x5650303097d0, opaque=0x7feac4003020) at ../../src/remote/remote_daemon_stream.c:297 #4 0x00007feacdd9f03b in virNetServerClientDispatchRead (client=0x5650303036f0) at ../../src/rpc/virnetserverclient.c:1288 #5 0x00007feacdd9f6ab in virNetServerClientDispatchEvent (sock=0x565030309a90, events=1, opaque=0x5650303036f0) at ../../src/rpc/virnetserverclient.c:1485 #6 0x00007feacdd907f9 in virNetSocketEventHandle (watch=19, fd=28, events=1, opaque=0x565030309a90) at ../../src/rpc/virnetsocket.c:2177 #7 0x00007feacdbff32d in virEventPollDispatchHandles (nfds=16, fds=0x5650302bf200) at ../../src/util/vireventpoll.c:503 #8 0x00007feacdbffb86 in virEventPollRunOnce () at ../../src/util/vireventpoll.c:658 #9 0x00007feacdbfdafb in virEventRunDefaultImpl () at ../../src/util/virevent.c:322 #10 0x00007feacdda1a0d in virNetDaemonRun (dmn=0x56503027f880) at ../../src/rpc/virnetdaemon.c:836 #11 0x000056502e98843f in main (argc=2, argv=0x7ffc308703a8) at ../../src/remote/remote_daemon.c:1427 But I think I have an idea how to fix this. Will post patches later today. Michal