Re: [libvirt] [PATCH] qemu: Fix segmentation fault on reconnect

On Fri, Jun 08, 2018 at 12:49:07PM +0200, Marc Hartmayer wrote: > On start up of libvirtd the worker pool of the QEMU driver must be > initialized before trying to reconnect to all the running QEMU > instances. Otherwise segmentation faults can occur if there are QEMU > monitor events emitted. > #0 __GI___pthread_mutex_lock (mutex=mutex@entry=0x40) at ../nptl/pthread_mutex_lock.c:67 > #1 0x000003fffdba9e62 in virMutexLock (m=m@entry=0x40) at ../../src/util/virthread.c:89 > #2 0x000003fffdbab2dc in virThreadPoolSendJob (pool=0x0, priority=priority@entry=0, jobData=0x1000b7210) at ../../src/util/virthreadpool.c:386 > #3 0x000003ffd8343b70 in qemuProcessHandleSerialChanged (mon=<optimized out>, vm=0x3ff70074340, devAlias=<optimized out>, connected=<optimized out>, opaque=0x3ff70013f70) at ../../src/qemu/qemu_process.c:1480 > #4 0x000003ffd836a776 in qemuMonitorEmitSerialChange (mon=mon@entry=0x3ff500013f0, devAlias=devAlias@entry=0x1000b6940 "channel0", connected=<optimized out>) at ../../src/qemu/qemu_monitor.c:1558 > #5 0x000003ffd8378e52 in qemuMonitorJSONHandleSerialChange (mon=0x3ff500013f0, data=0x1000b79a0) at ../../src/qemu/qemu_monitor_json.c:1133 > #6 0x000003ffd8378930 in qemuMonitorJSONIOProcessEvent (mon=<optimized out>, mon@entry=0x3ff500013f0, obj=obj@entry=0x1000b6780) at ../../src/qemu/qemu_monitor_json.c:182 > #7 0x000003ffd837edee in qemuMonitorJSONIOProcessLine (mon=0x3ff500013f0, mon@entry=<error reading variable: value has been optimized out>, line=0x1000b8760 "{\"timestamp\": {\"seconds\": 1528444997, \"microseconds\": 125231}, \"event\": \"VSERPORT_CHANGE\", \"data\": {\"open\": false, \"id\": \"channel0\"}}", msg=0x3ff917f9850, msg@entry=<error reading variable: value has been optimized out>) at ../../src/qemu/qemu_monitor_json.c:211 > #8 0x000003ffd837ef86 in qemuMonitorJSONIOProcess (mon=<optimized out>, mon@entry=0x3ff500013f0, data=0x10008c4c0 "{\"return\": [{\"arch\": \"other\", \"current\": true, \"props\": {\"core-id\": 0}, \"CPU\": 0, \"qom_path\": \"/machine/unattached/device[0]\", \"halted\": false, \"thread_id\": 131501}, {\"arch\": \"other\", \"current\": false"..., len=477, msg=<optimized out>, msg@entry=0x3ff917f9850) at ../../src/qemu/qemu_monitor_json.c:253 > #9 0x000003ffd836757a in qemuMonitorIOProcess (mon=mon@entry=0x3ff500013f0) at ../../src/qemu/qemu_monitor.c:447 > #10 0x000003ffd836863e in qemuMonitorIO (watch=<error reading variable: value has been optimized out>, fd=<optimized out>, events=0, events@entry=<error reading variable: value has been optimized out>, opaque=0x3ff500013f0, opaque@entry=<error reading variable: value has been optimized out>) at ../../src/qemu/qemu_monitor.c:702 > #11 0x000003fffdb4033a in virEventPollDispatchHandles (nfds=<optimized out>, fds=0x1000b7160) at ../../src/util/vireventpoll.c:508 > #12 0x000003fffdb4055e in virEventPollRunOnce () at ../../src/util/vireventpoll.c:657 > #13 0x000003fffdb3e782 in virEventRunDefaultImpl () at ../../src/util/virevent.c:327 > #14 0x000003fffdc89400 in virNetDaemonRun (dmn=0x1000b2510) at ../../src/rpc/virnetdaemon.c:850 > #15 0x000000010002a816 in main (argc=<optimized out>, argv=<optimized out>) at ../../src/remote/remote_daemon.c:1460 > Signed-off-by: Marc Hartmayer <mhartmay(a)linux.ibm.com> Reviewed-by: Bjoern Walk <bwalk(a)linux.ibm.com> Reviewed-by: Boris Fiuczynski <fiuczy(a)linux.ibm.com> --- > src/qemu/qemu_driver.c | 7 +++++-- > 1 file changed, 5 insertions(+), 2 deletions(-) > diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c > index 971cf3c2750a..28769878cc58 100644 > --- a/src/qemu/qemu_driver.c > +++ b/src/qemu/qemu_driver.c > @@ -932,12 +932,15 @@ qemuStateInitialize(bool privileged, > qemuDomainManagedSaveLoad, > qemu_driver); > - qemuProcessReconnectAll(qemu_driver); > - > + /* must be initialized before trying to reconnect to all the > + * running domains since there might occur some QEMU monitor > + * events that will be dispatched to the worker pool */ > qemu_driver->workerPool = virThreadPoolNew(0, 1, 0, qemuProcessEventHandler, qemu_driver); > if (!qemu_driver->workerPool) > goto error; > + qemuProcessReconnectAll(qemu_driver); > + > virNWFilterRegisterCallbackDriver(&qemuCallbackDriver); > return 0; Wow, how could we have missed this for so long... Reviewed-by: Erik Skultety <eskultet(a)redhat.com PS: I'll strip some bits of the backtrace so it looks more compact in the history before pushing.