[libvirt] [PATCH] qemu: Fix deadlock when qemuDomainOpenConsole cleans up a connection
The new safe console handling introduced a possibility to deadlock the qemu driver when a new console connection forcibly disconnects a previous console stream that belongs to an already closed connection. The virStreamFree function calls subsequently a the virReleaseConnect function that tries to lock the driver while discarding the connection, but the driver was already locked in qemuDomainOpenConsole. Backtrace of the deadlocked thread: 0 0x00007f66e5aa7f14 in __lll_lock_wait () from /lib64/libpthread.so.0 1 0x00007f66e5aa3411 in _L_lock_500 () from /lib64/libpthread.so.0 2 0x00007f66e5aa322a in pthread_mutex_lock () from/lib64/libpthread.so.0 3 0x0000000000462bbd in qemudClose () 4 0x00007f66e6e178eb in virReleaseConnect () from/usr/lib64/libvirt.so.0 5 0x00007f66e6e19c8c in virUnrefStream () from /usr/lib64/libvirt.so.0 6 0x00007f66e6e3d1de in virStreamFree () from /usr/lib64/libvirt.so.0 7 0x00007f66e6e09a5d in virConsoleHashEntryFree () from/usr/lib64/libvirt.so.0 8 0x00007f66e6db7282 in virHashRemoveEntry () from/usr/lib64/libvirt.so.0 9 0x00007f66e6e09c4e in virConsoleOpen () from /usr/lib64/libvirt.so.0 10 0x00000000004526e9 in qemuDomainOpenConsole () 11 0x00007f66e6e421f1 in virDomainOpenConsole () from/usr/lib64/libvirt.so.0 12 0x00000000004361e4 in remoteDispatchDomainOpenConsoleHelper () 13 0x00007f66e6e80375 in virNetServerProgramDispatch () from/usr/lib64/libvirt.so.0 14 0x00007f66e6e7ae11 in virNetServerHandleJob () from/usr/lib64/libvirt.so.0 15 0x00007f66e6da897d in virThreadPoolWorker () from/usr/lib64/libvirt.so.0 16 0x00007f66e6da7ff6 in virThreadHelper () from/usr/lib64/libvirt.so.0 17 0x00007f66e5aa0c5c in start_thread () from /lib64/libpthread.so.0 18 0x00007f66e57e7fcd in clone () from /lib64/libc.so.6 * src/qemu/qemu_driver.c: qemuDomainOpenConsole() -- unlock the qemu driver right after acquiring the domain object --- Found while writing tests for the libvirt-test-API, its 100% reproducible: 1: Start a domain with serial console and run "cat" on this console 2: download python reproducer: http://files.pipo.sk/console_deadlock.py 3: set guest name in the script 4: run the reproducer and "virsh console 'guestname' --force" after that 5: <deadlocked> src/qemu/qemu_driver.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-) diff --git a/src/qemu/qemu_driver.c b/src/qemu/qemu_driver.c index d9e35be..e18e72d 100644 --- a/src/qemu/qemu_driver.c +++ b/src/qemu/qemu_driver.c @@ -11494,6 +11494,7 @@ qemuDomainOpenConsole(virDomainPtr dom, qemuDriverLock(driver); virUUIDFormat(dom->uuid, uuidstr); vm = virDomainFindByUUID(&driver->domains, dom->uuid); + qemuDriverUnlock(driver); if (!vm) { qemuReportError(VIR_ERR_NO_DOMAIN, _("no domain with matching uuid '%s'"), uuidstr); @@ -11558,7 +11559,6 @@ qemuDomainOpenConsole(virDomainPtr dom, cleanup: if (vm) virDomainObjUnlock(vm); - qemuDriverUnlock(driver); return ret; } -- 1.7.3.4
On 04/10/2012 07:29 AM, Peter Krempa wrote:
The new safe console handling introduced a possibility to deadlock the qemu driver when a new console connection forcibly disconnects a previous console stream that belongs to an already closed connection.
The virStreamFree function calls subsequently a the virReleaseConnect function that tries to lock the driver while discarding the connection, but the driver was already locked in qemuDomainOpenConsole.
* src/qemu/qemu_driver.c: qemuDomainOpenConsole() -- unlock the qemu driver right after acquiring the domain object --- Found while writing tests for the libvirt-test-API, its 100% reproducible:
1: Start a domain with serial console and run "cat" on this console 2: download python reproducer: http://files.pipo.sk/console_deadlock.py 3: set guest name in the script 4: run the reproducer and "virsh console 'guestname' --force" after that 5: <deadlocked>
src/qemu/qemu_driver.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
ACK; the rest of the function does not refer to driver, so moving the unlock sooner is safe. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
On 04/10/2012 05:43 PM, Eric Blake wrote:
On 04/10/2012 07:29 AM, Peter Krempa wrote:
The new safe console handling introduced a possibility to deadlock the qemu driver when a new console connection forcibly disconnects a previous console stream that belongs to an already closed connection.
The virStreamFree function calls subsequently a the virReleaseConnect function that tries to lock the driver while discarding the connection, but the driver was already locked in qemuDomainOpenConsole.
* src/qemu/qemu_driver.c: qemuDomainOpenConsole() -- unlock the qemu driver right after acquiring the domain object --- src/qemu/qemu_driver.c | 2 +- 1 files changed, 1 insertions(+), 1 deletions(-)
ACK; the rest of the function does not refer to driver, so moving the unlock sooner is safe.
Thanks, pushed. Peter
participants (2)
-
Eric Blake -
Peter Krempa