[libvirt] [PATCH 0/3] Libvirt RPC dispatching and unresponsive QEMU

If there is an unresponsive qemu process and libvirt access it's monitor, it will not get any response and this thread will block indefinitely, until the qemu process resumes or it's destroyed. If users continues executing APIs against that domain, libvirt will run out of worker threads and hangs (if those APIs will access monitor as well). Although, they will timeout in approx 30 seconds, which will free some workers, during that time is libvirt unable to process any request. Even worse - if the number of unresponsive qemu exceeds the size of worker thread pool, libvirt will hangs forever, and even restarting the daemon will not make it any better. This patch set heals the daemon on several levels, so nothing from written above will cause it to hangs: 1. RPC dispatching - all APIs are now annotated as 'high' or 'low' priority. Then a special thread pool is created. Low priority APIs will be still placed into usual pool, but high priority can be placed into this new pool if the former has no free worker. Which APIs should be marked high and which low? The splitting presented here is my guessing. It is not something written in stone, but from the logic of things it is not safe to annotate any API which is NOT guaranteed to end in reasonable small time as high priority call. 2. Job queue size limit - this sets bound on the number of threads blocked by a stuck qemu. Okay, there exists timeout on this, but if user application continue dispatching low priority calls it can still consume all (low priority) worker threads and therefore affect other users/VMs. Even if they timeout in approx 30 secs. 3. Run monitor re-connect in a separate thread per VM. If libvirtd is restarted, it tries to reconnect to all running qemu processes. This is potentially risky - one stuck qemu block daemon startup. However, putting the monitor startup code into one thread per VM allows libvirtd to startup, accept client connections and work with all VMs which monitor was successfully re-opened. Unresponsive qemu will hold job until we open the monitor. So clever user application can destroy such domain. All APIs requiring job will just fail in acquiring lock. Michal Privoznik (3): daemon: Create priority workers pool qemu: Introduce job queue size limit qemu: Deal with stucked qemu on daemon startup daemon/libvirtd.aug | 1 + daemon/libvirtd.c | 10 +- daemon/libvirtd.conf | 6 + daemon/remote.c | 26 ++ daemon/remote.h | 2 + src/qemu/libvirtd_qemu.aug | 1 + src/qemu/qemu.conf | 7 + src/qemu/qemu_conf.c | 4 + src/qemu/qemu_conf.h | 2 + src/qemu/qemu_domain.c | 17 ++ src/qemu/qemu_domain.h | 2 + src/qemu/qemu_driver.c | 23 +-- src/qemu/qemu_process.c | 89 ++++++- src/remote/qemu_protocol.x | 13 +- src/remote/remote_protocol.x | 544 +++++++++++++++++++++--------------------- src/rpc/gendispatch.pl | 48 ++++- src/rpc/virnetserver.c | 32 +++- src/rpc/virnetserver.h | 6 +- src/util/threadpool.c | 38 ++- src/util/threadpool.h | 1 + 20 files changed, 554 insertions(+), 318 deletions(-) -- 1.7.3.4