Re: [libvirt] [PATCH] daemon: Dynamically create worker threads when some get stuck

On Thu, Jun 16, 2011 at 06:29:09PM +0100, Daniel P. Berrange wrote: > On Thu, Jun 16, 2011 at 04:29:55PM +0200, Michal Privoznik wrote: > > Up to now, we've created new worker threads only during new connection. > > This patch monitors worker threads for liveness and dynamically create > > new one if all are stuck, waiting for hypervisor to reply. This > > situation can happen. All one need to do is send STOP signal to > qemu. We will also need to consider the case of qemu processes in uninterruptible sleep, e.g., in the case of a failed NFS mount.

> > The amount of time when we evaluate thread as stuck is defined in > > WORKER_TIMEOUT macro. > > > > With this approach we don't need to create new worker thread on incoming > > connection. However, as number of active worker threads grows, it might > > happen we need to size up the pool of worker threads and hence exceed > > the max_worker configuration value. > > This is really not desirable. The max_workers limit is in the > configuration as a static limit, to prevent client applications > from making libvirtd spawn an unlimited number of threads. We > must *always* respect the max_workers limit. > > I don't think automatically spawning workers is the right way > to deal with the QEMU issue anyway. As mentioned before, we need > to improve the QEMU monitor driver so that we can safely allow > monitor commands to time out Dan, can you suggest some possible strategies here? I don't have a strong opinion on the implementation, although I agree with your concern about spawning unlimited numbers of threads.