Re: [libvirt] Failed to terminate process 1275 with SIGTERM: Device or resource busy

On Tue, Jan 19, 2016 at 12:31:48PM +0100, Kashyap Chamarthy wrote: > On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote: > > On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote: > > > I tried another workaround which was to get virt-resize to fsync the > > > output file before closing the libvirt connection, but that doesn't > > > work for reasons I don't understand so far - still studying this. > > > > I worked out what was happening here -- I'd inserted the fsync at the > > wrong place in virt-resize. So I have now successfully worked around > > this for the virt-resize case, however it's still a problem that could > > manifest itself in other uses of libvirt + qemu + slow devices. > > We've seen the "Failed to terminate process 1275 with SIGTERM: Device or > resource busy" error occur in context of OpenStack as well[1][2]. > > The behavior is from virDomainDestroy() API (src/libvirt-domain.c): > > [...] > * virDomainDestroy first requests that a guest terminate (e.g. > * SIGTERM), then waits for it to comply. After a reasonable timeout, > * if the guest still exists, virDomainDestroy will forcefully > * terminate the guest (e.g. SIGKILL) if necessary (which may produce > * undesirable results, for example unflushed disk cache in the > * guest). To avoid this possibility, it's recommended to instead > * call virDomainDestroyFlags, sending the > * VIR_DOMAIN_DESTROY_GRACEFUL flag. > [...] > > Dan Berrange explains[1]: > > There are two reasons why you'd get this failure ("Failed to terminate > process: Device or resource busy") from libvirt. > > - The host is so overloaded that the kernel was not able to clean up > the process in the time that libvirt was prepared to wait. If this > is the case, the process should eventually go away on its own > after a short while longer and everything should return to normal > > - There is some problem, causing the process to get stuck in an > uninterruptable wait state. This is usually due to something going > wrong in the storage stack, causing some I/O read/write operation > to hang in kernel space. In this case the process will stay around > in the zombie state forever, or until the storage problem is > resolved. Thanks for finding this documentation. The problem with this theory is we are passing the VIR_DOMAIN_DESTROY_GRACEFUL flag, so that would indicate that this flag is buggy.