On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote: > I tried another workaround which was to get virt-resize to fsync the > output file before closing the libvirt connection, but that doesn't > work for reasons I don't understand so far - still studying this. I worked out what was happening here -- I'd inserted the fsync at the wrong place in virt-resize. So I have now successfully worked around this for the virt-resize case, however it's still a problem that could manifest itself in other uses of libvirt + qemu + slow devices.

On Tue, Jan 19, 2016 at 12:31:48PM +0100, Kashyap Chamarthy wrote: > On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote: > > On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote: > > > I tried another workaround which was to get virt-resize to fsync the > > > output file before closing the libvirt connection, but that doesn't > > > work for reasons I don't understand so far - still studying this. > > > > I worked out what was happening here -- I'd inserted the fsync at the > > wrong place in virt-resize. So I have now successfully worked around > > this for the virt-resize case, however it's still a problem that could > > manifest itself in other uses of libvirt + qemu + slow devices. > > We've seen the "Failed to terminate process 1275 with SIGTERM: Device or > resource busy" error occur in context of OpenStack as well[1][2]. > > The behavior is from virDomainDestroy() API (src/libvirt-domain.c): > > [...] > * virDomainDestroy first requests that a guest terminate (e.g. > * SIGTERM), then waits for it to comply. After a reasonable timeout, > * if the guest still exists, virDomainDestroy will forcefully > * terminate the guest (e.g. SIGKILL) if necessary (which may produce > * undesirable results, for example unflushed disk cache in the > * guest). To avoid this possibility, it's recommended to instead > * call virDomainDestroyFlags, sending the > * VIR_DOMAIN_DESTROY_GRACEFUL flag. > [...] > > Dan Berrange explains[1]: > > There are two reasons why you'd get this failure ("Failed to terminate > process: Device or resource busy") from libvirt. > > - The host is so overloaded that the kernel was not able to clean up > the process in the time that libvirt was prepared to wait. If this > is the case, the process should eventually go away on its own > after a short while longer and everything should return to normal > > - There is some problem, causing the process to get stuck in an > uninterruptable wait state. This is usually due to something going > wrong in the storage stack, causing some I/O read/write operation > to hang in kernel space. In this case the process will stay around > in the zombie state forever, or until the storage problem is > resolved. Thanks for finding this documentation. The problem with this theory is we are passing the VIR_DOMAIN_DESTROY_GRACEFUL flag, so that would indicate that this flag is buggy.

On Tue, Jan 19, 2016 at 01:43:58PM +0000, Daniel P. Berrange wrote: > I don't think it does. Passing GRACEFUL flag means libvirt will try > /less/ hard to kill QEMU, so it is /more/ likely that you will get > the > > "Failed to terminate process 1275 with SIGTERM: Device or resource busy" > > In general, that error message is something to be expected from the > virDomainDestroy() API, as we won't wait for death forever. If an > app wishes to wait forever, they should either re-issue the destroy > API call, or wait for an event notification of VIR_DOMAIN_EVENT_STOPPED > to arrive. IIUC, we should trap this specific error, and then basically loop if it happens? Is there a good way to trap specific errors without trying to match error message text?

Upstream OpenStack Nova employs a similar suggestion (calling the virDomainDestroy() API about 3 more times) proposed by Dan in this thread: http://git.openstack.org/cgit/openstack/nova/commit/?id=3907867 -- libvirt: handle code=38 + sigkill (ebusy) in destroy()