Re: [libvirt] Failed to terminate process 1275 with SIGTERM: Device or resource busy

On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote: We've seen the "Failed to terminate process 1275 with SIGTERM: Device or resource busy" error occur in context of OpenStack as well[1][2]. The behavior is from virDomainDestroy() API (src/libvirt-domain.c): [...] * virDomainDestroy first requests that a guest terminate (e.g. * SIGTERM), then waits for it to comply. After a reasonable timeout, * if the guest still exists, virDomainDestroy will forcefully * terminate the guest (e.g. SIGKILL) if necessary (which may produce * undesirable results, for example unflushed disk cache in the * guest). To avoid this possibility, it's recommended to instead * call virDomainDestroyFlags, sending the * VIR_DOMAIN_DESTROY_GRACEFUL flag. [...] Dan Berrange explains[1]: There are two reasons why you'd get this failure ("Failed to terminate process: Device or resource busy") from libvirt. - The host is so overloaded that the kernel was not able to clean up the process in the time that libvirt was prepared to wait. If this is the case, the process should eventually go away on its own after a short while longer and everything should return to normal - There is some problem, causing the process to get stuck in an uninterruptable wait state. This is usually due to something going wrong in the storage stack, causing some I/O read/write operation to hang in kernel space. In this case the process will stay around in the zombie state forever, or until the storage problem is resolved. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1205647 -- nova.virt.libvirt.driver fails to shutdown reboot instance with error 'Code=38 Error=Failed to terminate process 4260 with SIGKILL: Device or resource busy' [2] https://bugs.launchpad.net/nova/+bug/1353939 -- Rescue fails with 'Failed to terminate process: Device or resource busy' in the n-cpu log -- /kashyap