[libvirt] Failed to terminate process 1275 with SIGTERM: Device or resource busy
I didn't file a bug about this because it's not really clear what the bug is. Anyway, here goes ... $ virt-builder fedora-23 -o /dev/sdX [ 2.3] Downloading: http://libguestfs.org/download/builder/fedora-23.xz [ 3.0] Planning how to build this image [ 3.0] Uncompressing [ 18.1] Resizing (using virt-resize) to expand the disk to 14.9G virt-resize: error: libguestfs error: could not destroy libvirt domain: Failed to terminate process 1275 with SIGTERM: Device or resource busy [code=38 domain=0] $ ps ax | grep qemu 1275 ? Dl 0:49 /usr/bin/qemu-system-x86_64 -machine accel=kvm [...] What's actually happening here is that /dev/sdX is an incredibly slow[1] USB key. The virt-resize operation completes successfully, but qemu is blocked writing/fsyncing to the USB key, and takes a few minutes to exit. libvirt has a loop (src/util/virprocess.c:virProcessKillPainfully) which waits 15 seconds [comment says 10 seconds], and this wait is not configurable. There seem to be several possible solutions, none of them very nice: - Hard-code a longer wait in libvirt. - Make the wait in libvirt configurable. - If the process is in 'D' state, wait indefinitely. I tried another workaround which was to get virt-resize to fsync the output file before closing the libvirt connection, but that doesn't work for reasons I don't understand so far - still studying this. Rich. [1] "Incredibly slow" ... but it's just a regular USB key, they are all very slow! -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote:
I tried another workaround which was to get virt-resize to fsync the output file before closing the libvirt connection, but that doesn't work for reasons I don't understand so far - still studying this.
I worked out what was happening here -- I'd inserted the fsync at the wrong place in virt-resize. So I have now successfully worked around this for the virt-resize case, however it's still a problem that could manifest itself in other uses of libvirt + qemu + slow devices. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-p2v converts physical machines to virtual machines. Boot with a live CD or over the network (PXE) and turn machines into KVM guests. http://libguestfs.org/virt-v2v
On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote:
On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote:
I tried another workaround which was to get virt-resize to fsync the output file before closing the libvirt connection, but that doesn't work for reasons I don't understand so far - still studying this.
I worked out what was happening here -- I'd inserted the fsync at the wrong place in virt-resize. So I have now successfully worked around this for the virt-resize case, however it's still a problem that could manifest itself in other uses of libvirt + qemu + slow devices.
We've seen the "Failed to terminate process 1275 with SIGTERM: Device or resource busy" error occur in context of OpenStack as well[1][2]. The behavior is from virDomainDestroy() API (src/libvirt-domain.c): [...] * virDomainDestroy first requests that a guest terminate (e.g. * SIGTERM), then waits for it to comply. After a reasonable timeout, * if the guest still exists, virDomainDestroy will forcefully * terminate the guest (e.g. SIGKILL) if necessary (which may produce * undesirable results, for example unflushed disk cache in the * guest). To avoid this possibility, it's recommended to instead * call virDomainDestroyFlags, sending the * VIR_DOMAIN_DESTROY_GRACEFUL flag. [...] Dan Berrange explains[1]: There are two reasons why you'd get this failure ("Failed to terminate process: Device or resource busy") from libvirt. - The host is so overloaded that the kernel was not able to clean up the process in the time that libvirt was prepared to wait. If this is the case, the process should eventually go away on its own after a short while longer and everything should return to normal - There is some problem, causing the process to get stuck in an uninterruptable wait state. This is usually due to something going wrong in the storage stack, causing some I/O read/write operation to hang in kernel space. In this case the process will stay around in the zombie state forever, or until the storage problem is resolved. [1] https://bugzilla.redhat.com/show_bug.cgi?id=1205647 -- nova.virt.libvirt.driver fails to shutdown reboot instance with error 'Code=38 Error=Failed to terminate process 4260 with SIGKILL: Device or resource busy' [2] https://bugs.launchpad.net/nova/+bug/1353939 -- Rescue fails with 'Failed to terminate process: Device or resource busy' in the n-cpu log -- /kashyap
On Tue, Jan 19, 2016 at 12:31:48PM +0100, Kashyap Chamarthy wrote:
On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote:
On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote:
I tried another workaround which was to get virt-resize to fsync the output file before closing the libvirt connection, but that doesn't work for reasons I don't understand so far - still studying this.
I worked out what was happening here -- I'd inserted the fsync at the wrong place in virt-resize. So I have now successfully worked around this for the virt-resize case, however it's still a problem that could manifest itself in other uses of libvirt + qemu + slow devices.
We've seen the "Failed to terminate process 1275 with SIGTERM: Device or resource busy" error occur in context of OpenStack as well[1][2].
The behavior is from virDomainDestroy() API (src/libvirt-domain.c):
[...] * virDomainDestroy first requests that a guest terminate (e.g. * SIGTERM), then waits for it to comply. After a reasonable timeout, * if the guest still exists, virDomainDestroy will forcefully * terminate the guest (e.g. SIGKILL) if necessary (which may produce * undesirable results, for example unflushed disk cache in the * guest). To avoid this possibility, it's recommended to instead * call virDomainDestroyFlags, sending the * VIR_DOMAIN_DESTROY_GRACEFUL flag. [...]
Dan Berrange explains[1]:
There are two reasons why you'd get this failure ("Failed to terminate process: Device or resource busy") from libvirt.
- The host is so overloaded that the kernel was not able to clean up the process in the time that libvirt was prepared to wait. If this is the case, the process should eventually go away on its own after a short while longer and everything should return to normal
- There is some problem, causing the process to get stuck in an uninterruptable wait state. This is usually due to something going wrong in the storage stack, causing some I/O read/write operation to hang in kernel space. In this case the process will stay around in the zombie state forever, or until the storage problem is resolved.
Thanks for finding this documentation. The problem with this theory is we are passing the VIR_DOMAIN_DESTROY_GRACEFUL flag, so that would indicate that this flag is buggy. I think what we need is a test case, so here goes. Note you must run these steps as *non-root*. (1) Download the attachment to /var/tmp (2) chmod +x /var/tmp/qemu.sh (3) killall libvirtd ;# kills the session libvirtd (4) LIBGUESTFS_HV=/var/tmp/qemu.sh guestfish -N fs exit -vx You should see at the end of the output: libguestfs: calling virDomainDestroy "guestfs-q94hsiz89t8jp418" flags=VIR_DOMAIN_DESTROY_GRACEFUL [pause of a few seconds] libguestfs: error: could not destroy libvirt domain: Failed to terminate process 11412 with SIGTERM: Device or resource busy [code=38 domain=0] If someone else can reproduce this, then I will file a bug. Rich.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1205647 -- nova.virt.libvirt.driver fails to shutdown reboot instance with error 'Code=38 Error=Failed to terminate process 4260 with SIGKILL: Device or resource busy' [2] https://bugs.launchpad.net/nova/+bug/1353939 -- Rescue fails with 'Failed to terminate process: Device or resource busy' in the n-cpu log
-- /kashyap
-- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Tue, Jan 19, 2016 at 01:39:41PM +0000, Richard W.M. Jones wrote:
On Tue, Jan 19, 2016 at 12:31:48PM +0100, Kashyap Chamarthy wrote:
On Mon, Jan 18, 2016 at 04:19:58PM +0000, Richard W.M. Jones wrote:
On Mon, Jan 18, 2016 at 03:33:25PM +0000, Richard W.M. Jones wrote:
I tried another workaround which was to get virt-resize to fsync the output file before closing the libvirt connection, but that doesn't work for reasons I don't understand so far - still studying this.
I worked out what was happening here -- I'd inserted the fsync at the wrong place in virt-resize. So I have now successfully worked around this for the virt-resize case, however it's still a problem that could manifest itself in other uses of libvirt + qemu + slow devices.
We've seen the "Failed to terminate process 1275 with SIGTERM: Device or resource busy" error occur in context of OpenStack as well[1][2].
The behavior is from virDomainDestroy() API (src/libvirt-domain.c):
[...] * virDomainDestroy first requests that a guest terminate (e.g. * SIGTERM), then waits for it to comply. After a reasonable timeout, * if the guest still exists, virDomainDestroy will forcefully * terminate the guest (e.g. SIGKILL) if necessary (which may produce * undesirable results, for example unflushed disk cache in the * guest). To avoid this possibility, it's recommended to instead * call virDomainDestroyFlags, sending the * VIR_DOMAIN_DESTROY_GRACEFUL flag. [...]
Dan Berrange explains[1]:
There are two reasons why you'd get this failure ("Failed to terminate process: Device or resource busy") from libvirt.
- The host is so overloaded that the kernel was not able to clean up the process in the time that libvirt was prepared to wait. If this is the case, the process should eventually go away on its own after a short while longer and everything should return to normal
- There is some problem, causing the process to get stuck in an uninterruptable wait state. This is usually due to something going wrong in the storage stack, causing some I/O read/write operation to hang in kernel space. In this case the process will stay around in the zombie state forever, or until the storage problem is resolved.
Thanks for finding this documentation.
The problem with this theory is we are passing the VIR_DOMAIN_DESTROY_GRACEFUL flag, so that would indicate that this flag is buggy.
I don't think it does. Passing GRACEFUL flag means libvirt will try /less/ hard to kill QEMU, so it is /more/ likely that you will get the "Failed to terminate process 1275 with SIGTERM: Device or resource busy" In general, that error message is something to be expected from the virDomainDestroy() API, as we won't wait for death forever. If an app wishes to wait forever, they should either re-issue the destroy API call, or wait for an event notification of VIR_DOMAIN_EVENT_STOPPED to arrive. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Jan 19, 2016 at 01:43:58PM +0000, Daniel P. Berrange wrote:
I don't think it does. Passing GRACEFUL flag means libvirt will try /less/ hard to kill QEMU, so it is /more/ likely that you will get the
"Failed to terminate process 1275 with SIGTERM: Device or resource busy"
In general, that error message is something to be expected from the virDomainDestroy() API, as we won't wait for death forever. If an app wishes to wait forever, they should either re-issue the destroy API call, or wait for an event notification of VIR_DOMAIN_EVENT_STOPPED to arrive.
IIUC, we should trap this specific error, and then basically loop if it happens? Is there a good way to trap specific errors without trying to match error message text? Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW
On Tue, Jan 19, 2016 at 01:48:10PM +0000, Richard W.M. Jones wrote:
On Tue, Jan 19, 2016 at 01:43:58PM +0000, Daniel P. Berrange wrote:
I don't think it does. Passing GRACEFUL flag means libvirt will try /less/ hard to kill QEMU, so it is /more/ likely that you will get the
"Failed to terminate process 1275 with SIGTERM: Device or resource busy"
In general, that error message is something to be expected from the virDomainDestroy() API, as we won't wait for death forever. If an app wishes to wait forever, they should either re-issue the destroy API call, or wait for an event notification of VIR_DOMAIN_EVENT_STOPPED to arrive.
IIUC, we should trap this specific error, and then basically loop if it happens? Is there a good way to trap specific errors without trying to match error message text?
You can match on err.code==VIR_ERR_SYSTEM_ERROR & err.int1 == EBUSY The latter match is not entirely recommended, since if talking to a libvirtd on a remote host with different operating system to the client host, the errno values won't match, but I think it is fine for libguestfs which always uses a local libvirt. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Tue, Jan 19, 2016 at 01:39:41PM +0000, Richard W.M. Jones wrote: [...]
The problem with this theory is we are passing the VIR_DOMAIN_DESTROY_GRACEFUL flag, so that would indicate that this flag is buggy.
I think what we need is a test case, so here goes. Note you must run these steps as *non-root*.
(1) Download the attachment to /var/tmp
(2) chmod +x /var/tmp/qemu.sh
(3) killall libvirtd ;# kills the session libvirtd
(4) LIBGUESTFS_HV=/var/tmp/qemu.sh guestfish -N fs exit -vx
You should see at the end of the output:
libguestfs: calling virDomainDestroy "guestfs-q94hsiz89t8jp418" flags=VIR_DOMAIN_DESTROY_GRACEFUL [pause of a few seconds] libguestfs: error: could not destroy libvirt domain: Failed to terminate process 11412 with SIGTERM: Device or resource busy [code=38 domain=0]
If someone else can reproduce this, then I will file a bug.
Yep, I could perfectly reproduce your test case (thanks for constructing that): $ LIBGUESTFS_HV=/var/tmp/qemu.sh guestfish -N fs exit -vx [...] fsync /dev/sda guestfsd: main_loop: proc 282 (internal_autosync) took 0.00 seconds libguestfs: trace: internal_autosync = 0 libguestfs: calling virDomainDestroy "guestfs-liimlnged1e23l0l" flags=VIR_DOMAIN_DESTROY_GRACEFUL libguestfs: error: could not destroy libvirt domain: Failed to terminate process 27718 with SIGTERM: Device or resource busy [code=38 domain=0] libguestfs: trace: shutdown = -1 (error) libguestfs: trace: close [...] Upstream OpenStack Nova employs a similar suggestion (calling the virDomainDestroy() API about 3 more times) proposed by Dan in this thread: http://git.openstack.org/cgit/openstack/nova/commit/?id=3907867 -- libvirt: handle code=38 + sigkill (ebusy) in destroy() -- /kashyap
On Tue, Jan 19, 2016 at 03:39:22PM +0100, Kashyap Chamarthy wrote:
Upstream OpenStack Nova employs a similar suggestion (calling the virDomainDestroy() API about 3 more times) proposed by Dan in this thread:
http://git.openstack.org/cgit/openstack/nova/commit/?id=3907867 -- libvirt: handle code=38 + sigkill (ebusy) in destroy()
Thanks Kashyap, Daniel. I posted v3 of the patch which follows this same approach. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top
participants (3)
-
Daniel P. Berrange -
Kashyap Chamarthy -
Richard W.M. Jones