[PATCH] qemu: add process_exit_wait configuration setting
From: Jim Fehlig <jfehlig@suse.com> When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host. Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error. Signed-off-by: Jim Fehlig <jfehlig@suse.com> --- src/qemu/libvirtd_qemu.aug | 1 + src/qemu/qemu.conf.in | 12 ++++++++++++ src/qemu/qemu_conf.c | 3 +++ src/qemu/qemu_conf.h | 1 + src/qemu/qemu_process.c | 13 ++++++++++--- src/qemu/test_libvirtd_qemu.aug.in | 1 + 6 files changed, 28 insertions(+), 3 deletions(-) diff --git a/src/qemu/libvirtd_qemu.aug b/src/qemu/libvirtd_qemu.aug index 311992e441..dc3d6d61df 100644 --- a/src/qemu/libvirtd_qemu.aug +++ b/src/qemu/libvirtd_qemu.aug @@ -132,6 +132,7 @@ module Libvirtd_qemu = | bool_entry "dump_guest_core" | str_entry "stdio_handler" | int_entry "max_threads_per_process" + | int_entry "process_exit_wait" | str_entry "sched_core" let device_entry = bool_entry "mac_filter" diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0 +# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0 + + # If max_core is set to a non-zero integer, then QEMU will be # permitted to create core dumps when it crashes, provided its # RAM size is smaller than the limit set. diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index e30b146634..adcba137cc 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -934,6 +934,9 @@ virQEMUDriverConfigLoadProcessEntry(virQEMUDriverConfig *cfg, cfg->schedCore = val; } + if (virConfGetValueUInt(conf, "process_exit_wait", &cfg->processExitWait) < 0) + return -1; + return 0; } diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index 1d29f35c5d..bc78982123 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -211,6 +211,7 @@ struct _virQEMUDriverConfig { unsigned int maxProcesses; unsigned int maxFiles; unsigned int maxThreadsPerProc; + unsigned int processExitWait; unsigned long long maxCore; bool dumpGuestCore; diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index b2506edce0..5042ffe339 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -9218,6 +9218,9 @@ qemuProcessInShutdownStartMonitor(virDomainObj *vm) int qemuProcessKill(virDomainObj *vm, unsigned int flags) { + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(priv->driver); + unsigned int delay = cfg->processExitWait; int ret = -1; VIR_DEBUG("vm=%p name=%s pid=%lld flags=0x%x", @@ -9238,11 +9241,15 @@ qemuProcessKill(virDomainObj *vm, unsigned int flags) return 0; } - /* Request an extra delay of two seconds per current nhostdevs - * to be safe against stalls by the kernel freeing up the resources */ + /* If the administrator has not requested an explicit shutdown wait period, + * add an extra delay of two seconds per current nhostdevs to be safe against + * stalls by the kernel freeing up the resources */ + if (delay == 0) + delay = vm->def->nhostdevs * 2; + ret = virProcessKillPainfullyDelay(vm->pid, !!(flags & VIR_QEMU_PROCESS_KILL_FORCE), - vm->def->nhostdevs * 2, + delay, false); if (ret < 0 && (flags & VIR_QEMU_PROCESS_KILL_MONITOR_ON_ERROR)) { diff --git a/src/qemu/test_libvirtd_qemu.aug.in b/src/qemu/test_libvirtd_qemu.aug.in index c4cf9cf634..fb6cb1a1d4 100644 --- a/src/qemu/test_libvirtd_qemu.aug.in +++ b/src/qemu/test_libvirtd_qemu.aug.in @@ -97,6 +97,7 @@ module Test_libvirtd_qemu = { "max_processes" = "0" } { "max_files" = "0" } { "max_threads_per_process" = "0" } +{ "process_exit_wait" = "0" } { "max_core" = "unlimited" } { "dump_guest_core" = "1" } { "mac_filter" = "1" } -- 2.51.0
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when you are seeing such an error?
Signed-off-by: Jim Fehlig <jfehlig@suse.com> --- src/qemu/libvirtd_qemu.aug | 1 + src/qemu/qemu.conf.in | 12 ++++++++++++ src/qemu/qemu_conf.c | 3 +++ src/qemu/qemu_conf.h | 1 + src/qemu/qemu_process.c | 13 ++++++++++--- src/qemu/test_libvirtd_qemu.aug.in | 1 + 6 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/src/qemu/libvirtd_qemu.aug b/src/qemu/libvirtd_qemu.aug index 311992e441..dc3d6d61df 100644 --- a/src/qemu/libvirtd_qemu.aug +++ b/src/qemu/libvirtd_qemu.aug @@ -132,6 +132,7 @@ module Libvirtd_qemu = | bool_entry "dump_guest_core" | str_entry "stdio_handler" | int_entry "max_threads_per_process" + | int_entry "process_exit_wait" | str_entry "sched_core"
let device_entry = bool_entry "mac_filter" diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be vastly different and setting this globally will possibly be unable to satisfy new VMs without restart of the daemon. This ... if required ... should be a per-VM setting.
+ + # If max_core is set to a non-zero integer, then QEMU will be # permitted to create core dumps when it crashes, provided its # RAM size is smaller than the limit set. diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index e30b146634..adcba137cc 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -934,6 +934,9 @@ virQEMUDriverConfigLoadProcessEntry(virQEMUDriverConfig *cfg, cfg->schedCore = val; }
+ if (virConfGetValueUInt(conf, "process_exit_wait", &cfg->processExitWait) < 0) + return -1; + return 0; }
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index 1d29f35c5d..bc78982123 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -211,6 +211,7 @@ struct _virQEMUDriverConfig { unsigned int maxProcesses; unsigned int maxFiles; unsigned int maxThreadsPerProc; + unsigned int processExitWait; unsigned long long maxCore; bool dumpGuestCore;
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index b2506edce0..5042ffe339 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -9218,6 +9218,9 @@ qemuProcessInShutdownStartMonitor(virDomainObj *vm) int qemuProcessKill(virDomainObj *vm, unsigned int flags) { + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(priv->driver); + unsigned int delay = cfg->processExitWait; int ret = -1;
VIR_DEBUG("vm=%p name=%s pid=%lld flags=0x%x", @@ -9238,11 +9241,15 @@ qemuProcessKill(virDomainObj *vm, unsigned int flags) return 0; }
- /* Request an extra delay of two seconds per current nhostdevs - * to be safe against stalls by the kernel freeing up the resources */ + /* If the administrator has not requested an explicit shutdown wait period, + * add an extra delay of two seconds per current nhostdevs to be safe against + * stalls by the kernel freeing up the resources */ + if (delay == 0) + delay = vm->def->nhostdevs * 2; + ret = virProcessKillPainfullyDelay(vm->pid, !!(flags & VIR_QEMU_PROCESS_KILL_FORCE), - vm->def->nhostdevs * 2, + delay, false);
Can you please elaborate what the problem is? This function has additional code after this which waits for the process to terminate on background if killing didn't work so the process should be cleaned up. Adding extra delay will not cover all cases so that's why I'm not a fan of another tunable which users don't have a reasonable way of setting.
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when you are seeing such an error?
When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the host # virsh destroy 4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011 error: Failed to destroy domain '4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011' error: Failed to terminate process 71785 with SIGKILL: Device or resource busy
Signed-off-by: Jim Fehlig <jfehlig@suse.com> --- src/qemu/libvirtd_qemu.aug | 1 + src/qemu/qemu.conf.in | 12 ++++++++++++ src/qemu/qemu_conf.c | 3 +++ src/qemu/qemu_conf.h | 1 + src/qemu/qemu_process.c | 13 ++++++++++--- src/qemu/test_libvirtd_qemu.aug.in | 1 + 6 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/src/qemu/libvirtd_qemu.aug b/src/qemu/libvirtd_qemu.aug index 311992e441..dc3d6d61df 100644 --- a/src/qemu/libvirtd_qemu.aug +++ b/src/qemu/libvirtd_qemu.aug @@ -132,6 +132,7 @@ module Libvirtd_qemu = | bool_entry "dump_guest_core" | str_entry "stdio_handler" | int_entry "max_threads_per_process" + | int_entry "process_exit_wait" | str_entry "sched_core"
let device_entry = bool_entry "mac_filter" diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be vastly different and setting this globally will possibly be unable to satisfy new VMs without restart of the daemon.
I'm having difficulty understanding how that is possible...
This ... if required ... should be a per-VM setting.
Any suggestions on where such setting would be added? A 'destroy_timeout' attribute on the <on_poweroff> element could be a possiblity. But I look at it as encoding why the process exited, now how libvirt controls the exit. The only other idea I have is a top-level 'destroy_timeout' element. E.g. <domain> ... <destroy_timeout seconds='120'/> </domain>
+ + # If max_core is set to a non-zero integer, then QEMU will be # permitted to create core dumps when it crashes, provided its # RAM size is smaller than the limit set. diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index e30b146634..adcba137cc 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -934,6 +934,9 @@ virQEMUDriverConfigLoadProcessEntry(virQEMUDriverConfig *cfg, cfg->schedCore = val; }
+ if (virConfGetValueUInt(conf, "process_exit_wait", &cfg->processExitWait) < 0) + return -1; + return 0; }
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index 1d29f35c5d..bc78982123 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -211,6 +211,7 @@ struct _virQEMUDriverConfig { unsigned int maxProcesses; unsigned int maxFiles; unsigned int maxThreadsPerProc; + unsigned int processExitWait; unsigned long long maxCore; bool dumpGuestCore;
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index b2506edce0..5042ffe339 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -9218,6 +9218,9 @@ qemuProcessInShutdownStartMonitor(virDomainObj *vm) int qemuProcessKill(virDomainObj *vm, unsigned int flags) { + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(priv->driver); + unsigned int delay = cfg->processExitWait; int ret = -1;
VIR_DEBUG("vm=%p name=%s pid=%lld flags=0x%x", @@ -9238,11 +9241,15 @@ qemuProcessKill(virDomainObj *vm, unsigned int flags) return 0; }
- /* Request an extra delay of two seconds per current nhostdevs - * to be safe against stalls by the kernel freeing up the resources */ + /* If the administrator has not requested an explicit shutdown wait period, + * add an extra delay of two seconds per current nhostdevs to be safe against + * stalls by the kernel freeing up the resources */ + if (delay == 0) + delay = vm->def->nhostdevs * 2; + ret = virProcessKillPainfullyDelay(vm->pid, !!(flags & VIR_QEMU_PROCESS_KILL_FORCE), - vm->def->nhostdevs * 2, + delay, false);
Can you please elaborate what the problem is?
Hitting the following error after waiting 40s for a qemu process to disappear https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virprocess.c?ref_t...
This function has additional code after this which waits for the process to terminate on background if killing didn't work so the process should be cleaned up.
We've observed qemu process cleanup/exit times exceeding the 40s total wait time.
Adding extra delay will not cover all cases so that's why I'm not a fan of another tunable which users don't have a reasonable way of setting.
Personally, I felt a qemu.conf option was a reasonable approach. It maintains existing behavior and doesn't affect processes that exit within 40s, yet gives users a mechansim to handle those with long cleanup/exit times. I'm fine making this a per-VM setting if preferred. Also open to other options I may be overlooking, although simply increasing the existing hardcoded value is less appealing :-). Regards, Jim
On 9/1/26 00:53, Jim Fehlig wrote:
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when you are seeing such an error?
When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the host
We recommend to use hugepages to back large VMs, but for flexibility reasons and other constraints, users end up with these mid to large size VMs backed by normal memory. We have reports of this occuring on NUMA servers with even smaller VMs, f.e 512GiB. Normally with a fairly modern enterprise CPU and RAM this can take ~25s to terminate the QEMU process in isolation, but in certain conditions this can increase up to the 40 seconds. This was fine up until some time ago since the timeout was bugged and did not trigger: as the timeout was fixed, users started seeing their use cases break.
# virsh destroy 4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011 error: Failed to destroy domain '4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011' error: Failed to terminate process 71785 with SIGKILL: Device or resource busy
Signed-off-by: Jim Fehlig <jfehlig@suse.com> --- src/qemu/libvirtd_qemu.aug | 1 + src/qemu/qemu.conf.in | 12 ++++++++++++ src/qemu/qemu_conf.c | 3 +++ src/qemu/qemu_conf.h | 1 + src/qemu/qemu_process.c | 13 ++++++++++--- src/qemu/test_libvirtd_qemu.aug.in | 1 + 6 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/src/qemu/libvirtd_qemu.aug b/src/qemu/libvirtd_qemu.aug index 311992e441..dc3d6d61df 100644 --- a/src/qemu/libvirtd_qemu.aug +++ b/src/qemu/libvirtd_qemu.aug @@ -132,6 +132,7 @@ module Libvirtd_qemu = | bool_entry "dump_guest_core" | str_entry "stdio_handler" | int_entry "max_threads_per_process" + | int_entry "process_exit_wait" | str_entry "sched_core"
let device_entry = bool_entry "mac_filter" diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be vastly different and setting this globally will possibly be unable to satisfy new VMs without restart of the daemon.
FWIW, in my view it would be both more convenient and more meaningful for the admin to configure this as a single global parameter when planning the host configuration, resources and workloads. A restart of the daemon if the admin doesn't get it "right" the first time is not that bad imo, existing VMs continue to run, and libvirt daemons restart fairly quickly. The thing to keep in mind for an admin / provisioning rule / agent /... is the max size of a normal memory backed VM that could land on this host, and the worst case pressure the workloads on the host / socket can generate on the memory system, given the host architecture and topology.
I'm having difficulty understanding how that is possible...
This ... if required ... should be a per-VM setting.
Any suggestions on where such setting would be added? A 'destroy_timeout' attribute on the <on_poweroff> element could be a possiblity. But I look at it as encoding why the process exited, now how libvirt controls the exit. The only other idea I have is a top-level 'destroy_timeout' element. E.g.
<domain> ... <destroy_timeout seconds='120'/> </domain>
I don't think that the timeout depends fully on the VM configuration. The total amount of memory to unmap is one important multiplier that influences the total QEMU process shutdown time, but a huge constant can depend on the hardware platform (NUMA topology, cpu, memory controllers and bandwidth), and general host configuration (amount/size of static hugepages, THP setting, mitigations...). At the extreme end, if static 1G hugepages are configured and used, the process termination times become negligible in all realistic scenarios. So I am of the mind that when the users configure the host environment, decide on host kernel command line arguments, plan the memory resources for the host and the workloads to run on it (manually or automatically), that is the time where this global setting should also be set, in light of all the elements that could influence the QEMU process shutdown time. It can default to the previously hard-coded value. The same VM configuration ran on two different hosts (hardware and configuration), may behave very differently in terms of the QEMU process termination times. So if you think about live migration as well, the QEMU process can take much longer on one host to be terminated compared with another host, so one would have to re-adjust the VM configuration after live-migration? It does not seem to be a VM property.
+ + # If max_core is set to a non-zero integer, then QEMU will be # permitted to create core dumps when it crashes, provided its # RAM size is smaller than the limit set. diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index e30b146634..adcba137cc 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -934,6 +934,9 @@ virQEMUDriverConfigLoadProcessEntry(virQEMUDriverConfig *cfg, cfg->schedCore = val; }
+ if (virConfGetValueUInt(conf, "process_exit_wait", &cfg->processExitWait) < 0) + return -1; + return 0; }
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index 1d29f35c5d..bc78982123 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -211,6 +211,7 @@ struct _virQEMUDriverConfig { unsigned int maxProcesses; unsigned int maxFiles; unsigned int maxThreadsPerProc; + unsigned int processExitWait; unsigned long long maxCore; bool dumpGuestCore;
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index b2506edce0..5042ffe339 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -9218,6 +9218,9 @@ qemuProcessInShutdownStartMonitor(virDomainObj *vm) int qemuProcessKill(virDomainObj *vm, unsigned int flags) { + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(priv->driver); + unsigned int delay = cfg->processExitWait; int ret = -1;
VIR_DEBUG("vm=%p name=%s pid=%lld flags=0x%x", @@ -9238,11 +9241,15 @@ qemuProcessKill(virDomainObj *vm, unsigned int flags) return 0; }
- /* Request an extra delay of two seconds per current nhostdevs - * to be safe against stalls by the kernel freeing up the resources */ + /* If the administrator has not requested an explicit shutdown wait period, + * add an extra delay of two seconds per current nhostdevs to be safe against + * stalls by the kernel freeing up the resources */ + if (delay == 0) + delay = vm->def->nhostdevs * 2; + ret = virProcessKillPainfullyDelay(vm->pid, !!(flags & VIR_QEMU_PROCESS_KILL_FORCE), - vm->def->nhostdevs * 2, + delay, false);
Can you please elaborate what the problem is?
Hitting the following error after waiting 40s for a qemu process to disappear
https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virprocess.c?ref_t...
This function has additional code after this which waits for the process to terminate on background if killing didn't work so the process should be cleaned up.
We've observed qemu process cleanup/exit times exceeding the 40s total wait time.
Adding extra delay will not cover all cases so that's why I'm not a fan of another tunable which users don't have a reasonable way of setting.
I think that advanced users do have the elements necessary to set this to a reasonable value for their specific hardware, host os settings and based on their resource planning for the host; and even just empirically, this can be then adjusted after monitoring the qemu process termination times for their VMs.
Personally, I felt a qemu.conf option was a reasonable approach. It maintains existing behavior and doesn't affect processes that exit within 40s, yet gives users a mechansim to handle those with long cleanup/exit times.
I think this is reasonable too.
I'm fine making this a per-VM setting if preferred. Also open to other options I may be overlooking, although simply increasing the existing hardcoded value is less appealing :-).
I am skeptical of the per-VM setting, as mentioned above, as QEMU shutdown times can change substantially depending on host hardware and configuration.
Regards, Jim
Thanks! Claudio
On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote:
On 9/1/26 00:53, Jim Fehlig wrote:
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when you are seeing such an error?
When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the host
We recommend to use hugepages to back large VMs, but for flexibility reasons and other constraints, users end up with these mid to large size VMs backed by normal memory.
We have reports of this occuring on NUMA servers with even smaller VMs, f.e 512GiB.
Normally with a fairly modern enterprise CPU and RAM this can take ~25s to terminate the QEMU process in isolation, but in certain conditions this can increase up to the 40 seconds.
This was fine up until some time ago since the timeout was bugged and did not trigger: as the timeout was fixed, users started seeing their use cases break.
FWIW, IBM developed a feature in QEMU/libvirt to deal with this <async-teardown enabled='yes'/> an implication of this approach though is that the RAM is not immediately available for launching a new guest, and of course it requires per-guest config knobs to be enabled.
diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be vastly different and setting this globally will possibly be unable to satisfy new VMs without restart of the daemon.
FWIW, in my view it would be both more convenient and more meaningful for the admin to configure this as a single global parameter when planning the host configuration, resources and workloads.
A restart of the daemon if the admin doesn't get it "right" the first time is not that bad imo, existing VMs continue to run, and libvirt daemons restart fairly quickly.
The thing to keep in mind for an admin / provisioning rule / agent /... is the max size of a normal memory backed VM that could land on this host, and the worst case pressure the workloads on the host / socket can generate on the memory system, given the host architecture and topology.
The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect. Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach. We need to ensure libvirt "does the right thing" out of the box, as best as we can. IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz. Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On 9/1/26 11:35, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote:
On 9/1/26 00:53, Jim Fehlig wrote:
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when you are seeing such an error?
When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the host
We recommend to use hugepages to back large VMs, but for flexibility reasons and other constraints, users end up with these mid to large size VMs backed by normal memory.
We have reports of this occuring on NUMA servers with even smaller VMs, f.e 512GiB.
Normally with a fairly modern enterprise CPU and RAM this can take ~25s to terminate the QEMU process in isolation, but in certain conditions this can increase up to the 40 seconds.
This was fine up until some time ago since the timeout was bugged and did not trigger: as the timeout was fixed, users started seeing their use cases break.
FWIW, IBM developed a feature in QEMU/libvirt to deal with this
<async-teardown enabled='yes'/>
an implication of this approach though is that the RAM is not immediately available for launching a new guest, and of course it requires per-guest config knobs to be enabled.
diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be vastly different and setting this globally will possibly be unable to satisfy new VMs without restart of the daemon.
FWIW, in my view it would be both more convenient and more meaningful for the admin to configure this as a single global parameter when planning the host configuration, resources and workloads.
A restart of the daemon if the admin doesn't get it "right" the first time is not that bad imo, existing VMs continue to run, and libvirt daemons restart fairly quickly.
The thing to keep in mind for an admin / provisioning rule / agent /... is the max size of a normal memory backed VM that could land on this host, and the worst case pressure the workloads on the host / socket can generate on the memory system, given the host architecture and topology.
The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect.
Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach.
We need to ensure libvirt "does the right thing" out of the box, as best as we can.
Hi Daniel, in principle I agree it would be the best outcome, just seems hard to get it right.
IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz.
Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX.
Indeed. And like this there are many more factors that can change the result substantially (hugepages settings, NUMA, cpu and load state @memory controller, ...). Each one of these factors would need to be extracted or sampled and multiplied for the total RAM size. In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket. I am concerned that getting to the "right" number might prove impractical?
With regards, Daniel
Thanks! CLaudio
On Tue, Sep 01, 2026 at 12:48:32 +0200, Claudio Fontana wrote:
On 9/1/26 11:35, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote:
On 9/1/26 00:53, Jim Fehlig wrote:
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
[...]
The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect.
Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach.
We need to ensure libvirt "does the right thing" out of the box, as best as we can.
Hi Daniel,
in principle I agree it would be the best outcome, just seems hard to get it right.
IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz.
Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX.
Indeed. And like this there are many more factors that can change the result substantially (hugepages settings, NUMA, cpu and load state @memory controller, ...).
Each one of these factors would need to be extracted or sampled and multiplied for the total RAM size.
In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket.
I am concerned that getting to the "right" number might prove impractical?
The same thing applies to a config option too. And much more for a global one, as the size of the VM may differ. If there is no good way to determine the correct value, users who would want to set the config option will also have difficult time setting the correct value in the config option.
On 9/1/26 13:00, Peter Krempa wrote:
On Tue, Sep 01, 2026 at 12:48:32 +0200, Claudio Fontana wrote:
On 9/1/26 11:35, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote:
On 9/1/26 00:53, Jim Fehlig wrote:
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote: > From: Jim Fehlig <jfehlig@suse.com>
[...]
The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect.
Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach.
We need to ensure libvirt "does the right thing" out of the box, as best as we can.
Hi Daniel,
in principle I agree it would be the best outcome, just seems hard to get it right.
IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz.
Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX.
Indeed. And like this there are many more factors that can change the result substantially (hugepages settings, NUMA, cpu and load state @memory controller, ...).
Each one of these factors would need to be extracted or sampled and multiplied for the total RAM size.
In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket.
I am concerned that getting to the "right" number might prove impractical?
The same thing applies to a config option too. And much more for a global one, as the size of the VM may differ. If there is no good way to determine the correct value, users who would want to set the config option will also have difficult time setting the correct value in the config option.
Not the same thing at all. The difference being, the users have the actual hardware system to test on, they are in control of their host configuration, and _can_ sample and monitor system load and the QEMU process shutdown times _in their environment_. These are all things that are almost impossible to do from the libvirt code. And surely a global config is better than the magic hardcoded "let it be 40" number we have now? Thanks, Claudio
On Tue, Sep 01, 2026 at 12:48:32PM +0200, Claudio Fontana wrote:
On 9/1/26 11:35, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 10:11:04AM +0200, Claudio Fontana wrote:
On 9/1/26 00:53, Jim Fehlig wrote:
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when you are seeing such an error?
When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the host
We recommend to use hugepages to back large VMs, but for flexibility reasons and other constraints, users end up with these mid to large size VMs backed by normal memory.
We have reports of this occuring on NUMA servers with even smaller VMs, f.e 512GiB.
Normally with a fairly modern enterprise CPU and RAM this can take ~25s to terminate the QEMU process in isolation, but in certain conditions this can increase up to the 40 seconds.
This was fine up until some time ago since the timeout was bugged and did not trigger: as the timeout was fixed, users started seeing their use cases break.
FWIW, IBM developed a feature in QEMU/libvirt to deal with this
<async-teardown enabled='yes'/>
an implication of this approach though is that the RAM is not immediately available for launching a new guest, and of course it requires per-guest config knobs to be enabled.
diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be vastly different and setting this globally will possibly be unable to satisfy new VMs without restart of the daemon.
FWIW, in my view it would be both more convenient and more meaningful for the admin to configure this as a single global parameter when planning the host configuration, resources and workloads.
A restart of the daemon if the admin doesn't get it "right" the first time is not that bad imo, existing VMs continue to run, and libvirt daemons restart fairly quickly.
The thing to keep in mind for an admin / provisioning rule / agent /... is the max size of a normal memory backed VM that could land on this host, and the worst case pressure the workloads on the host / socket can generate on the memory system, given the host architecture and topology.
The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect.
Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach.
We need to ensure libvirt "does the right thing" out of the box, as best as we can.
Hi Daniel,
in principle I agree it would be the best outcome, just seems hard to get it right.
IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz.
Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX.
Indeed. And like this there are many more factors that can change the result substantially (hugepages settings, NUMA, cpu and load state @memory controller, ...).
Looking at the code I find we already introduced an extension for PCI devidces /* Request an extra delay of two seconds per current nhostdevs * to be safe against stalls by the kernel freeing up the resources */ ret = virProcessKillPainfullyDelay(vm->pid, !!(flags & VIR_QEMU_PROCESS_KILL_FORCE), vm->def->nhostdevs * 2, false);
Each one of these factors would need to be extracted or sampled and multiplied for the total RAM size.
In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket.
Effectively there is no "right" number at all, because "right" means wait long enough that a correctly operating QEMU exits, without waiting so long that a QEMU stuck in uninterruptible sleep will hang the Destroy API too long. We had gone for a relatively short timeout to avoid hung QEMU's delaying libvirt. We also have VIR_DOMAIN_DESTROY_GRACEFUL flag which lets us send SIGTERM but skip the SIGKILL part. IIUC the logic though, we still wait exactly the same amount of time in total, just don't send KILL.
I am concerned that getting to the "right" number might prove impractical?
Effectively we'll be forced to over-estimate the worst case timeout by a very significant factor, to minimize chance of false errors. I wonder what behaviour suits mgmt apps best ? If virDomainDestroy reaches the timeout and returns an error, it doesn't mean cleanup has failed. If a mgmt app is monitoring the running VM list, it will eventually see the VM go away. IOW, an error from virDomainDestroy isn't really a serious error in many cases - it is effectively more of a warning. The biggest challenge I see is there there is no way to understand if QEMU is "making progress" in shutting down, as opposed to become stuck. If you look at the underlying process, if QEMU were stuck in the "D" state that's bad, but if we've sent SIGKILL then it should be reaped by the kernel in all other states, so it is just a matter of waiting. Should we offer a VIR_DOMAIN_DESTROY_ASYNC flag such that * ASYNC -> send TERM, wait, send KILL, return success * ASYNC|GRACEFUL -> send TERM, wait, return success with the intent that applications simply monitor the VM list for ongoing status ? If it is stuck in "D" state it'll never go away, but that's something we can never solve in libvirt. If it is in any other state then as long as GRACEFUL was not set, you can be confident QEMU will go away eventually. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On 9/1/26 5:35 AM, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 12:48:32PM +0200, Claudio Fontana wrote: [...]
In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket.
Effectively there is no "right" number at all, because "right" means wait long enough that a correctly operating QEMU exits, without waiting so long that a QEMU stuck in uninterruptible sleep will hang the Destroy API too long.
We had gone for a relatively short timeout to avoid hung QEMU's delaying libvirt.
I think this is a worthy goal and we should avoid extending it if possible.
We also have VIR_DOMAIN_DESTROY_GRACEFUL flag which lets us send SIGTERM but skip the SIGKILL part. IIUC the logic though, we still wait exactly the same amount of time in total, just don't send KILL.
I am concerned that getting to the "right" number might prove impractical?
Effectively we'll be forced to over-estimate the worst case timeout by a very significant factor, to minimize chance of false errors.
I wonder what behaviour suits mgmt apps best ?
If virDomainDestroy reaches the timeout and returns an error, it doesn't mean cleanup has failed.
If a mgmt app is monitoring the running VM list, it will eventually see the VM go away.
This is effectively the current behavior, which I think is fine.
IOW, an error from virDomainDestroy isn't really a serious error in many cases - it is effectively more of a warning.
I suspect others, if encountering the EBUSY error, have treated it as a warning.
The biggest challenge I see is there there is no way to understand if QEMU is "making progress" in shutting down, as opposed to become stuck. If you look at the underlying process, if QEMU were stuck in the "D" state that's bad, but if we've sent SIGKILL then it should be reaped by the kernel in all other states, so it is just a matter of waiting.
Right. And we actually already do that in the shutdown case, which btw was the case for the reporter of the issue influencing this patch. Due to some miscommunication, we mistakenly started chasing timeouts in the destroy operation, when in fact the reporter (using libvirt 11.0.0) hit https://gitlab.com/libvirt/libvirt/-/work_items/853 by issuing 'systemctl poweroff' within the guest. In the end, huge facepalm on my part. Sorry for the disruption, but thanks for being vigilant against unneeded hacks :-). Self NACK for this patch. But perhaps not all time is wasted. During shutdown, a long-exiting QEMU could exceed the timeout, causing qemuProcessBeginStopJob() to fail and return early in processMonitorEOFEvent() https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_driver.c?ref_... None of the code from line 4190 through the rest of the function is executed. When the QEMU process finally disappears, processShutdownCompletedEvent() is eventually executed, where we only call qemuDomainRemoveInactive(). Do we miss sending a lifecycle event in this case? Regards, Jim
On Fri, Sep 04, 2026 at 03:07:30PM -0600, Jim Fehlig wrote:
On 9/1/26 5:35 AM, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 12:48:32PM +0200, Claudio Fontana wrote: [...]
In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket.
Effectively there is no "right" number at all, because "right" means wait long enough that a correctly operating QEMU exits, without waiting so long that a QEMU stuck in uninterruptible sleep will hang the Destroy API too long.
We had gone for a relatively short timeout to avoid hung QEMU's delaying libvirt.
I think this is a worthy goal and we should avoid extending it if possible.
We also have VIR_DOMAIN_DESTROY_GRACEFUL flag which lets us send SIGTERM but skip the SIGKILL part. IIUC the logic though, we still wait exactly the same amount of time in total, just don't send KILL.
I am concerned that getting to the "right" number might prove impractical?
Effectively we'll be forced to over-estimate the worst case timeout by a very significant factor, to minimize chance of false errors.
I wonder what behaviour suits mgmt apps best ?
If virDomainDestroy reaches the timeout and returns an error, it doesn't mean cleanup has failed.
If a mgmt app is monitoring the running VM list, it will eventually see the VM go away.
This is effectively the current behavior, which I think is fine.
IOW, an error from virDomainDestroy isn't really a serious error in many cases - it is effectively more of a warning.
I suspect others, if encountering the EBUSY error, have treated it as a warning.
The biggest challenge I see is there there is no way to understand if QEMU is "making progress" in shutting down, as opposed to become stuck. If you look at the underlying process, if QEMU were stuck in the "D" state that's bad, but if we've sent SIGKILL then it should be reaped by the kernel in all other states, so it is just a matter of waiting.
Right. And we actually already do that in the shutdown case, which btw was the case for the reporter of the issue influencing this patch. Due to some miscommunication, we mistakenly started chasing timeouts in the destroy operation, when in fact the reporter (using libvirt 11.0.0) hit
https://gitlab.com/libvirt/libvirt/-/work_items/853
by issuing 'systemctl poweroff' within the guest. In the end, huge facepalm on my part. Sorry for the disruption, but thanks for being vigilant against unneeded hacks :-). Self NACK for this patch.
But perhaps not all time is wasted. During shutdown, a long-exiting QEMU could exceed the timeout, causing qemuProcessBeginStopJob() to fail and return early in processMonitorEOFEvent()
https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_driver.c?ref_...
None of the code from line 4190 through the rest of the function is executed. When the QEMU process finally disappears, processShutdownCompletedEvent() is eventually executed, where we only call qemuDomainRemoveInactive(). Do we miss sending a lifecycle event in this case?
Not sure, it would have to be tested to validate it. With regards, Daniel -- |: https://berrange.com ~~ https://hachyderm.io/@berrange :| |: https://libvirt.org ~~ https://entangle-photo.org :| |: https://pixelfed.art/berrange ~~ https://fstop138.berrange.com :|
On 9/7/26 2:06 AM, Daniel P. Berrangé wrote:
On Fri, Sep 04, 2026 at 03:07:30PM -0600, Jim Fehlig wrote:
But perhaps not all time is wasted. During shutdown, a long-exiting QEMU could exceed the timeout, causing qemuProcessBeginStopJob() to fail and return early in processMonitorEOFEvent()
https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_driver.c?ref_...
None of the code from line 4190 through the rest of the function is executed. When the QEMU process finally disappears, processShutdownCompletedEvent() is eventually executed, where we only call qemuDomainRemoveInactive(). Do we miss sending a lifecycle event in this case?
Not sure, it would have to be tested to validate it.
I verified the event is missed and the audit entry is dropped. Patch sent to fix it https://lists.libvirt.org/archives/list/devel@lists.libvirt.org/thread/F3KTV... Regards, Jim
Back to thinking about this a bit more... On 9/1/26 3:35 AM, Daniel P. Berrangé wrote: [snip]
The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect.
Should we even be sending the KILL to a QEMU undergoing a "graceful" shutdown (e.g. 'virsh shutdown' or 'systemctl poweroff' within the guest)? It's fine for a destroy operation, but sending KILL to a QEMU in the middle of a proper cleanup/exit seems to open the window for corruption inside the guest filesystem.
Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach.
We need to ensure libvirt "does the right thing" out of the box, as best as we can.
IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz.
We could attempt to calculate a "safe" wait time before sending KILL, but I wonder if it's even needed in the graceful shutdown case? Would it be better to send TERM only? Doing so could result in behavior change. E.g. processes that previously needed the KILL to terminate would now remain in a "in shutdown" state. A subsequent destroy operation would be needed to issue the KILL.
Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX.
I hadn't considered CoCo guests. An ill-timed KILL sent during their cleanup/exit could be more undesirable than non-CoCo guests. Regards, Jim
On 9/9/26 23:35, Jim Fehlig via Devel wrote:
Back to thinking about this a bit more...
On 9/1/26 3:35 AM, Daniel P. Berrangé wrote:
[snip]
The whole TERM, wait 10 seconds, KILL, wait 30 seconds approach was designed from the POV that a normally behaving QEMU will "die" very quickly. IOW, any scenario where we reached the KILL stage was almost certainly a broken QEMU/kernel in some respect.
Should we even be sending the KILL to a QEMU undergoing a "graceful" shutdown (e.g. 'virsh shutdown' or 'systemctl poweroff' within the guest)? It's fine for a destroy operation, but sending KILL to a QEMU in the middle of a proper cleanup/exit seems to open the window for corruption inside the guest filesystem.
Clearly this is no longer a valid assumption. When "normal" behaviour or QEMU no longer matches libvirt's default mgmt action behaviour then I don't think a global qemu.conf setting or a per-VM setting is the ideal approach.
We need to ensure libvirt "does the right thing" out of the box, as best as we can.
IMHO, this suggests we need to dynamically increase our wait time before KILL based on the guest RAM size. eg Add 5 seconds for each 100 GB of small page RAM. I pulled that number out of the air, you would need to pick something better based on a typical system, plus some buffer/fuzz.
We could attempt to calculate a "safe" wait time before sending KILL, but I wonder if it's even needed in the graceful shutdown case? Would it be better to send TERM only? Doing so could result in behavior change. E.g. processes that previously needed the KILL to terminate would now remain in a "in shutdown" state. A subsequent destroy operation would be needed to issue the KILL.
Also I've noticed that TDX guests are painfully slow to teardown, even with tiny RAM sizes. So we might need to increase wait times even more when using TDX.
I hadn't considered CoCo guests. An ill-timed KILL sent during their cleanup/exit could be more undesirable than non-CoCo guests.
Regards, Jim
I was surprised to learn that libvirt sends SIGKILL in the graceful shutdown case as well after that grace time. I think it should be up to the user (or higher level platform) to decide when it is fed up waiting and issue a virsh destroy separately. So FWIW I would be in favor of not sending SIGKILL to a VM that is gracefully shutting down without user/platform requesting to do so. Ciao, CLaudio
participants (4)
-
Claudio Fontana -
Daniel P. Berrangé -
Jim Fehlig -
Peter Krempa