On 9/1/26 00:53, Jim Fehlig wrote:
On 8/31/26 9:34 AM, Peter Krempa wrote:
On Fri, Aug 28, 2026 at 14:16:17 -0600, Jim Fehlig via Devel wrote:
From: Jim Fehlig <jfehlig@suse.com>
When shutting down a VM, libvirt sends the associated QEMU process SIGTERM, waits 10 seconds for it to disappear, optionally sends SIGKILL, then waits up to another 30 seconds for the process to exit before reporting and returning an error. Commit be2ca04447 added 2 seconds per assigned host device to the total time libvirt waits for a QEMU process to terminate. Other scenarios than the one described in be2ca04447 could delay the clean exit of QEMU, e.g. reclaiming memory of VMs with large memory allocations backed by 4k pages on the host.
Instead of trying to cover all such scenarios based on VM configuration, introduce a 'process_exit_wait' setting in qemu.conf to control how much additional time (in seconds) libvirt will wait for a QEMU process to terminate before reporting an error.
You are stating that an error is reported. Can you please elaborate when you are seeing such an error?
When e.g. destroying a 900GB VM whose memory is backed by 4k pages on the host
We recommend to use hugepages to back large VMs, but for flexibility reasons and other constraints, users end up with these mid to large size VMs backed by normal memory. We have reports of this occuring on NUMA servers with even smaller VMs, f.e 512GiB. Normally with a fairly modern enterprise CPU and RAM this can take ~25s to terminate the QEMU process in isolation, but in certain conditions this can increase up to the 40 seconds. This was fine up until some time ago since the timeout was bugged and did not trigger: as the timeout was fixed, users started seeing their use cases break.
# virsh destroy 4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011 error: Failed to destroy domain '4ff7e1ea-a4d4-458b-8b27-a7b67d7b8011' error: Failed to terminate process 71785 with SIGKILL: Device or resource busy
Signed-off-by: Jim Fehlig <jfehlig@suse.com> --- src/qemu/libvirtd_qemu.aug | 1 + src/qemu/qemu.conf.in | 12 ++++++++++++ src/qemu/qemu_conf.c | 3 +++ src/qemu/qemu_conf.h | 1 + src/qemu/qemu_process.c | 13 ++++++++++--- src/qemu/test_libvirtd_qemu.aug.in | 1 + 6 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/src/qemu/libvirtd_qemu.aug b/src/qemu/libvirtd_qemu.aug index 311992e441..dc3d6d61df 100644 --- a/src/qemu/libvirtd_qemu.aug +++ b/src/qemu/libvirtd_qemu.aug @@ -132,6 +132,7 @@ module Libvirtd_qemu = | bool_entry "dump_guest_core" | str_entry "stdio_handler" | int_entry "max_threads_per_process" + | int_entry "process_exit_wait" | str_entry "sched_core"
let device_entry = bool_entry "mac_filter" diff --git a/src/qemu/qemu.conf.in b/src/qemu/qemu.conf.in index 97b0141cf6..2f24059053 100644 --- a/src/qemu/qemu.conf.in +++ b/src/qemu/qemu.conf.in @@ -844,6 +844,18 @@ #max_threads_per_process = 0
+# When shutting down a VM, libvirt will wait up to 40 seconds for +# the associated QEMU process to exit before reporting an error. +# For some VM configurations, QEMU might require more time to +# cleanup and exit, e.g. VMs with very large memory allocations. +# +# If process_exit_wait is set to a positive interger, libvirt +# will use the value as additional time to wait for the QEMU +# process to exit before reporting it cannot be terminated. +# +#process_exit_wait = 0
I don't like this as a global option. The specifics of a VM can be vastly different and setting this globally will possibly be unable to satisfy new VMs without restart of the daemon.
FWIW, in my view it would be both more convenient and more meaningful for the admin to configure this as a single global parameter when planning the host configuration, resources and workloads. A restart of the daemon if the admin doesn't get it "right" the first time is not that bad imo, existing VMs continue to run, and libvirt daemons restart fairly quickly. The thing to keep in mind for an admin / provisioning rule / agent /... is the max size of a normal memory backed VM that could land on this host, and the worst case pressure the workloads on the host / socket can generate on the memory system, given the host architecture and topology.
I'm having difficulty understanding how that is possible...
This ... if required ... should be a per-VM setting.
Any suggestions on where such setting would be added? A 'destroy_timeout' attribute on the <on_poweroff> element could be a possiblity. But I look at it as encoding why the process exited, now how libvirt controls the exit. The only other idea I have is a top-level 'destroy_timeout' element. E.g.
<domain> ... <destroy_timeout seconds='120'/> </domain>
I don't think that the timeout depends fully on the VM configuration. The total amount of memory to unmap is one important multiplier that influences the total QEMU process shutdown time, but a huge constant can depend on the hardware platform (NUMA topology, cpu, memory controllers and bandwidth), and general host configuration (amount/size of static hugepages, THP setting, mitigations...). At the extreme end, if static 1G hugepages are configured and used, the process termination times become negligible in all realistic scenarios. So I am of the mind that when the users configure the host environment, decide on host kernel command line arguments, plan the memory resources for the host and the workloads to run on it (manually or automatically), that is the time where this global setting should also be set, in light of all the elements that could influence the QEMU process shutdown time. It can default to the previously hard-coded value. The same VM configuration ran on two different hosts (hardware and configuration), may behave very differently in terms of the QEMU process termination times. So if you think about live migration as well, the QEMU process can take much longer on one host to be terminated compared with another host, so one would have to re-adjust the VM configuration after live-migration? It does not seem to be a VM property.
+ + # If max_core is set to a non-zero integer, then QEMU will be # permitted to create core dumps when it crashes, provided its # RAM size is smaller than the limit set. diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c index e30b146634..adcba137cc 100644 --- a/src/qemu/qemu_conf.c +++ b/src/qemu/qemu_conf.c @@ -934,6 +934,9 @@ virQEMUDriverConfigLoadProcessEntry(virQEMUDriverConfig *cfg, cfg->schedCore = val; }
+ if (virConfGetValueUInt(conf, "process_exit_wait", &cfg->processExitWait) < 0) + return -1; + return 0; }
diff --git a/src/qemu/qemu_conf.h b/src/qemu/qemu_conf.h index 1d29f35c5d..bc78982123 100644 --- a/src/qemu/qemu_conf.h +++ b/src/qemu/qemu_conf.h @@ -211,6 +211,7 @@ struct _virQEMUDriverConfig { unsigned int maxProcesses; unsigned int maxFiles; unsigned int maxThreadsPerProc; + unsigned int processExitWait; unsigned long long maxCore; bool dumpGuestCore;
diff --git a/src/qemu/qemu_process.c b/src/qemu/qemu_process.c index b2506edce0..5042ffe339 100644 --- a/src/qemu/qemu_process.c +++ b/src/qemu/qemu_process.c @@ -9218,6 +9218,9 @@ qemuProcessInShutdownStartMonitor(virDomainObj *vm) int qemuProcessKill(virDomainObj *vm, unsigned int flags) { + qemuDomainObjPrivate *priv = vm->privateData; + g_autoptr(virQEMUDriverConfig) cfg = virQEMUDriverGetConfig(priv->driver); + unsigned int delay = cfg->processExitWait; int ret = -1;
VIR_DEBUG("vm=%p name=%s pid=%lld flags=0x%x", @@ -9238,11 +9241,15 @@ qemuProcessKill(virDomainObj *vm, unsigned int flags) return 0; }
- /* Request an extra delay of two seconds per current nhostdevs - * to be safe against stalls by the kernel freeing up the resources */ + /* If the administrator has not requested an explicit shutdown wait period, + * add an extra delay of two seconds per current nhostdevs to be safe against + * stalls by the kernel freeing up the resources */ + if (delay == 0) + delay = vm->def->nhostdevs * 2; + ret = virProcessKillPainfullyDelay(vm->pid, !!(flags & VIR_QEMU_PROCESS_KILL_FORCE), - vm->def->nhostdevs * 2, + delay, false);
Can you please elaborate what the problem is?
Hitting the following error after waiting 40s for a qemu process to disappear
https://gitlab.com/libvirt/libvirt/-/blob/master/src/util/virprocess.c?ref_t...
This function has additional code after this which waits for the process to terminate on background if killing didn't work so the process should be cleaned up.
We've observed qemu process cleanup/exit times exceeding the 40s total wait time.
Adding extra delay will not cover all cases so that's why I'm not a fan of another tunable which users don't have a reasonable way of setting.
I think that advanced users do have the elements necessary to set this to a reasonable value for their specific hardware, host os settings and based on their resource planning for the host; and even just empirically, this can be then adjusted after monitoring the qemu process termination times for their VMs.
Personally, I felt a qemu.conf option was a reasonable approach. It maintains existing behavior and doesn't affect processes that exit within 40s, yet gives users a mechansim to handle those with long cleanup/exit times.
I think this is reasonable too.
I'm fine making this a per-VM setting if preferred. Also open to other options I may be overlooking, although simply increasing the existing hardcoded value is less appealing :-).
I am skeptical of the per-VM setting, as mentioned above, as QEMU shutdown times can change substantially depending on host hardware and configuration.
Regards, Jim
Thanks! Claudio