On 9/1/26 5:35 AM, Daniel P. Berrangé wrote:
On Tue, Sep 01, 2026 at 12:48:32PM +0200, Claudio Fontana wrote: [...]
In one case I have seen two CPUs of the same generation, ~ same # of sockets, same base clock, (but slightly different CPU model), same hugepages settings, comparable NUMA topology ... differ in terms of QEMU process termination times by a factor of at least 4, _presumeably_ due to a combination of slower memory speeds coupled with heavy memory-intensive workloads on the same socket.
Effectively there is no "right" number at all, because "right" means wait long enough that a correctly operating QEMU exits, without waiting so long that a QEMU stuck in uninterruptible sleep will hang the Destroy API too long.
We had gone for a relatively short timeout to avoid hung QEMU's delaying libvirt.
I think this is a worthy goal and we should avoid extending it if possible.
We also have VIR_DOMAIN_DESTROY_GRACEFUL flag which lets us send SIGTERM but skip the SIGKILL part. IIUC the logic though, we still wait exactly the same amount of time in total, just don't send KILL.
I am concerned that getting to the "right" number might prove impractical?
Effectively we'll be forced to over-estimate the worst case timeout by a very significant factor, to minimize chance of false errors.
I wonder what behaviour suits mgmt apps best ?
If virDomainDestroy reaches the timeout and returns an error, it doesn't mean cleanup has failed.
If a mgmt app is monitoring the running VM list, it will eventually see the VM go away.
This is effectively the current behavior, which I think is fine.
IOW, an error from virDomainDestroy isn't really a serious error in many cases - it is effectively more of a warning.
I suspect others, if encountering the EBUSY error, have treated it as a warning.
The biggest challenge I see is there there is no way to understand if QEMU is "making progress" in shutting down, as opposed to become stuck. If you look at the underlying process, if QEMU were stuck in the "D" state that's bad, but if we've sent SIGKILL then it should be reaped by the kernel in all other states, so it is just a matter of waiting.
Right. And we actually already do that in the shutdown case, which btw was the case for the reporter of the issue influencing this patch. Due to some miscommunication, we mistakenly started chasing timeouts in the destroy operation, when in fact the reporter (using libvirt 11.0.0) hit https://gitlab.com/libvirt/libvirt/-/work_items/853 by issuing 'systemctl poweroff' within the guest. In the end, huge facepalm on my part. Sorry for the disruption, but thanks for being vigilant against unneeded hacks :-). Self NACK for this patch. But perhaps not all time is wasted. During shutdown, a long-exiting QEMU could exceed the timeout, causing qemuProcessBeginStopJob() to fail and return early in processMonitorEOFEvent() https://gitlab.com/libvirt/libvirt/-/blob/master/src/qemu/qemu_driver.c?ref_... None of the code from line 4190 through the rest of the function is executed. When the QEMU process finally disappears, processShutdownCompletedEvent() is eventually executed, where we only call qemuDomainRemoveInactive(). Do we miss sending a lifecycle event in this case? Regards, Jim