11:57 a.m.
On 2/22/21 10:34 AM, Michal Privoznik wrote:
On 2/22/21 6:02 PM, Jim Fehlig wrote:
> On 2/22/21 5:46 AM, Michal Privoznik wrote:
>> On 2/20/21 1:19 AM, Jim Fehlig wrote:
>>> Commit fa30ee04a2 caused a regression in normal domain shutown.
>>> Initiating a shutdown from within the domain or via 'virsh shutdown'
>>> does cause the guest OS running in the domain to shutdown, but libvirt
>>> never reaps the domain so it is always shown in a running state until
>>> calling 'virsh destroy'.
>>>
>>> The shutdown thread is also an internal user of the driver shutdown
>>> machinery and eventually calls libxlDomainDestroyInternal where
>>> the ignoreDeathEvent inhibitor is set, but running in a thread
>>> introduces the possibility of racing with the death event from
>>> libxl. This can be prevented by setting ignoreDeathEvent before
>>> running the shutdown thread.
>>>
>>> An additional improvement is to handle the destroy event synchronously
>>> instead of spawning a thread. The time consuming aspects of destroying
>>> a domain have been completed when the destroy event is delivered.
>>>
>>> Signed-off-by: Jim Fehlig <jfehlig(a)suse.com>
>>> ---
>>> src/libxl/libxl_domain.c | 115 ++++++++++++++++++---------------------
>>> 1 file changed, 54 insertions(+), 61 deletions(-)
>>>
>>> diff --git a/src/libxl/libxl_domain.c b/src/libxl/libxl_domain.c
>>> index d59153fffa..32dc503089 100644
>>> --- a/src/libxl/libxl_domain.c
>>> +++ b/src/libxl/libxl_domain.c
>>
>>> @@ -685,42 +658,62 @@ libxlDomainEventHandler(void *data,
>>> VIR_LIBXL_EVENT_CONST libxl_event *event)
>>> * after calling libxl_domain_suspend() are handled by its callers.
>>> */
>>> if (xl_reason == LIBXL_SHUTDOWN_REASON_SUSPEND)
>>> - goto error;
>>> + goto cleanup;
>>> - /*
>>> - * Start a thread to handle shutdown. We don't want to be tying up
>>> - * libxl's event machinery by doing a potentially lengthy shutdown.
>>> - */
>>> - shutdown_info = g_new0(struct libxlShutdownThreadInfo, 1);
>>> + vm = virDomainObjListFindByID(driver->domains, event->domid);
>>> + if (!vm) {
>>> + /* Nothing to do if we can't find the virDomainObj */
>>> + goto cleanup;
>>> + }
>>> - shutdown_info->driver = driver;
>>> - shutdown_info->event = (libxl_event *)event;
>>> - name = g_strdup_printf("ev-%d", event->domid);
>>> - if (event->type == LIBXL_EVENT_TYPE_DOMAIN_SHUTDOWN)
>>> - ret = virThreadCreateFull(&thread, false,
libxlDomainShutdownThread,
>>> - name, false, shutdown_info);
>>> - else if (event->type == LIBXL_EVENT_TYPE_DOMAIN_DEATH)
>>> - ret = virThreadCreateFull(&thread, false,
libxlDomainDeathThread,
>>> - name, false, shutdown_info);
>>> + if (event->type == LIBXL_EVENT_TYPE_DOMAIN_SHUTDOWN) {
>>> + libxlDomainObjPrivatePtr priv = vm->privateData;
>>> + struct libxlShutdownThreadInfo *shutdown_info = NULL;
>>> + virThread thread;
>>> + g_autofree char *name = NULL;
>>> - if (ret < 0) {
>>> /*
>>> - * Not much we can do on error here except log it.
>>> + * Start a thread to handle shutdown. We don't want to be tying
up
>>> + * libxl's event machinery by doing a potentially lengthy
shutdown.
>>> */
>>> - VIR_ERROR(_("Failed to create thread to handle domain
shutdown"));
>>> - goto error;
>>> - }
>>> + shutdown_info = g_new0(struct libxlShutdownThreadInfo, 1);
>>> - /*
>>> - * libxlShutdownThreadInfo and libxl_event are freed in shutdown thread
>>> - */
>>> - return;
>>> + shutdown_info->driver = driver;
>>> + shutdown_info->vm = vm;
>>
>> Aaah, wanted to suggest that you increment @vm's refcounter here, because ..
>>
>>> + shutdown_info->event = (libxl_event *)event;
>>> + name = g_strdup_printf("ev-%d", event->domid);
>>> + /*
>>> + * Cleanup will be handled by the shutdown thread.
>>> + * Ignore the forthcoming death event from libxl
>>> + */
>>> + priv->ignoreDeathEvent = true;
>>> + if (virThreadCreateFull(&thread, false,
libxlDomainShutdownThread,
>>> + name, false, shutdown_info) < 0) {
>>
>> .. what if this thread is scheduled after [1]? But then noticed this subtle ..
>>
>>> + /*
>>> + * Not much we can do on error here except log it.
>>> + */
>>> + VIR_ERROR(_("Failed to create thread to handle domain
shutdown"));
>>> + VIR_FREE(shutdown_info);
>>> + goto cleanup;
>>> + }
>>> + /*
>>> + * virDomainObjEndAPI is called in the shutdown thread, where
>>> + * libxlShutdownThreadInfo and libxl_event are also freed.
>>> + */
>>> + return;
>>
>> .. return. So the refcount is okay :-)
>
> The subtle-ness definitely crossed my mind :-). Freeing the libxl event and
> shutdown info was already handled in the same way. The comment was a helpful
> reminder about the subtle behavior. I did consider options before settling on
> this approach, e.g. using g_autoptr to avoid the cleanup label altogether. For
> the shutdown info I would only have to create a free function, but the
> libxl_event would require defining a new struct to hold the event and
> libxl_ctx, along with a free func. It seemed a bit overkill, but I can pursue
> that if you'd like.
Not sure if it's worth it. But if you write the patch I'm more than happy to
review it.
I'll look at it in conjunction with other cleanups and improvements I'm working
on. In the meantime I've pushed this fix.
> BTW, while reading your reply I noticed that ignoreDeathEvent
should be set to
> false if the thread creation fails, although it probably doesn't matter much
> at that point. If the approach here is still acceptable, I'll squash in the
> below diff before pushing. Another BTW: is it fine to push this bug/regression
> fix even though we've now hit 7.1.0 freeze?
That's exactly what freeze is for! I mean, if we weren't allowed to push fixes
then that's the point in having freeze?
Yes, thanks for point out the obvious. Another cup of coffee would have also
done that :-).
Regards,
Jim