11:27 a.m.
On 29/03/2022 20:25, Nir Soffer wrote:
On Wed, Mar 16, 2022 at 1:55 PM lejeczek <peljasz(a)yahoo.co.uk>
wrote:
>
>
> On 15/03/2022 11:21, Daniel P. Berrangé wrote:
>> On Tue, Mar 15, 2022 at 10:39:50AM +0000, lejeczek wrote:
>>> Hi guys.
>>>
>>> Without explicitly, manually using watchdog device for a VM, the VM (centOS
>>> 8 Stream 4.18.0-365.el8.x86_64) shows '/dev/watchdog' exists.
>>> To double check - 'dumpxml' does not show any such device - what kind
of a
>>> 'watchdog' that is?
>> The kernel can always provide a pure software watchdog IIRC. It can be
>> useful if a userspace app wants a watchdog. The limitation is that it
>> relies on the kernel remaining functional, as there's no hardware
>> backing it up.
>>
>> Regards,
>> Daniel
> On a related note - with 'i6300esb' watchdog which I tested
> and I believe is working.
> I get often in my VMs from 'dmesg':
> ...
> watchdog: BUG: soft lockup - CPU#0 stuck for xxxs! [swapper/0:0]
> rcu: INFO: rcu_sched self-detected stall on CPU
> ...
> This above is from Ubuntu and CentOS alike and when this
> happens, console via VNC responds to until first 'enter'
> then is non-resposive.
> This happens after VM(s) was migrated between hosts, but
> anyway..
> I do not see what I expected from 'watchdog' - there is no
> action whatsoever, which should be 'reset'. VM remains in
> such 'frozen' state forever.
>
> any & all shared thoughts much appreciated.
> L.
You need to run some userspace tool that will open the watchdog
device, and pet it periodically, telling the kernel that userspace is alive.
If this tool will stop petting the watchdog, maybe because of a soft lockup
or other trouble, the watchdog device will reset the VM.
watchdog(8) may be the tool you need.
See also
https://www.kernel.org/doc/Documentation/watchdog/watchdog-api.rst
Nir
I do not think that 'i6300esb' watchog works under those
soft-lockups, whether it's qemu or OS end I cannot say.
With:
<watchdog model='i6300esb' action='reset'/>
in dom xml OS sees:
-> $ llr /dev/watchdog*
crw-------. 1 root root 10, 130 Apr 5 16:59 /dev/watchdog
crw-------. 1 root root 248, 0 Apr 5 16:59 /dev/watchdog0
crw-------. 1 root root 248, 1 Apr 5 16:59 /dev/watchdog1
and
-> $ wdctl
Device: /dev/watchdog
Identity: i6300ESB timer [version 0]
Timeout: 30 seconds
Pre-timeout: 0 seconds
FLAG DESCRIPTION STATUS BOOT-STATUS
KEEPALIVEPING Keep alive ping reply 1 0
MAGICCLOSE Supports magic close char 0 0
SETTIMEOUT Set timeout (in seconds) 0 0
If it worked, the HW watchdog, then 'i6300esb' should reset
the VM if nothing is pinging the watchdog - I read that it's
possible to exit 'software' watchdog and not to cause HW
watchdog take action. I do not know it that's happening here
when I just 'systemclt stop watchdog'
In '/etc/watchdog.conf' I do not point to any specific
device, which I believe makes watchdogd do its things.
Simple test:
-> $ cat >> /dev/watchdog
& 'Enter' press twice
does invoke 'reset' action and I was to believe 'wdctl' that
is HW watchdog working. But!...
The main issue I have are those "soft lockups" where VM's OS
becomes frozen, but nothing from the watchdog, no action -
though, as VM is in such frozen state host shows high CPU
for the VM.
I do not anything fancy so I really wonder if what I see is
that rare.
Soft-lockup occur I think usually, cannot say that uniquely
though, during or after VM live-migration.
thanks, L.