On 3/12/21 1:44 AM, Peter Krempa wrote:
On Thu, Mar 11, 2021 at 16:47:54 -0700, Jim Fehlig wrote:
On 3/10/21 9:37 AM, Peter Krempa wrote:
Commit 94e45d1042e broke exec-restart of virtlogd and virtlockd as the code waiting for the daemon shutdown closed the daemons before exec-restarting.
This reminds me of an odd issue we encountered three years ago, fixed by Daniel
https://listman.redhat.com/archives/libvir-list/2018-March/msg00298.html
I tested your patches but notice locks are still lost on re-exec.
qemu.conf: lock_manager = "lockd"
qemu-lockd.conf: file_lockspace_dir = "/var/lib/libvirt/lockspace"
/var/lib/libvirt/lockspace is nothing special, xfs on a local disk. After starting a VM
# ls /var/lib/libvirt/lockspace/ a89872e150e6b9e4cbd59ef2bd289bc6cd0a8fa6fbf533c41957f77a90381e9c # lslocks | grep lockd virtlockd 95009 POSIX WRITE 0 0 0 /var/lib/libvirt/lockspace/a89872e150e6b9e4cbd59ef2bd289bc6cd0a8fa6fbf533c41957f77a90381e9c virtlockd 95009 POSIX 5B WRITE 0 0 0 /run/virtlockd.pid # systemctl reload virtlockd
Could you make sure that the virtlockd process before and after has the same pid, so that it wasn't actually restarted by systemct?
I thought I checked it, but apparently not...
I'm asking because in my current test I've encountered another crash when exec-restarting:
2021-03-12 08:41:31.649+0000: 2765718: error : virJSONValueToBuffer:1946 : internal error: failed to convert virJSONValue to yajl data double free or corruption (fasttop)
Program received signal SIGABRT, Aborted. 0x00007ffff77819d5 in raise () from /lib64/libc.so.6 (gdb) bt #0 0x00007ffff77819d5 in raise () at /lib64/libc.so.6 #1 0x00007ffff776a8a4 in abort () at /lib64/libc.so.6 #2 0x00007ffff77c4177 in __libc_message () at /lib64/libc.so.6 #3 0x00007ffff77cbe6c in annobin_top_check.start () at /lib64/libc.so.6 #4 0x00007ffff77cd393 in _int_free () at /lib64/libc.so.6 #5 0x00007ffff7a0b70d in g_free () at /lib64/libglib-2.0.so.0 #6 0x00007ffff7c0977f in virJSONValueFree (value=0x5555555710b0) at ../../../libvirt/src/util/virjson.c:401 #7 0x000055555555c3f2 in glib_autoptr_clear_virJSONValue (_ptr=0x5555555c4250) at ../../../libvirt/src/util/virjson.h:173 #8 glib_autoptr_cleanup_virJSONValue (_ptr=<synthetic pointer>) at ../../../libvirt/src/util/virjson.h:173 #9 virLockDaemonPreExecRestart (argv=0x7fffffffe428, dmn=<optimized out>, state_file=<optimized out>) at ../../../libvirt/src/locking/lock_daemon.c:700 #10 main (argc=<optimized out>, argv=0x7fffffffe428) at ../../../libvirt/src/locking/lock_daemon.c:1148
because looking again I'm seeing the same crash. Facepalm!
Looks like a double free. I'll post patches later for this.
I noticed your patches are pushed. A quick test verified all is working well now. Thanks! Regards, Jim