Re: [libvirt] [PATCH v4 00/23] Introduce metadata locking

Michal Privoznik <mprivozn(a)redhat.com&gt; [2018-09-12, 11:32AM +0200]: > On 09/12/2018 07:19 AM, Bjoern Walk wrote: > > Michal Privoznik <mprivozn(a)redhat.com&gt; [2018-09-10, 11:36AM +0200]: > >> Technically, this is v4 of: > >> > >> https://www.redhat.com/archives/libvir-list/2018-August/msg01627.html > >> > >> However, this is implementing different approach than any of the > >> previous versions. > >> > >> One of the problems with previous version was that it was too > >> complicated. The main reason for that was that we could not close the > >> connection whilst there was a file locked. So we had to invent a > >> mechanism that would prevent that (on the client side). > >> > >> These patches implement different approach. They rely on secdriver's > >> transactions which bring all the paths we want to label into one place > >> so that they can be relabelled within different namespace. > >> I'm extending this idea so that transactions run all the time > >> (regardless of domain namespacing) and only at the very last moment is > >> decided which namespace would the relabeling run in. > >> > >> Metadata locking is then as easy as putting lock/unlock calls around one > >> function. > >> > >> You can find the patches at my github too: > >> > >> https://github.com/zippy2/libvirt/tree/disk_metadata_lock_v4_alt > > > > Hey Michal, > > > > is was running a quick test with this patch series with two domains > > sharing a disk image without <shareable/> and SELinux enabled. When > > starting the second domain, the whole libvirtd daemon hangs for almost a > > minute until giving the error that the image is locked. I haven't > > debugged it yet to figure out what happens. > > Is this on two different hosts or one? On the same host. > I'm unable to reproduce, so can you please attach debugger and share 't > a a bt' output with me? (gdb) thread apply all bt Thread 17 (Thread 0x3ff48dff910 (LWP 193353)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff6678c5a8 in udevEventHandleThread (opaque=<optimized out>) at node_device/node_device_udev.c:1603 #3 0x000003ff8c6bad16 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 16 (Thread 0x3ff4acfe910 (LWP 193312)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 15 (Thread 0x3ff4b4ff910 (LWP 193311)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 14 (Thread 0x3ff64efd910 (LWP 193310)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 13 (Thread 0x3ff656fe910 (LWP 193309)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 12 (Thread 0x3ff65eff910 (LWP 193308)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 11 (Thread 0x3ff67fff910 (LWP 193307)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 10 (Thread 0x3ff84bf7910 (LWP 193306)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 9 (Thread 0x3ff853f8910 (LWP 193305)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 8 (Thread 0x3ff85bf9910 (LWP 193304)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 7 (Thread 0x3ff863fa910 (LWP 193303)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 6 (Thread 0x3ff86bfb910 (LWP 193302)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 5 (Thread 0x3ff873fc910 (LWP 193301)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 4 (Thread 0x3ff87bfd910 (LWP 193300)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 3 (Thread 0x3ff883fe910 (LWP 193299)): #0 0x000003ff8b7113d4 in read () from /lib64/libpthread.so.0 #1 0x000003ff8c65a648 in saferead () from /lib64/libvirt.so.0 #2 0x000003ff8c65a718 in saferead_lim () from /lib64/libvirt.so.0 #3 0x000003ff8c65abe4 in virFileReadHeaderFD () from /lib64/libvirt.so.0 #4 0x000003ff8c69cb5a in virProcessRunInMountNamespace () from /lib64/libvirt.so.0 #5 0x000003ff8c767bd6 in virSecuritySELinuxTransactionCommit () from /lib64/libvirt.so.0 #6 0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0 #7 0x000003ff8c75bb70 in virSecurityStackTransactionCommit () from /lib64/libvirt.so.0 #8 0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0 #9 0x000003ff6612c492 in qemuSecuritySetAllLabel () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #10 0x000003ff660bcfd6 in qemuProcessLaunch () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #11 0x000003ff660c0916 in qemuProcessStart () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #12 0x000003ff66124a00 in qemuDomainObjStart.constprop.52 () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #13 0x000003ff66125030 in qemuDomainCreateWithFlags () from /usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so #14 0x000003ff8c87bfca in virDomainCreate () from /lib64/libvirt.so.0 #15 0x000002aa1f2d516c in remoteDispatchDomainCreate (server=<optimized out>, msg=0x2aa4c488b00, args=<optimized out>, rerr=0x3ff883fdae8, client=<optimized out>) at remote/remote_daemon_dispatch_stubs.h:4434 #16 remoteDispatchDomainCreateHelper (server=<optimized out>, client=<optimized out>, msg=0x2aa4c488b00, rerr=0x3ff883fdae8, args=<optimized out>, ret=0x3ff78004c80) at remote/remote_daemon_dispatch_stubs.h:4410 #17 0x000003ff8c78bc80 in virNetServerProgramDispatch () from /lib64/libvirt.so.0 #18 0x000003ff8c792aba in virNetServerHandleJob () from /lib64/libvirt.so.0 #19 0x000003ff8c6bbb16 in virThreadPoolWorker () from /lib64/libvirt.so.0 #20 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #21 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #22 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 2 (Thread 0x3ff88bff910 (LWP 193298)): #0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0 #1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0 #2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0 #3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0 #4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0 #5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6 Thread 1 (Thread 0x3ff8cbd7970 (LWP 193297)): #0 0x000003ff8b5ee502 in poll () from /lib64/libc.so.6 #1 0x000003ff8c65629c in virEventPollRunOnce () from /lib64/libvirt.so.0 #2 0x000003ff8c654caa in virEventRunDefaultImpl () from /lib64/libvirt.so.0 #3 0x000003ff8c7921de in virNetDaemonRun () from /lib64/libvirt.so.0 #4 0x000002aa1f2a4c12 in main (argc=<optimized out>, argv=<optimized out>) at remote/remote_daemon.c:1461 > When I try to reproduce I always get one domain running and the other > failing to start because qemu is unable to do its locking. When I put > <shareable/> in both, they start successfully. Yes, I get the same (expected) behaviour, just with the timeout. > TBH, I don't run SELinux enabled host so I'm testing DAC only, but the > changes to the code are merely the same. So I'm wondering if this is > really an issue in my SELinux impl or somewhere else. > > BTW: The one minute timeout comes from patch 16/23: > > #define LOCK_ACQUIRE_TIMEOUT 60 > > Michal It's entirely possible that my setup is broken, this was just a quick test. If you can't reproduce it or make sense out of the backtrace then it's fine, I will try to put some time to debug it next week. I was just interested if this would fix a long-standing bug where this scenario would crash the first domain because the label changed. Bjoern