6:17 a.m.
Michal Privoznik <mprivozn(a)redhat.com> [2018-09-12, 11:32AM +0200]:
On 09/12/2018 07:19 AM, Bjoern Walk wrote:
> Michal Privoznik <mprivozn(a)redhat.com> [2018-09-10, 11:36AM +0200]:
>> Technically, this is v4 of:
>>
>> https://www.redhat.com/archives/libvir-list/2018-August/msg01627.html
>>
>> However, this is implementing different approach than any of the
>> previous versions.
>>
>> One of the problems with previous version was that it was too
>> complicated. The main reason for that was that we could not close the
>> connection whilst there was a file locked. So we had to invent a
>> mechanism that would prevent that (on the client side).
>>
>> These patches implement different approach. They rely on secdriver's
>> transactions which bring all the paths we want to label into one place
>> so that they can be relabelled within different namespace.
>> I'm extending this idea so that transactions run all the time
>> (regardless of domain namespacing) and only at the very last moment is
>> decided which namespace would the relabeling run in.
>>
>> Metadata locking is then as easy as putting lock/unlock calls around one
>> function.
>>
>> You can find the patches at my github too:
>>
>> https://github.com/zippy2/libvirt/tree/disk_metadata_lock_v4_alt
>
> Hey Michal,
>
> is was running a quick test with this patch series with two domains
> sharing a disk image without <shareable/> and SELinux enabled. When
> starting the second domain, the whole libvirtd daemon hangs for almost a
> minute until giving the error that the image is locked. I haven't
> debugged it yet to figure out what happens.
Is this on two different hosts or one?
On the same host.
I'm unable to reproduce, so can you please attach debugger and
share 't
a a bt' output with me?
(gdb) thread apply all bt
Thread 17 (Thread 0x3ff48dff910 (LWP 193353)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff6678c5a8 in udevEventHandleThread (opaque=<optimized out>) at
node_device/node_device_udev.c:1603
#3 0x000003ff8c6bad16 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 16 (Thread 0x3ff4acfe910 (LWP 193312)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 15 (Thread 0x3ff4b4ff910 (LWP 193311)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 14 (Thread 0x3ff64efd910 (LWP 193310)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 13 (Thread 0x3ff656fe910 (LWP 193309)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 12 (Thread 0x3ff65eff910 (LWP 193308)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 11 (Thread 0x3ff67fff910 (LWP 193307)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 10 (Thread 0x3ff84bf7910 (LWP 193306)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 9 (Thread 0x3ff853f8910 (LWP 193305)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 8 (Thread 0x3ff85bf9910 (LWP 193304)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 7 (Thread 0x3ff863fa910 (LWP 193303)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbba0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 6 (Thread 0x3ff86bfb910 (LWP 193302)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 5 (Thread 0x3ff873fc910 (LWP 193301)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 4 (Thread 0x3ff87bfd910 (LWP 193300)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 3 (Thread 0x3ff883fe910 (LWP 193299)):
#0 0x000003ff8b7113d4 in read () from /lib64/libpthread.so.0
#1 0x000003ff8c65a648 in saferead () from /lib64/libvirt.so.0
#2 0x000003ff8c65a718 in saferead_lim () from /lib64/libvirt.so.0
#3 0x000003ff8c65abe4 in virFileReadHeaderFD () from /lib64/libvirt.so.0
#4 0x000003ff8c69cb5a in virProcessRunInMountNamespace () from /lib64/libvirt.so.0
#5 0x000003ff8c767bd6 in virSecuritySELinuxTransactionCommit () from /lib64/libvirt.so.0
#6 0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0
#7 0x000003ff8c75bb70 in virSecurityStackTransactionCommit () from /lib64/libvirt.so.0
#8 0x000003ff8c75fb1c in virSecurityManagerTransactionCommit () from /lib64/libvirt.so.0
#9 0x000003ff6612c492 in qemuSecuritySetAllLabel () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#10 0x000003ff660bcfd6 in qemuProcessLaunch () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#11 0x000003ff660c0916 in qemuProcessStart () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#12 0x000003ff66124a00 in qemuDomainObjStart.constprop.52 () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#13 0x000003ff66125030 in qemuDomainCreateWithFlags () from
/usr/lib64/libvirt/connection-driver/libvirt_driver_qemu.so
#14 0x000003ff8c87bfca in virDomainCreate () from /lib64/libvirt.so.0
#15 0x000002aa1f2d516c in remoteDispatchDomainCreate (server=<optimized out>,
msg=0x2aa4c488b00, args=<optimized out>, rerr=0x3ff883fdae8, client=<optimized
out>) at remote/remote_daemon_dispatch_stubs.h:4434
#16 remoteDispatchDomainCreateHelper (server=<optimized out>, client=<optimized
out>, msg=0x2aa4c488b00, rerr=0x3ff883fdae8, args=<optimized out>,
ret=0x3ff78004c80) at remote/remote_daemon_dispatch_stubs.h:4410
#17 0x000003ff8c78bc80 in virNetServerProgramDispatch () from /lib64/libvirt.so.0
#18 0x000003ff8c792aba in virNetServerHandleJob () from /lib64/libvirt.so.0
#19 0x000003ff8c6bbb16 in virThreadPoolWorker () from /lib64/libvirt.so.0
#20 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#21 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#22 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 2 (Thread 0x3ff88bff910 (LWP 193298)):
#0 0x000003ff8b70d7b8 in pthread_cond_wait@(a)GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x000003ff8c6baf92 in virCondWait () from /lib64/libvirt.so.0
#2 0x000003ff8c6bbbe0 in virThreadPoolWorker () from /lib64/libvirt.so.0
#3 0x000003ff8c6bace6 in virThreadHelper () from /lib64/libvirt.so.0
#4 0x000003ff8b7079a8 in start_thread () from /lib64/libpthread.so.0
#5 0x000003ff8b5f9706 in thread_start () from /lib64/libc.so.6
Thread 1 (Thread 0x3ff8cbd7970 (LWP 193297)):
#0 0x000003ff8b5ee502 in poll () from /lib64/libc.so.6
#1 0x000003ff8c65629c in virEventPollRunOnce () from /lib64/libvirt.so.0
#2 0x000003ff8c654caa in virEventRunDefaultImpl () from /lib64/libvirt.so.0
#3 0x000003ff8c7921de in virNetDaemonRun () from /lib64/libvirt.so.0
#4 0x000002aa1f2a4c12 in main (argc=<optimized out>, argv=<optimized out>) at
remote/remote_daemon.c:1461
When I try to reproduce I always get one domain running and the
other
failing to start because qemu is unable to do its locking. When I put
<shareable/> in both, they start successfully.
Yes, I get the same (expected) behaviour, just with the timeout.
TBH, I don't run SELinux enabled host so I'm testing DAC
only, but the
changes to the code are merely the same. So I'm wondering if this is
really an issue in my SELinux impl or somewhere else.
BTW: The one minute timeout comes from patch 16/23:
#define LOCK_ACQUIRE_TIMEOUT 60
Michal
It's entirely possible that my setup is broken, this was just a quick
test. If you can't reproduce it or make sense out of the backtrace then
it's fine, I will try to put some time to debug it next week. I was just
interested if this would fix a long-standing bug where this scenario
would crash the first domain because the label changed.
Bjoern
--
IBM Systems
Linux on Z & Virtualization Development
------------------------------------------------------------------------
IBM Deutschland Research & Development GmbH
Schönaicher Str. 220, 71032 Böblingen
Phone: +49 7031 16 1819
------------------------------------------------------------------------
Vorsitzende des Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen
Registergericht: Amtsgericht Stuttgart, HRB 243294