On Thu, Oct 17, 2019 at 11:34 AM Daniel P. Berrangé <berrange(a)redhat.com>
wrote:
On Thu, Oct 17, 2019 at 11:26:12AM +0200, Roman Mohr wrote:
> Hi,
>
> In kubevirt we are running into a strange permission problem on
> libvirt-5.0. We see transient "Permission Denied" errors when
"virAuditSend"
> wants to send an audit log. [1] shows the logs of one of these
containers.
> Here an example:
>
>
{"component":"virt-launcher","level":"warning","msg":"Failed
to send
audit
> message virt=kvm
>
vm=\"kubevirt-test-default_testvmit2pqrkrlrwbhptcjcs4n67jn6pjqvmtd7pkrpdmkrl5sldzs4rxr9zdg8m45jxz\"
> uuid=56a33283-f6d7-4002-b188-1fed83186545 vm-ctx=+107:+107
> img-ctx=+107:+107 model=dac: Permission
>
denied","pos":"virAuditSend:141","subcomponent":"libvirt","thread":"30","timestamp":"2019-10-08T23:58:40.651000Z"}
>
> We recently switched in kubevirt to a dedicated selinux policy and remove
> the general "privileged" flag from the containers where we run libvirt
in.
> This is very likely related to it, but we can't make sense out of it,
> because:
>
> * It randomly affects one out of a few hundred containers which we start
> * It is not bound to a specific node
> * It is only transient on that container. After a few denials libvirt
can
> just continue.
The transient nature makes this really bizarre.
The error message above comes from the audit_log_user_message() call
that libvirt makes. This is implemented in libaudit.so using the
sendto() syscall on a netlink socket opened with
socket(PF_NETLINK, SOCK_RAW, NETLINK_AUDIT);
So in terms of code being run there, there's very little - the EPERM
is coming back from the kernel when sending the message.
I'm not sure what scenario could cause this - perhaps the audit log
buffer in the kernel is full or something like that ?
I will adjust our collectors to collect the selinux auditlog and dmesg.
Maybe I can see more there. Will update the thread when I have more
information.
> * Sometimes it is accompanied with a transient "Permission denied" on
> /dev/null from our code in that container (so not from something which
> libvirt tries to do).
>
> Has someone seen something like this before in different environments?
Never seen anything like this reported before.
Thanks Daniel.
Roman