Re: [libvirt PATCH] apparmor: Allow running /usr/libexec/qemu-kvm
On Thu, Nov 03, 2022 at 05:23:27PM +0000, Daniel P. Berrangé wrote:
On Thu, Nov 03, 2022 at 12:35:15PM -0400, Andrea Bolognani wrote:
> On Thu, Nov 03, 2022 at 03:39:44PM +0100, Peter Krempa wrote:
> > On Thu, Nov 03, 2022 at 12:13:53 +0100, Andrea Bolognani wrote:
> > > Distros that use AppArmor, such as Debian and Ubuntu, install
> > > QEMU under /usr/bin/qemu-system-*, and our AppArmor profile is
> > > written with that assumption in mind.
> > >
> > > If you try to run the RHEL or CentOS version of libvirt and
> > > QEMU inside a privileged container on such distros, however,
> > > that will result in an error, because the path
> > > /usr/libexec/qemu-kvm is used instead.
> >
> > So IIUC by this patch you modify the profile which gets installed into
> > the Debian/Ubuntu host system by the Debian/Ubuntu package which then in
> > turn allows the non-Debian/Ubuntu libvirt in the container to do it's
> > job?
>
> Pretty much.
>
> > I'm basing the above on the fact that the RHEL/Centos package is
> > compiled with:
> >
> > -Dapparmor=disabled \
> > -Dapparmor_profiles=disabled \
> > -Dsecdriver_apparmor=disabled \
> >
> > By extension, does that mean that you have to install libvirt on your
> > host so that you can in turn run a container (which I'd presume is
> > opaque) with libvirt bundled inside?
>
> It's actually the other way around :)
>
> If you don't have libvirt installed on the Debian/Ubuntu host, then
> the AppArmor profile won't be present and the containerized CentOS
> libvirt will be allowed to start the containerized CentOS QEMU.
>
> If you *do* have libvirt installed on the Debian/Ubuntu host, then
> the AppArmor profile will also be applied to the containerized CentOS
> libvirt and running the containerized CentOS QEMU will be forbidden.
>
> Patching the AppArmor policy is supposed to help with the second
> scenario.
I don't see how this can work properly.
If running with AppArmor, I would expect libvirtd itself needs to be
built with AppArmor, so that when launching a VM it can spawn
virt-aa-helper to create the per-VM customized profile. The CentOS
based libvirt running inside the container will be built without
virt-aa-helper, so won't load this.
I would rather expect that AppArmor does not attempt to control
any processes inside the containers, other than with a generic
'docker' AppArmor profile. It makes no sense for profiles from
the host OS install to apply to stuff in containers, as we can't
assume the host + container installs are the same versions. Are
you sure the KubeVirt problem isn't simply a mis-configuration
of the host environment allowing AppArmor to leak inside the
container.
IIUC a specific profile (cri-containerd.apparmor.d) is used for
unprivileged containers such as virt-launcher, but a privileged one
such as virt-handler falls under the same profile as the host.
This makes some amount of sense to me: unprivileged containers are
already limited in what they can do by the usual restrictions on user
processes. Privileged containers, on the other hand, are effectively
root processes, so it's advisable to be significantly more cautious
with them.
Note that this is just my current understanding of the situation, and
I'm far from an expert when it comes to containers in general and
their interactions with AppArmor in particular. I recommend taking a
look at
https://github.com/kubevirt/kubevirt/pull/8692
and the issues linked therein, which will provide more context coming
from people who actually know what they're talking about :)
Now that I've typed all of the above, I wonder if the problem
wouldn't be better solved by making sure that KubeVirt runs the
libvirtd instance used to figure out node capabilities in an
unprivileged container? Maybe there's something that prevents them
from doing so.
I'll bring up the idea and see what they think of it.
--
Andrea Bolognani / Red Hat / Virtualization