7:21 a.m.
Hello all,
tl;dr, can you point me to the point in the libvirt repo where it's trying
to change a tap-device's SELinux label?
I am trying to create a tap device with libvirt on a
super-privileged container, and then use it on another,
unprivileged container with libvirt.
User wise, I know I need the super-privileged container to open the tap
device with the user of the unprivileged one - that I already did and it's
not the issue.
But I have a problem when I open the tap device in the
non-privileged container: the tap device currently has the spc_t label
since the tun_socket inherited the selinux context from the
super-privileged container who creates it. then libvirt is trying to change
the SELinux labels, and since it's not privileged then it fails.
But I didn't find where and how libvirt is trying to change the tap
device's label.
Can you point me to that specific code on libvirt?
Ram Lavi
Senior Software Engineer
Red Hat Israel <https://www.redhat.com/>
Yerushalaim Road 34, Ra'anana
ralavi(a)redhat.com IM: ralavi
@RedHat <https://twitter.com/redhat> Red Hat
<https://www.linkedin.com/company/red-hat> Red Hat
<https://www.facebook.com/RedHatInc>
<https://www.redhat.com/>
7:33 a.m.
On Tue, Jul 14, 2020 at 03:21:17PM +0300, Ram Lavi wrote:
Hello all,
tl;dr, can you point me to the point in the libvirt repo where it's trying
to change a tap-device's SELinux label?
I am trying to create a tap device with libvirt on a
super-privileged container, and then use it on another,
unprivileged container with libvirt.
User wise, I know I need the super-privileged container to open the tap
device with the user of the unprivileged one - that I already did and it's
not the issue.
But I have a problem when I open the tap device in the
non-privileged container: the tap device currently has the spc_t label
since the tun_socket inherited the selinux context from the
super-privileged container who creates it. then libvirt is trying to change
the SELinux labels, and since it's not privileged then it fails.
But I didn't find where and how libvirt is trying to change the tap
device's label.
Can you point me to that specific code on libvirt?
If the SELinux policy that libvirtd is running under prevents it from
re-labelling, then TAP devices label failure is just going to be one
out of 100's of labelling failures.
Either the SELinux policy needs to be changed to allow libvirtd to
relabel stuff in the normal manner, or you will have to turn off
SELinux support in libvirtd. in /etc/libvirt/qemu.conf via the
param security_driver = "none". If you turn off SELinux in
libvirt, then you no longer have separation of QEMU processes
which may be a security flaw depending on your deplyoment
scenario.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
8:02 a.m.
On Tue, Jul 14, 2020 at 3:33 PM Daniel P. Berrangé <berrange(a)redhat.com>
wrote:
On Tue, Jul 14, 2020 at 03:21:17PM +0300, Ram Lavi wrote:
> Hello all,
>
> tl;dr, can you point me to the point in the libvirt repo where it's
trying
> to change a tap-device's SELinux label?
>
> I am trying to create a tap device with libvirt on a
> super-privileged container, and then use it on another,
> unprivileged container with libvirt.
> User wise, I know I need the super-privileged container to open the tap
> device with the user of the unprivileged one - that I already did and
it's
> not the issue.
> But I have a problem when I open the tap device in the
> non-privileged container: the tap device currently has the spc_t label
> since the tun_socket inherited the selinux context from the
> super-privileged container who creates it. then libvirt is trying to
change
> the SELinux labels, and since it's not privileged then it fails.
> But I didn't find where and how libvirt is trying to change the tap
> device's label.
>
> Can you point me to that specific code on libvirt?
If the SELinux policy that libvirtd is running under prevents it from
re-labelling, then TAP devices label failure is just going to be one
out of 100's of labelling failures.
IIUC normally libvirtd would use devices created by itself, so there
shouldn't be relabeling failures, right?
Either the SELinux policy needs to be changed to allow libvirtd to
relabel stuff in the normal manner, or you will have to turn off
SELinux support in libvirtd. in /etc/libvirt/qemu.conf via the
param security_driver = "none". If you turn off SELinux in
libvirt, then you no longer have separation of QEMU processes
which may be a security flaw depending on your deplyoment
scenario.
turning SELinux in libvirtd off or allowing libvirt to relabel are tempting
options but it is not an option I'm afraid due to security concerns.
Our plan (or at least this particular effort) is to try to relabel the
tun-socket after created in the super-privileged container to be the same
one as the one used in the unprivileged one, then it won't have an issue
consuming it.
btw the change I (or rather Migule's my teammate) made is in this PR, where
I want to add a tap device in virt-handler (i.e. the super privileged
container) to be further uses in virt-launcher (i.e. the non-privileged
container): https://github.com/kubevirt/kubevirt/pull/3290
Regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|
10:02 a.m.
On Tue, Jul 14, 2020 at 04:02:17PM +0300, Ram Lavi wrote:
On Tue, Jul 14, 2020 at 3:33 PM Daniel P. Berrangé
<berrange(a)redhat.com>
wrote:
> On Tue, Jul 14, 2020 at 03:21:17PM +0300, Ram Lavi wrote:
> > Hello all,
> >
> > tl;dr, can you point me to the point in the libvirt repo where it's
> trying
> > to change a tap-device's SELinux label?
> >
> > I am trying to create a tap device with libvirt on a
> > super-privileged container, and then use it on another,
> > unprivileged container with libvirt.
> > User wise, I know I need the super-privileged container to open the tap
> > device with the user of the unprivileged one - that I already did and
> it's
> > not the issue.
> > But I have a problem when I open the tap device in the
> > non-privileged container: the tap device currently has the spc_t label
> > since the tun_socket inherited the selinux context from the
> > super-privileged container who creates it. then libvirt is trying to
> change
> > the SELinux labels, and since it's not privileged then it fails.
> > But I didn't find where and how libvirt is trying to change the tap
> > device's label.
> >
> > Can you point me to that specific code on libvirt?
>
> If the SELinux policy that libvirtd is running under prevents it from
> re-labelling, then TAP devices label failure is just going to be one
> out of 100's of labelling failures.
>
IIUC normally libvirtd would use devices created by itself, so there
shouldn't be relabeling failures, right?
Whether or not there are relabeling failures is determined by what
SELinux policy libvirtd is running under.
>
> Either the SELinux policy needs to be changed to allow libvirtd to
> relabel stuff in the normal manner, or you will have to turn off
> SELinux support in libvirtd. in /etc/libvirt/qemu.conf via the
> param security_driver = "none". If you turn off SELinux in
> libvirt, then you no longer have separation of QEMU processes
> which may be a security flaw depending on your deplyoment
> scenario.
>
turning SELinux in libvirtd off or allowing libvirt to relabel are tempting
options but it is not an option I'm afraid due to security concerns.
Our plan (or at least this particular effort) is to try to relabel the
tun-socket after created in the super-privileged container to be the same
one as the one used in the unprivileged one, then it won't have an issue
consuming it.
btw the change I (or rather Migule's my teammate) made is in this PR, where
I want to add a tap device in virt-handler (i.e. the super privileged
container) to be further uses in virt-launcher (i.e. the non-privileged
container): https://github.com/kubevirt/kubevirt/pull/3290
In normal host OS deployment, libvirtd runs under virtd_t, and when
it spawns QEMU, it will relabel files to svirt_image_t:s0:$MCS, and
spawn QEMU as svirt_t:s0:$MCS.
My understanding is what in kubevirt, things work differently. Docker
(or podman), launch the container as container_t:s0:$MCS. libvirtd
*and* QEMU thus both run as container_t:s0:$MCS. ie All the labelling
is setup when the container is launched and libvirtd should not do
anything.
So I'm really not sure why you have libvirtd configured to do relabelling
at all ? I'd be expecting it to have security_driver=none in the qemu.conf
file so that libvirtd doesn't do anything.
If libvirtd is doing relabelling, I'm not sure how it works for anything,
let alone tap devices.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
3:22 a.m.
On Tue, Jul 14, 2020 at 6:03 PM Daniel P. Berrangé <berrange(a)redhat.com>
wrote:
On Tue, Jul 14, 2020 at 04:02:17PM +0300, Ram Lavi wrote:
> On Tue, Jul 14, 2020 at 3:33 PM Daniel P. Berrangé <berrange(a)redhat.com>
> wrote:
>
> > On Tue, Jul 14, 2020 at 03:21:17PM +0300, Ram Lavi wrote:
> > > Hello all,
> > >
> > > tl;dr, can you point me to the point in the libvirt repo where it's
> > trying
> > > to change a tap-device's SELinux label?
> > >
> > > I am trying to create a tap device with libvirt on a
> > > super-privileged container, and then use it on another,
> > > unprivileged container with libvirt.
> > > User wise, I know I need the super-privileged container to open the
tap
> > > device with the user of the unprivileged one - that I already did and
> > it's
> > > not the issue.
> > > But I have a problem when I open the tap device in the
> > > non-privileged container: the tap device currently has the spc_t
label
> > > since the tun_socket inherited the selinux context from the
> > > super-privileged container who creates it. then libvirt is trying to
> > change
> > > the SELinux labels, and since it's not privileged then it fails.
> > > But I didn't find where and how libvirt is trying to change the tap
> > > device's label.
> > >
> > > Can you point me to that specific code on libvirt?
> >
> > If the SELinux policy that libvirtd is running under prevents it from
> > re-labelling, then TAP devices label failure is just going to be one
> > out of 100's of labelling failures.
> >
> IIUC normally libvirtd would use devices created by itself, so there
> shouldn't be relabeling failures, right?
Whether or not there are relabeling failures is determined by what
SELinux policy libvirtd is running under.
> >
> > Either the SELinux policy needs to be changed to allow libvirtd to
> > relabel stuff in the normal manner, or you will have to turn off
> > SELinux support in libvirtd. in /etc/libvirt/qemu.conf via the
> > param security_driver = "none". If you turn off SELinux in
> > libvirt, then you no longer have separation of QEMU processes
> > which may be a security flaw depending on your deplyoment
> > scenario.
> >
> turning SELinux in libvirtd off or allowing libvirt to relabel are
tempting
> options but it is not an option I'm afraid due to security concerns.
> Our plan (or at least this particular effort) is to try to relabel the
> tun-socket after created in the super-privileged container to be the same
> one as the one used in the unprivileged one, then it won't have an issue
> consuming it.
>
> btw the change I (or rather Migule's my teammate) made is in this PR,
where
> I want to add a tap device in virt-handler (i.e. the super privileged
> container) to be further uses in virt-launcher (i.e. the non-privileged
> container): https://github.com/kubevirt/kubevirt/pull/3290
In normal host OS deployment, libvirtd runs under virtd_t, and when
it spawns QEMU, it will relabel files to svirt_image_t:s0:$MCS, and
spawn QEMU as svirt_t:s0:$MCS.
My understanding is what in kubevirt, things work differently. Docker
(or podman), launch the container as container_t:s0:$MCS. libvirtd
*and* QEMU thus both run as container_t:s0:$MCS. ie All the labelling
is setup when the container is launched and libvirtd should not do
anything.
So I'm really not sure why you have libvirtd configured to do relabelling
at all ? I'd be expecting it to have security_driver=none in the qemu.conf
file so that libvirtd doesn't do anything.
I checked the dumpxml of the virt-launcher pod (that runs the qemu in
kubevirt) - it has dynamic policy.
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+107:+107</label>
<imagelabel>+107:+107</imagelabel>
</seclabel>
Are you saying this a wrong configuration for a kubevirt vmi?
If libvirtd is doing relabelling, I'm not sure how it works for anything,
let alone tap devices.
I want to emphasize that in this current situation the reason we fail is
that we are trying to give libvirt in a non-privileged pod (i.e.
virt-launcher) a tap device that was created externally (by a privileged
pod - virt-handler).
I think the reason it works so far is that libvirt was using only devices
which were self created so there was no issue to relabel.
Our thought was that if we know how libvirt is relabeling then we could
also do it so that the externally created tap's label will match the
virt-launcher's.
Is this were libvirt does the relabeling
https://github.com/libvirt/libvirt/blob/e71e13488dc1aa65456e54a4b41bc9258...
?
btw the error we get is (from audit)
type=AVC msg=audit(1586956552.265:513): avc: denied { relabelfrom }
for pid=27423 comm="libvirtd"
scontext=system_u:system_r:container_t:s0:c143,c582
tcontext=system_u:system_r:spc_t:s0 tclass=tun_socket permissive=0
Regards,
Daniel
--
|: https://berrange.com -o-
https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-
https://fstop138.berrange.com :|
|: https://entangle-photo.org -o-
https://www.instagram.com/dberrange :|
4:12 a.m.
On Thu, Jul 16, 2020 at 11:22:26AM +0300, Ram Lavi wrote:
On Tue, Jul 14, 2020 at 6:03 PM Daniel P. Berrangé
<berrange(a)redhat.com>
wrote:
I checked the dumpxml of the virt-launcher pod (that runs the qemu in
kubevirt) - it has dynamic policy.
<seclabel type='dynamic' model='dac' relabel='yes'>
<label>+107:+107</label>
<imagelabel>+107:+107</imagelabel>
</seclabel>
Are you saying this a wrong configuration for a kubevirt vmi?
No, this is exactly what I'm expecting and matches the behaviour I
mentioned in my previous mail, where SELinux is not enabled in libvirtd.
This setting is completely unrelated to SELinux. This seclabel is for the
"dac" model, aka Discretionary Access Control, aka plain old POSIX users
and groups. It is used when libvirtd runs as root, and then launches QEMU
as a separate "qemu:qemu" user:group pair.
If selinux / sVirt was in use you would see a separate <seclabel> with
model="selinux".
If you don't see that, then libvirtd is *not* doing any SELinux labelling
at all.
This makes me wonder how you hit the relabelling AVC ? It feels very much
like your custom work to deal with passing in TAP devices has mistakenly
changed something else and thus incorrectly turned on SELinux/sVirt in
libvirtd.
> If libvirtd is doing relabelling, I'm not sure how it works
for anything,
> let alone tap devices.
>
I want to emphasize that in this current situation the reason we fail is
that we are trying to give libvirt in a non-privileged pod (i.e.
virt-launcher) a tap device that was created externally (by a privileged
pod - virt-handler).
I think the reason it works so far is that libvirt was using only devices
which were self created so there was no issue to relabel.
I believe the reason it works historically is that libvirtd is not using
SELinux, per your <seclabel> config above, and so was not attempting
todo any relabelling in the first place.
Our thought was that if we know how libvirt is relabeling then we
could
also do it so that the externally created tap's label will match the
virt-launcher's.
You almost certainly *will* need to relabel the tap device when passing
it from your privileged container into the unprivileged, but this is not
due to libvirtd. This is because podman is likely using a different
SELinux label for the two containers as they have different privilege
levels. The tap device needs to be given the SELinux label matching the
unprivileged container.
Is this were libvirt does the relabeling
https://github.com/libvirt/libvirt/blob/e71e13488dc1aa65456e54a4b41bc9258...
?
That's for setting contexts on files, for FDs you need to call
fsetfilecon_raw
btw the error we get is (from audit)
type=AVC msg=audit(1586956552.265:513): avc: denied { relabelfrom }
for pid=27423 comm="libvirtd"
scontext=system_u:system_r:container_t:s0:c143,c582
tcontext=system_u:system_r:spc_t:s0 tclass=tun_socket permissive=0
This just suggests libvirtd has been misconfigured in some way that causes
SELinux to become enabled again.
You need to do relabelling of the TAP device, but that isn't going to
avoid this AVC - it will just change what you see as the tcontext in
the AVC. You still need to fix the configuration of libvirtd I believe.
If you run 'virsh capabilities' in your container that is showing this
problem, I expect it will say:
<secmodel>
<model>selinux</model>
<doi>0</doi>
<baselabel type='kvm'>system_u:system_r:svirt_t:s0</baselabel>
<baselabel
type='qemu'>system_u:system_r:svirt_tcg_t:s0</baselabel>
</secmodel>
If you compare with the "virsh capabilities" from a correctly operating
existing KubeVirt install, then I expect selinux won't be reported at
all, which would match your <seclabel> example from earlier.
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
1625
days inactive
1627
days old
5 comments
2 participants
participants (2)
-
Daniel P. Berrangé -
Ram Lavi