Re: [libvirt] [ovs-dev] [PATCH v2 2/2] netdev-dpdk: Support user-defined socket attribs
On Mon, May 30, 2016 at 12:29 AM, Christian Ehrhardt <christian.ehrhardt@canonical.com> wrote:
On Tue, May 24, 2016 at 4:10 PM, Aaron Conole <aconole@redhat.com> wrote:
Daniele Di Proietto <diproiettod@vmware.com> writes:
Hi Aaron,
I'm still a little bit nervous about calling chown on a (partially) user controlled file name.
I agree, that always seems scary.
Before moving forward I wanted to discuss a couple of other options:
* Ansis (in CC) suggested using -runas parameter in qemu. This way qemu can open the socket as root and drop privileges before starting guest execution.
I'm not sure how to do this with libvirt, or via the OpenStack Neutron plugin. I also don't know if it would be an acceptable workaround for users. Additionally, I recall there being something of a "don't even know if this works" around it. Maybe Christian or Ansis (both in CC) can expound on it.
Cross-posting to libvirt mailing list to hear opinion from libvirt developers. In short - the problem is that libvirtd process starts qemu process under non-root user. Since qemu starts under non-root process, then qemu can't connect to DPDK unix domain sockets created by ovs-vswitcd process that runs under "root". There are two solutions to this problem: 1. let ovs-vswitchd process to chown its socket from "root" to "libvirt" group and/or user (this is what Aarons patch proposes) 2. make libvirtd process to start qemu process under "root" but then let qemu to downgrade via "-run-as" flag after qemu has opened the Unix Domain socket. Regarding solution #2. I think the necessary changes roughly would be to: 1. invoke virCommandAddArgPair(cmd, "-runas", "libvirt") before starting qemu process; AND 2. revert virCommandSetUID() that automatically downgrades user from "root" to "libvirt" even before qemu starts. I would like to hear feasibility of such solution from libvirt developers? Or maybe there is even a better solution that I am missing? Regarding solution #1. There have already been precedents that other daemons are chowning their sockets so that non-privileged daemons could connect to them. IMHO this is not elegant from security perspective but will at least get the job done if Linux Distribution maintainers (Aaron and Christian) need to ship something out ASAP . The thing I am concerned about this solutions is that ovs-vswitchd could be used to chown() arbitrary files and sockets on file system if there happens to be a bug. Also, this socket is not created by ovs-vswitchd code base itself but rather from Intel DPDK library where ovs-vswitchd calls into.
Hi, IIRC we kind of agree that long term a proper MAC will be much better but most involved people needed something to get it working like "now". Since they are complementary (other than the fix removing a bit of the urgency for more MAC) it was kind of the least bad option.
You have to be aware that I brought up the discussion on dev@dpdk.org - see [1] and [2]: But this will take time and eventually still be the applications task to "do something" - no matter if via API or via the chmod's right now. So Aaron is trying to get something that works now until the long term things are in place, which I appreciate.
FYI - I was even more in a hurry as it was clear that OVS-2.5 won't get this in time I run with [3] for now. I never intended to suggest that, but with the discussion in place, one could ask if you (Aaron) want to pick up that instead. That would keep OVS free for now until DPDK made up the API (see [2]) for socket ownership control and this then could be implemented in OVS?
(I hope) In some months/years we will all be happy to drop this bunch of interim solutions, never the less we need it for now.
[1]: http://dpdk.org/dev/patchwork/patch/12222/ [2]: http://dpdk.org/ml/archives/dev/2015-December/030326.html [3]: https://git.launchpad.net/~ubuntu-server/dpdk/commit/?h=ubuntu-xenial-to-dpdk2.2&id=f3c7aa1b2ddea8e092ad4a89e41a0e19d01ed4e7
[...]
I think originally we quickly discussed 4 possible solutions (and hopefully I captured them correctly):
1. OVS downgrades to the ovs user, and kvm runs under the ovs group. I don't actually like this solution because kvm could then pollute the ovs database.
2. OVS runs as some user and sets the user/group ownership of the socket via chown/chmod where permissions come from the database (the original context had ovs running as root - but as I described above it doesn't need to be root provided ovs+DPDK can start without root).
3. OVS runs as some user, kvm starts as root, opens the socket and downgrades. IIRC, this doesn't actually work, or it may have implications on other projects. I don't remember exactly what was not as great about this solution, TBH.
4. OVS and KVM run as whatever users; MAC is used to enforce the layering between them.
I think solution 2 and solution 4 don't actually interfere with each other, and can be used to a complementary effect (if implemented properly) so that the MAC layer enforces access, but even without MAC, the DAC layer can provide appropriate whitelisting behavior.
I also remember several complex changes needed for the #1 and #3 that always would end up with huge effort and a high risk not being accepted. Probably that is what you refer to with "implications on other projects".
Also keep in mind the position of dpdk out of the last few discussions which I'd like to summarize as "dpdk got this path from an app, so this app OWNS that path". _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
On Mon, May 30, 2016 at 01:27:46PM -0700, Ansis Atteka wrote:
On Mon, May 30, 2016 at 12:29 AM, Christian Ehrhardt <christian.ehrhardt@canonical.com> wrote:
On Tue, May 24, 2016 at 4:10 PM, Aaron Conole <aconole@redhat.com> wrote:
Daniele Di Proietto <diproiettod@vmware.com> writes:
Hi Aaron,
I'm still a little bit nervous about calling chown on a (partially) user controlled file name.
I agree, that always seems scary.
Before moving forward I wanted to discuss a couple of other options:
* Ansis (in CC) suggested using -runas parameter in qemu. This way qemu can open the socket as root and drop privileges before starting guest execution.
I'm not sure how to do this with libvirt, or via the OpenStack Neutron plugin. I also don't know if it would be an acceptable workaround for users. Additionally, I recall there being something of a "don't even know if this works" around it. Maybe Christian or Ansis (both in CC) can expound on it.
Cross-posting to libvirt mailing list to hear opinion from libvirt developers.
In short - the problem is that libvirtd process starts qemu process under non-root user. Since qemu starts under non-root process, then qemu can't connect to DPDK unix domain sockets created by ovs-vswitcd process that runs under "root". There are two solutions to this problem: 1. let ovs-vswitchd process to chown its socket from "root" to "libvirt" group and/or user (this is what Aarons patch proposes) 2. make libvirtd process to start qemu process under "root" but then let qemu to downgrade via "-run-as" flag after qemu has opened the Unix Domain socket.
Regarding solution #2. I think the necessary changes roughly would be to: 1. invoke virCommandAddArgPair(cmd, "-runas", "libvirt") before starting qemu process; AND 2. revert virCommandSetUID() that automatically downgrades user from "root" to "libvirt" even before qemu starts. I would like to hear feasibility of such solution from libvirt developers? Or maybe there is even a better solution that I am missing?
That's not going to happen. Libvirt consider QEMU to be untrustworthy in general and so apply multiple techniques to confine QEMU before it is exec'd. This include dropping to non-0 uid/gid, applying apparmour/selinux policies, putting it in restricted cgroups, and potentially more in the future such as putting it in custom namespaces. We've no desire to use qemu's -runas, as that means we're trusting QEMU to actually drop privileges as it claims to. Libvirt's model is that libvirt will setup policies to allow QEMU access to the specific resources that it needs. so eg libvirt will chown the disk images associated with the VM to give it access. I'm missing history of this thread, but IIUC, the issue here is access to the UNIX domain socket associated with the vhost-user network backend for QEMU. If so, then I think it is a case where libvirt should be setting ownershup on that socket before starting QEMU, in order to grant access. This of course assumes there is a separate UNIX domain socket per VM that is launched. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 31 May 2016 at 09:36, Daniel P. Berrange <berrange@redhat.com> wrote:
On Mon, May 30, 2016 at 01:27:46PM -0700, Ansis Atteka wrote:
On Mon, May 30, 2016 at 12:29 AM, Christian Ehrhardt <christian.ehrhardt@canonical.com> wrote:
On Tue, May 24, 2016 at 4:10 PM, Aaron Conole <aconole@redhat.com> wrote:
Daniele Di Proietto <diproiettod@vmware.com> writes:
Hi Aaron,
I'm still a little bit nervous about calling chown on a (partially) user controlled file name.
I agree, that always seems scary.
Before moving forward I wanted to discuss a couple of other options:
* Ansis (in CC) suggested using -runas parameter in qemu. This way qemu can open the socket as root and drop privileges before starting guest execution.
I'm not sure how to do this with libvirt, or via the OpenStack Neutron plugin. I also don't know if it would be an acceptable workaround for users. Additionally, I recall there being something of a "don't even know if this works" around it. Maybe Christian or Ansis (both in CC) can expound on it.
Cross-posting to libvirt mailing list to hear opinion from libvirt developers.
In short - the problem is that libvirtd process starts qemu process under non-root user. Since qemu starts under non-root process, then qemu can't connect to DPDK unix domain sockets created by ovs-vswitcd process that runs under "root". There are two solutions to this problem: 1. let ovs-vswitchd process to chown its socket from "root" to "libvirt" group and/or user (this is what Aarons patch proposes) 2. make libvirtd process to start qemu process under "root" but then let qemu to downgrade via "-run-as" flag after qemu has opened the Unix Domain socket.
Regarding solution #2. I think the necessary changes roughly would be to: 1. invoke virCommandAddArgPair(cmd, "-runas", "libvirt") before starting qemu process; AND 2. revert virCommandSetUID() that automatically downgrades user from "root" to "libvirt" even before qemu starts. I would like to hear feasibility of such solution from libvirt developers? Or maybe there is even a better solution that I am missing?
That's not going to happen. Libvirt consider QEMU to be untrustworthy in general and so apply multiple techniques to confine QEMU before it is exec'd. This include dropping to non-0 uid/gid, applying apparmour/selinux policies, putting it in restricted cgroups, and potentially more in the future such as putting it in custom namespaces. We've no desire to use qemu's -runas, as that means we're trusting QEMU to actually drop privileges as it claims to.
Thanks for reply, Daniel! Yes, with -run-as flag it would be left at qemu's discretion to give up 'root' privileges and I understand you that it may not fit into security model chosen by libvirt.
Libvirt's model is that libvirt will setup policies to allow QEMU access to the specific resources that it needs. so eg libvirt will chown the disk images associated with the VM to give it access.
I'm missing history of this thread, but IIUC, the issue here is access to the UNIX domain socket associated with the vhost-user network backend for QEMU. If so, then I think it is a case where libvirt should be setting ownershup on that socket before starting QEMU, in order to grant access. This of course assumes there is a separate UNIX domain socket per VM that is launched.
If the current libvirt security model is that libvirt is already chown()'ing resources needed by qemu, then perhaps vhost user socket may be another thing that libvirt needs to chown()? Or do you think that it would be better for libvirt to tell the user to which Open vSwitch needs to chown the socket (via the ovs-vsctl call in libvirt/src/util/virnetdevopenvswitch.c)? Also, did I understand it correctly that libvirt also changes SELinux context for resources that qemu would be consuming so that qemu would be confined by additional Mandatory Access Control layer? If so, then I think the current libvirt security model suggests that chown()'ing and chcon()'ing should happen from libvirt and should not use Open vSwitch as a proxy to do that, because otherwise Open vSwitch SELinux policy would need to be loosened up to do such things. Also, what got me concerned is that Open vSwitch already has a Mandatory Access Control enforced under RHEL and Fedora distributions. For example, if libvirt changes SELinux context for files and sockets created by Open vSwitch then I am not sure how Open vSwitch would be able to cleanup them without getting permission denied errors. I will try this out on Fedora to see if my concern is justified. Thanks, Ansis
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
On 31 May 2016 at 10:59, Ansis Atteka <ansisatteka@gmail.com> wrote:
On 31 May 2016 at 09:36, Daniel P. Berrange <berrange@redhat.com> wrote:
On Mon, May 30, 2016 at 12:29 AM, Christian Ehrhardt <christian.ehrhardt@canonical.com> wrote:
On Tue, May 24, 2016 at 4:10 PM, Aaron Conole <aconole@redhat.com> wrote:
Daniele Di Proietto <diproiettod@vmware.com> writes:
Hi Aaron,
I'm still a little bit nervous about calling chown on a (partially) user controlled file name.
I agree, that always seems scary.
Before moving forward I wanted to discuss a couple of other
On Mon, May 30, 2016 at 01:27:46PM -0700, Ansis Atteka wrote: options:
* Ansis (in CC) suggested using -runas parameter in qemu. This way qemu can open the socket as root and drop privileges before
starting
guest execution.
I'm not sure how to do this with libvirt, or via the OpenStack Neutron plugin. I also don't know if it would be an acceptable workaround for users. Additionally, I recall there being something of a "don't even know if this works" around it. Maybe Christian or Ansis (both in CC) can expound on it.
Cross-posting to libvirt mailing list to hear opinion from libvirt developers.
In short - the problem is that libvirtd process starts qemu process under non-root user. Since qemu starts under non-root process, then qemu can't connect to DPDK unix domain sockets created by ovs-vswitcd process that runs under "root". There are two solutions to this problem: 1. let ovs-vswitchd process to chown its socket from "root" to "libvirt" group and/or user (this is what Aarons patch proposes) 2. make libvirtd process to start qemu process under "root" but then let qemu to downgrade via "-run-as" flag after qemu has opened the Unix Domain socket.
Regarding solution #2. I think the necessary changes roughly would be to: 1. invoke virCommandAddArgPair(cmd, "-runas", "libvirt") before starting qemu process; AND 2. revert virCommandSetUID() that automatically downgrades user from "root" to "libvirt" even before qemu starts. I would like to hear feasibility of such solution from libvirt developers? Or maybe there is even a better solution that I am missing?
That's not going to happen. Libvirt consider QEMU to be untrustworthy in general and so apply multiple techniques to confine QEMU before it is exec'd. This include dropping to non-0 uid/gid, applying apparmour/selinux policies, putting it in restricted cgroups, and potentially more in the future such as putting it in custom namespaces. We've no desire to use qemu's -runas, as that means we're trusting QEMU to actually drop privileges as it claims to.
Thanks for reply, Daniel! Yes, with -run-as flag it would be left at qemu's discretion to give up 'root' privileges and I understand you that it may not fit into security model chosen by libvirt.
Libvirt's model is that libvirt will setup policies to allow QEMU access to the specific resources that it needs. so eg libvirt will chown the disk images associated with the VM to give it access.
I'm missing history of this thread, but IIUC, the issue here is access to the UNIX domain socket associated with the vhost-user network backend for QEMU. If so, then I think it is a case where libvirt should be setting ownershup on that socket before starting QEMU, in order to grant access. This of course assumes there is a separate UNIX domain socket per VM that is launched.
If the current libvirt security model is that libvirt is already chown()'ing resources needed by qemu, then perhaps vhost user socket may be another thing that libvirt needs to chown()? Or do you think that it would be better for libvirt to tell the user to which Open vSwitch needs to chown the socket (via the ovs-vsctl call in libvirt/src/util/virnetdevopenvswitch.c)?
Also, did I understand it correctly that libvirt also changes SELinux context for resources that qemu would be consuming so that qemu would be confined by additional Mandatory Access Control layer? If so, then I think the current libvirt security model suggests that chown()'ing and chcon()'ing should happen from libvirt and should not use Open vSwitch as a proxy to do that, because otherwise Open vSwitch SELinux policy would need to be loosened up to do such things.
Also, what got me concerned is that Open vSwitch already has a Mandatory Access Control enforced under RHEL and Fedora distributions. For example, if libvirt changes SELinux context for files and sockets created by Open vSwitch then I am not sure how Open vSwitch would be able to cleanup them without getting permission denied errors. I will try this out on Fedora to see if my concern is justified.
I think that Libvirt chown()'ing DPDK sockets that were created by Open vSwitch may work ok, because Open vSwitch is running under root user and can reclaim the socket even after libvirt chown()'ed them away. However, I think that Libvirt chcon()'ing sockets created by Open vSwitch would not work very well with current SELinux policies. This is because by default Open vSwitch is running under SELinux openvswitch_t domain and as minimum would not be able to clean after itself in case of abrupt Open vSwitch crash and if libvirt chcon()'ed the DPDK socket away from it: # ls -laZ /var/run/openvswitch/db.sock srwxr-x---. 1 root root system_u:object_r:openvswitch_var_run_t:s0 0 May 31 18:16 /var/run/openvswitch/db.sock # chcon -t virt_var_run_t /var/run/openvswitch/db.sock # ls -laZ /var/run/openvswitch/db.sock srwxr-x---. 1 root root system_u:object_r:virt_var_run_t:s0 0 May 31 18:16 /var/run/openvswitch/db.sock # kill -SIGSEGV `pidof ovsdb-server -s` # tail /var/log/openvswitch/ovsdb-server.log 2016-06-01T01:16:04.265Z|00592|stream_unix|ERR|/var/run/openvswitch/db.sock: binding failed: Permission denied 2016-06-01T01:16:04.268Z|00593|socket_util_unix|WARN|unlinking "/var/run/openvswitch/db.sock": Permission denied 2016-06-01T01:16:04.268Z|00594|fatal_signal|WARN|could not unlink "/var/run/openvswitch/db.sock" (Permission denied) * the example above is not using DPDK socket, but OVSDB socket. Also, I probably did not set the right SELinux type. However, I think this should not change the result. I think that ideally chown()'ing and chcon()'ing of DPDK sockets should be done both in the same place - either both in libvirt or both in Open vSwitch. Daniel, if libvirt would depend on Open vSwitch to do the actual chown() and chcon() calls would that violate libvirt security model as well? What are your thoughts?
Thanks, Ansis
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :| _______________________________________________ dev mailing list dev@openvswitch.org http://openvswitch.org/mailman/listinfo/dev
On Tue, May 31, 2016 at 10:59:18AM -0700, Ansis Atteka wrote:
On 31 May 2016 at 09:36, Daniel P. Berrange <berrange@redhat.com> wrote:
On Mon, May 30, 2016 at 01:27:46PM -0700, Ansis Atteka wrote:
On Mon, May 30, 2016 at 12:29 AM, Christian Ehrhardt <christian.ehrhardt@canonical.com> wrote:
On Tue, May 24, 2016 at 4:10 PM, Aaron Conole <aconole@redhat.com> wrote:
Daniele Di Proietto <diproiettod@vmware.com> writes:
Hi Aaron,
I'm still a little bit nervous about calling chown on a (partially) user controlled file name.
I agree, that always seems scary.
Before moving forward I wanted to discuss a couple of other options:
* Ansis (in CC) suggested using -runas parameter in qemu. This way qemu can open the socket as root and drop privileges before starting guest execution.
I'm not sure how to do this with libvirt, or via the OpenStack Neutron plugin. I also don't know if it would be an acceptable workaround for users. Additionally, I recall there being something of a "don't even know if this works" around it. Maybe Christian or Ansis (both in CC) can expound on it.
Cross-posting to libvirt mailing list to hear opinion from libvirt developers.
In short - the problem is that libvirtd process starts qemu process under non-root user. Since qemu starts under non-root process, then qemu can't connect to DPDK unix domain sockets created by ovs-vswitcd process that runs under "root". There are two solutions to this problem: 1. let ovs-vswitchd process to chown its socket from "root" to "libvirt" group and/or user (this is what Aarons patch proposes) 2. make libvirtd process to start qemu process under "root" but then let qemu to downgrade via "-run-as" flag after qemu has opened the Unix Domain socket.
Regarding solution #2. I think the necessary changes roughly would be to: 1. invoke virCommandAddArgPair(cmd, "-runas", "libvirt") before starting qemu process; AND 2. revert virCommandSetUID() that automatically downgrades user from "root" to "libvirt" even before qemu starts. I would like to hear feasibility of such solution from libvirt developers? Or maybe there is even a better solution that I am missing?
That's not going to happen. Libvirt consider QEMU to be untrustworthy in general and so apply multiple techniques to confine QEMU before it is exec'd. This include dropping to non-0 uid/gid, applying apparmour/selinux policies, putting it in restricted cgroups, and potentially more in the future such as putting it in custom namespaces. We've no desire to use qemu's -runas, as that means we're trusting QEMU to actually drop privileges as it claims to.
Thanks for reply, Daniel! Yes, with -run-as flag it would be left at qemu's discretion to give up 'root' privileges and I understand you that it may not fit into security model chosen by libvirt.
Libvirt's model is that libvirt will setup policies to allow QEMU access to the specific resources that it needs. so eg libvirt will chown the disk images associated with the VM to give it access.
I'm missing history of this thread, but IIUC, the issue here is access to the UNIX domain socket associated with the vhost-user network backend for QEMU. If so, then I think it is a case where libvirt should be setting ownershup on that socket before starting QEMU, in order to grant access. This of course assumes there is a separate UNIX domain socket per VM that is launched.
If the current libvirt security model is that libvirt is already chown()'ing resources needed by qemu, then perhaps vhost user socket may be another thing that libvirt needs to chown()? Or do you think that it would be better for libvirt to tell the user to which Open vSwitch needs to chown the socket (via the ovs-vsctl call in libvirt/src/util/virnetdevopenvswitch.c)?
Also, did I understand it correctly that libvirt also changes SELinux context for resources that qemu would be consuming so that qemu would be confined by additional Mandatory Access Control layer? If so, then I think the current libvirt security model suggests that chown()'ing and chcon()'ing should happen from libvirt and should not use Open vSwitch as a proxy to do that, because otherwise Open vSwitch SELinux policy would need to be loosened up to do such things.
Yes, I think libvirt should be in charge of granting access, not openvswitch, since only libvirt has the world view of what the guest is supposed to have access to.
Also, what got me concerned is that Open vSwitch already has a Mandatory Access Control enforced under RHEL and Fedora distributions. For example, if libvirt changes SELinux context for files and sockets created by Open vSwitch then I am not sure how Open vSwitch would be able to cleanup them without getting permission denied errors. I will try this out on Fedora to see if my concern is justified.
It is entirely likely that we will need to make SELinux policy additions to allow integration between svirt & openvswitch. I'd be surprised if it worked as-is without triggering AVCs. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (3)
-
Ansis Atteka -
Ansis Atteka -
Daniel P. Berrange