[libvirt] [RFC] Adding new filesystem 'proxy' to 9p
Pass-through security model in QEMU 9p server needs root privilege to do few file operations (like chown, chmod to any mode/uid:gid). There are two issues in pass-through security model 1) TOCTTOU vulnerability: Following symbolic links in the server could provide access to files beyond 9p export path. 2) When libvirt is configured to run qemu as non-root user (for example, if qemu is configured to run as normal user 'qemu'), running file operations on pass-through security model would fail because it needs root privileges. To overcome above issues, following approach is suggested: A new filesytem type 'proxy' is introduced. Proxy FS uses chroot + socket combination for securing the vulnerability known with following symbolic links. Intention of adding a new filesystem type is to allow qemu to run in non-root mode, but doing privileged operations using socket IO. A new binary (known as proxy helper) will be provided as part of qemu. Proxy helper will chroot into 9p export path and create a socket pair or a named socket based on the command line parameter. Qemu and proxy helper will communicate using this socket. We need following changes in the libvirt code to accomodate new 'proxy' filesystem type: If qemu 9p server is configured to use 'proxy' FS, libvirt will do * Create a socket pair * invoke proxy_helper binary with one of the socket id from the pair as command line parameters to it with root privilege * invoke qemu with one of socket id from the pair as paramter to qemu virtfs after dropping to the configured user privilege. ie, libvirt will invoke proxy_helper as: proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export> and qemu will be invoked with following virtfs parameter: -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair> ,path=/tmp/,security_model=prox,mount_tag=v_pass People who want to use proxy_helper without libvirt can use following interface: $ proxy_helper -s </socket/path> -p <9p-path-to-export> With following qemu fsdev parameter: -virtfs proxy,id=<id>,socket=</socket/path>,path=/tmp/, security_model=prox,mount_tag=v_pass -- Regards, M. Mohan Kumar
On Wed, Sep 28, 2011 at 07:49:34PM +0530, M. Mohan Kumar wrote:
Pass-through security model in QEMU 9p server needs root privilege to do few file operations (like chown, chmod to any mode/uid:gid). There are two issues in pass-through security model
1) TOCTTOU vulnerability: Following symbolic links in the server could provide access to files beyond 9p export path.
2) When libvirt is configured to run qemu as non-root user (for example, if qemu is configured to run as normal user 'qemu'), running file operations on pass-through security model would fail because it needs root privileges.
To overcome above issues, following approach is suggested: A new filesytem type 'proxy' is introduced. Proxy FS uses chroot + socket combination for securing the vulnerability known with following symbolic links. Intention of adding a new filesystem type is to allow qemu to run in non-root mode, but doing privileged operations using socket IO.
A new binary (known as proxy helper) will be provided as part of qemu. Proxy helper will chroot into 9p export path and create a socket pair or a named socket based on the command line parameter. Qemu and proxy helper will communicate using this socket.
We need following changes in the libvirt code to accomodate new 'proxy' filesystem type: If qemu 9p server is configured to use 'proxy' FS, libvirt will do * Create a socket pair * invoke proxy_helper binary with one of the socket id from the pair as command line parameters to it with root privilege * invoke qemu with one of socket id from the pair as paramter to qemu virtfs after dropping to the configured user privilege. ie, libvirt will invoke proxy_helper as: proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export>
and qemu will be invoked with following virtfs parameter: -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair> ,path=/tmp/,security_model=prox,mount_tag=v_pass
Interesting proposal. Explicitly comparing the security characteristics of running QEMU as root, vs using the proxy helper * QEMU run as root - QEMU is root, with full capabilities - QEMU has read/write any file/dir, regardless of whether it is exported via 9p * QEMU run as non-root, with proxy_helper root - QEMU is non-root, with no capabilities - QEMU has write to only files with matching UID/GID - proxy_helper is root, with full capabilities - proxy_helper has read/write to any file/dir Since QEMU can send arbitrary FS calls to the proxy_helper, the overall security of the system clearly depends on the security of the proxy_helper process. If we assume that QEMU gets exploited, and that QEMU can find some flaw in the proxy_helper that it can exploit, what damage can the proxy_helper do ? Clearly we want it to not be able to read/write any files other than those exported, even when it becomes compromised, ideally also without requiring SELinux/AppArmour to make it safe. If proxy_helper is running as root with full capabilities, it can trivially escape the chroot[1], so this isn't all that nice for overall system security. The attacker simply needs to find a vulnerability in QEMU and in the proxy_helper, instead of just in QEMU. We want a stronger guarantee than that. In order to be able to chown/chmod files etc, the proxy_helper is going to require some elevated privileges, however, this does not actually imply that proxy_helper needs to run as root. You don't want the proxy_helper to be allowed todo various other things that root would be allowed, even if it is inside a chroot. eg you don't want it being allowed to reboot, mlock, change network setting, mount volumes, make device nodes, and quite alot more. Looking at the man page 'capabilities(7)', I'd venture to suggest that the proxy_helper does not in fact need to run as the 'root' UID/GID at all. All it needs is to have the CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH, CAP_FOWNER capabilities, at which point it could still run as the 'qemu' user ID (or perhaps another 'qemuhelper' user ID). So in summary, I think you have a good idea to separate the file handling code into a separate proxy_helper binary. It should not, however, be run as root with all capabilities. It can start as root, and then drop to a non-root user, only keeping the few capabilities it needs. Or in libvirt case, libvirt can spawn it with the minimal set of capabilities directly. eg * QEMU run as non-root, with proxy_helper partially privileged - QEMU is non-root, with no capabilities - QEMU has write to only files with matching UID/GID - proxy_helper is non-root, with selected capabilities - proxy_helper has read/write to any file/dir inside the chroot for the 9p export. Regards, Daniel [1] http://www.bpfh.net/simes/computing/chroot-break.html -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wednesday, September 28, 2011 08:29:06 PM Daniel P. Berrange wrote:
On Wed, Sep 28, 2011 at 07:49:34PM +0530, M. Mohan Kumar wrote:
Pass-through security model in QEMU 9p server needs root privilege to do few file operations (like chown, chmod to any mode/uid:gid). There are two issues in pass-through security model
1) TOCTTOU vulnerability: Following symbolic links in the server could provide access to files beyond 9p export path.
2) When libvirt is configured to run qemu as non-root user (for example, if qemu is configured to run as normal user 'qemu'), running file operations on pass-through security model would fail because it needs root privileges.
To overcome above issues, following approach is suggested: A new filesytem type 'proxy' is introduced. Proxy FS uses chroot + socket combination for securing the vulnerability known with following symbolic links. Intention of adding a new filesystem type is to allow qemu to run in non-root mode, but doing privileged operations using socket IO.
A new binary (known as proxy helper) will be provided as part of qemu. Proxy helper will chroot into 9p export path and create a socket pair or a named socket based on the command line parameter. Qemu and proxy helper will communicate using this socket.
We need following changes in the libvirt code to accomodate new 'proxy' filesystem type: If qemu 9p server is configured to use 'proxy' FS, libvirt will do * Create a socket pair * invoke proxy_helper binary with one of the socket id from the pair as command line parameters to it with root privilege * invoke qemu with one of socket id from the pair as paramter to qemu virtfs after dropping to the configured user privilege.
ie, libvirt will invoke proxy_helper as: proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export>
and qemu will be invoked with following virtfs parameter: -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair>
,path=/tmp/,security_model=prox,mount_tag=v_pass
Interesting proposal. Explicitly comparing the security characteristics of running QEMU as root, vs using the proxy helper
* QEMU run as root
- QEMU is root, with full capabilities - QEMU has read/write any file/dir, regardless of whether it is exported via 9p
* QEMU run as non-root, with proxy_helper root
- QEMU is non-root, with no capabilities - QEMU has write to only files with matching UID/GID - proxy_helper is root, with full capabilities - proxy_helper has read/write to any file/dir
Since QEMU can send arbitrary FS calls to the proxy_helper, the overall security of the system clearly depends on the security of the proxy_helper process. QEMU can't send arbitrary FS calls to the proxy helper. All interfaces are defined and if qemu sends arbitrary FS command, proxy helper would return an EIO. For example T_OPEN is one of the supported interface between 9p qemu server and proxy helper. For more information please look at my chroot
Thank you Daniel for the quick response, I was on leave and I could not respond to you immediately. patchset that I posted to qemu-devel few weeks ago. http://lists.gnu.org/archive/html/qemu-devel/2011-09/msg00671.html
If we assume that QEMU gets exploited, and that QEMU can find some flaw in the proxy_helper that it can exploit, what damage can the proxy_helper do ? Clearly we want it to not be able to read/write any files other than those exported, even when it becomes compromised, ideally also without requiring SELinux/AppArmour to make it safe. If proxy_helper is running as root with full capabilities, it can trivially escape the chroot[1], so this isn't all that nice for overall system security.
When proxy helper process is forked, its made sure that all open file descriptor are closed except the socket descriptor, so that proxy helper can't escape chroot jail. Main aim of providing proxy helper is to 1) Access beyond 9p export path should not be given 2) Remove the limitation that qemu should be run by root user to use pass- through security model
The attacker simply needs to find a vulnerability in QEMU and in the proxy_helper, instead of just in QEMU. We want a stronger guarantee than that.
In order to be able to chown/chmod files etc, the proxy_helper is going to require some elevated privileges, however, this does not actually imply that proxy_helper needs to run as root. You don't want the proxy_helper to be allowed todo various other things that root would be allowed, even if it is inside a chroot. eg you don't want it being allowed to reboot, mlock, change network setting, mount volumes, make device nodes, and quite alot more.
Proxy helper operations are categorized as: 1) Open/create file and return the descriptor 2) Create directory, link, symbolic link, device nodes, FIFOs etc 3) Remove/rename filesystem object 4) Return stat information 5) Return readlink() information
Looking at the man page 'capabilities(7)', I'd venture to suggest that the proxy_helper does not in fact need to run as the 'root' UID/GID at all. All it needs is to have the CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH, CAP_FOWNER capabilities, at which point it could still run as the 'qemu' user ID (or perhaps another 'qemuhelper' user ID).
IMHO, proxy helper can run as root because it will run only predefined file operations.
On Thu, Sep 29, 2011 at 11:42:47PM +0530, M. Mohan Kumar wrote:
On Wednesday, September 28, 2011 08:29:06 PM Daniel P. Berrange wrote:
On Wed, Sep 28, 2011 at 07:49:34PM +0530, M. Mohan Kumar wrote:
Pass-through security model in QEMU 9p server needs root privilege to do few file operations (like chown, chmod to any mode/uid:gid). There are two issues in pass-through security model
1) TOCTTOU vulnerability: Following symbolic links in the server could provide access to files beyond 9p export path.
2) When libvirt is configured to run qemu as non-root user (for example, if qemu is configured to run as normal user 'qemu'), running file operations on pass-through security model would fail because it needs root privileges.
To overcome above issues, following approach is suggested: A new filesytem type 'proxy' is introduced. Proxy FS uses chroot + socket combination for securing the vulnerability known with following symbolic links. Intention of adding a new filesystem type is to allow qemu to run in non-root mode, but doing privileged operations using socket IO.
A new binary (known as proxy helper) will be provided as part of qemu. Proxy helper will chroot into 9p export path and create a socket pair or a named socket based on the command line parameter. Qemu and proxy helper will communicate using this socket.
We need following changes in the libvirt code to accomodate new 'proxy' filesystem type: If qemu 9p server is configured to use 'proxy' FS, libvirt will do * Create a socket pair * invoke proxy_helper binary with one of the socket id from the pair as command line parameters to it with root privilege * invoke qemu with one of socket id from the pair as paramter to qemu virtfs after dropping to the configured user privilege.
ie, libvirt will invoke proxy_helper as: proxy_helper -i <socket_fd_from_socket_pair> -p <9p-path-to-export>
and qemu will be invoked with following virtfs parameter: -virtfs proxy,id=<id>,sock_fd=<socket_fd_from_socket_pair>
,path=/tmp/,security_model=prox,mount_tag=v_pass
Thank you Daniel for the quick response, I was on leave and I could not respond to you immediately.
Interesting proposal. Explicitly comparing the security characteristics of running QEMU as root, vs using the proxy helper
* QEMU run as root
- QEMU is root, with full capabilities - QEMU has read/write any file/dir, regardless of whether it is exported via 9p
* QEMU run as non-root, with proxy_helper root
- QEMU is non-root, with no capabilities - QEMU has write to only files with matching UID/GID - proxy_helper is root, with full capabilities - proxy_helper has read/write to any file/dir
Since QEMU can send arbitrary FS calls to the proxy_helper, the overall security of the system clearly depends on the security of the proxy_helper process. QEMU can't send arbitrary FS calls to the proxy helper. All interfaces are defined and if qemu sends arbitrary FS command, proxy helper would return an EIO. For example T_OPEN is one of the supported interface between 9p qemu server and proxy helper. For more information please look at my chroot patchset that I posted to qemu-devel few weeks ago. http://lists.gnu.org/archive/html/qemu-devel/2011-09/msg00671.html
Ok, perhaps I didn't explain what I meant by 'arbitrary FS calls' very well here. Lets, say the communication protocol between QEMU and the proxy helper has a set of FS calls to cover 'read', 'write', 'open', 'unlink', 'link'. And we have a 9pfs directory to give to the guest '/export/to-the-guest'. In normal use, the QEMU process can send things like open("/export/to-the-guest/somefile"). A malicious QEMU can send open("/etc/shadow") Or any other path that it cares about, whether or not, the path is within the location exported to the 9pfs server. The security of the overall system, thus relies on the proxy helper enforcing that the arbitrary FS calls from QEMU, are in fact only within '/export/to-the-guest' and not other directories.
If we assume that QEMU gets exploited, and that QEMU can find some flaw in the proxy_helper that it can exploit, what damage can the proxy_helper do ? Clearly we want it to not be able to read/write any files other than those exported, even when it becomes compromised, ideally also without requiring SELinux/AppArmour to make it safe. If proxy_helper is running as root with full capabilities, it can trivially escape the chroot[1], so this isn't all that nice for overall system security.
When proxy helper process is forked, its made sure that all open file descriptor are closed except the socket descriptor, so that proxy helper can't escape chroot jail.
Unfortunately that is not sufficient. Even if you have closed all file descriptors except the socket, you can still escape the chroot jail. There is a nice demo program for escaping from a chroot on this webpage: http://www.bpfh.net/simes/computing/chroot-break.html If you want to run as root, then at the very least you need to explicitly drop the CAP_SYS_CHROOT capability to block this avenue of attack, and also clear it from the boundingset to prevent it being re-acquired. Even that is likely not sufficient on its own though, if you retain all other capabilities, since those can be used to compromise other processes outside the chroot into doing things. Basically if the process inside the chroot, is still running as root, then it is not securely contained.
Main aim of providing proxy helper is to 1) Access beyond 9p export path should not be given 2) Remove the limitation that qemu should be run by root user to use pass- through security model
Yep, I agree with those aims.
The attacker simply needs to find a vulnerability in QEMU and in the proxy_helper, instead of just in QEMU. We want a stronger guarantee than that.
In order to be able to chown/chmod files etc, the proxy_helper is going to require some elevated privileges, however, this does not actually imply that proxy_helper needs to run as root. You don't want the proxy_helper to be allowed todo various other things that root would be allowed, even if it is inside a chroot. eg you don't want it being allowed to reboot, mlock, change network setting, mount volumes, make device nodes, and quite alot more.
Proxy helper operations are categorized as: 1) Open/create file and return the descriptor 2) Create directory, link, symbolic link, device nodes, FIFOs etc 3) Remove/rename filesystem object 4) Return stat information 5) Return readlink() information
Looking at the man page 'capabilities(7)', I'd venture to suggest that the proxy_helper does not in fact need to run as the 'root' UID/GID at all. All it needs is to have the CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_DAC_READ_SEARCH, CAP_FOWNER capabilities, at which point it could still run as the 'qemu' user ID (or perhaps another 'qemuhelper' user ID).
IMHO, proxy helper can run as root because it will run only predefined file operations.
That is the case if the proxy helper code is perfectly written. I am trying to think about the scenario where there is a bug (eg heap corruption / stack overflow) which allows a malicious non-root QEMU process to exploit the proxy helper to run code that it was *not* intended to run. If the proxy helper is running root with all capabilities, then a bug in the proxy helper can easily turn into a full root exploit. If the proxy helper starts as root, chroots, and then immediately drops to a non-root user, keeping only the CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FOWNER and CAP_DAC_READ_SEARCH capabilities, then a bug in the proxy helper can only be used to access files within the designated 9pfs export. If the exported directory does not contain any important host system files, then it is unlikely it can be used to create a full root exploit. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
That is the case if the proxy helper code is perfectly written. I am trying to think about the scenario where there is a bug (eg heap corruption / stack overflow) which allows a malicious non-root QEMU process to exploit the proxy helper to run code that it was *not* intended to run.
If the proxy helper is running root with all capabilities, then a bug in the proxy helper can easily turn into a full root exploit.
If the proxy helper starts as root, chroots, and then immediately drops to a non-root user, keeping only the CAP_CHOWN, CAP_DAC_OVERRIDE, CAP_FOWNER and CAP_DAC_READ_SEARCH capabilities, then a bug in the proxy helper can only be used to access files within the designated 9pfs export. If the exported directory does not contain any important host system files, then it is unlikely it can be used to create a full root exploit.
Thanks Daniel, I will add 'capabiliies' to proxy helper. CAP_FOWNER capability also need. I am working on the patches. I will post them in few days.
"Daniel P. Berrange" <berrange@redhat.com> writes:
On Thu, Sep 29, 2011 at 11:42:47PM +0530, M. Mohan Kumar wrote:
On Wednesday, September 28, 2011 08:29:06 PM Daniel P. Berrange wrote: [...]
If we assume that QEMU gets exploited, and that QEMU can find some flaw in the proxy_helper that it can exploit, what damage can the proxy_helper do ? Clearly we want it to not be able to read/write any files other than those exported, even when it becomes compromised, ideally also without requiring SELinux/AppArmour to make it safe. If proxy_helper is running as root with full capabilities, it can trivially escape the chroot[1], so this isn't all that nice for overall system security.
When proxy helper process is forked, its made sure that all open file descriptor are closed except the socket descriptor, so that proxy helper can't escape chroot jail.
Unfortunately that is not sufficient. Even if you have closed all file descriptors except the socket, you can still escape the chroot jail. There is a nice demo program for escaping from a chroot on this webpage:
http://www.bpfh.net/simes/computing/chroot-break.html [...] Basically if the process inside the chroot, is still running as root, then it is not securely contained.
As Alan Cox observed, "chroot is not and never has been a security tool." It's a tool to configure the root of absolute path names. [...]
participants (3)
-
Daniel P. Berrange -
M. Mohan Kumar -
Markus Armbruster