[libvirt] NFS over AF_VSOCK in <filesystem>
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server. Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax. The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down. Does this sound like a responsibility that libvirt should handle? Stefan
On Mon, May 09, 2016 at 04:57:17PM +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax.
The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down.
Can you elaborate on the interaction between QEMU and the NFS server on the host ? What actually needed changing in nfs-ganesha to support virtio-vsock ? I thought that on the host side we wouldn't need any changes, because QEMU would just talk to a regular NFS server over TCP, and the only virtio-vsock changes would be in QEMU and the guest kernel.
Does this sound like a responsibility that libvirt should handle?
Certainly, we'll want to have libvirt manage host side setup tasks that may be required, not least to control security policy wrt VMs accessing filesystem exports. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, May 09, 2016 at 05:18:42PM +0100, Daniel P. Berrange wrote:
On Mon, May 09, 2016 at 04:57:17PM +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax.
The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down.
Can you elaborate on the interaction between QEMU and the NFS server on the host ? What actually needed changing in nfs-ganesha to support virtio-vsock ? I thought that on the host side we wouldn't need any changes, because QEMU would just talk to a regular NFS server over TCP, and the only virtio-vsock changes would be in QEMU and the guest kernel.
The NFS protocol (and SUNRPC) is aware of the transport its running over. In order to fully support the protocol it needs to know about AF_VSOCK and addressing. The NFS server changes allow specifying an AF_VSOCK listen port. The machine name format in /etc/exports or equivalent config also needs to support vsock. Stefan
On Tue, May 10, 2016 at 01:03:48PM +0100, Stefan Hajnoczi wrote:
On Mon, May 09, 2016 at 05:18:42PM +0100, Daniel P. Berrange wrote:
On Mon, May 09, 2016 at 04:57:17PM +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax.
The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down.
Can you elaborate on the interaction between QEMU and the NFS server on the host ? What actually needed changing in nfs-ganesha to support virtio-vsock ? I thought that on the host side we wouldn't need any changes, because QEMU would just talk to a regular NFS server over TCP, and the only virtio-vsock changes would be in QEMU and the guest kernel.
The NFS protocol (and SUNRPC) is aware of the transport its running over. In order to fully support the protocol it needs to know about AF_VSOCK and addressing.
The NFS server changes allow specifying an AF_VSOCK listen port. The machine name format in /etc/exports or equivalent config also needs to support vsock.
So from host POV, in our current model of exposing host FS to the guest where libvirt wants control over managing exports, I don't think we would be inclined to use the in-kernel NFS server at all, nor would we likely use the main system ganesha server instance. Instead what I think we'd want is to run an isolated instance of ganesha for each QEMU VM that requires filesystem passthrough. First of all this removes any unpredictability around setup, as arbitrary admin config changes to the default system ganesha server would not conflict with settings libvirt needed to make for QEMU. Second it would let us place the ganesha server associated with a VM in the same cgroup, so we can ensure resources limits associated with the VM get applied. Third it would let us apply the per-VM svirt MCS level to each ganesha, to ensure that there's no risk of cross-VM attack vectors via the ganesha services. Ideally QEMU would talk to ganesha over a private UNIX domain socket though it looks like ganesha only has the ability to use TCP or UDP right now, so that'd be something we need to add support for. Meanwhile we could also extend the libvirt <filesystem> passthrough syntax, so that instead of just allowing passthrough from a local host directory, we could tell it to passthrough straight from some external server. eg, we could tell it to passthrough straight from some externally managed ganesha, whether it be on the local host or on a remote host. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Wed, May 11, 2016 at 03:42:34PM +0100, Daniel P. Berrange wrote:
On Tue, May 10, 2016 at 01:03:48PM +0100, Stefan Hajnoczi wrote:
On Mon, May 09, 2016 at 05:18:42PM +0100, Daniel P. Berrange wrote:
On Mon, May 09, 2016 at 04:57:17PM +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax.
The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down.
Can you elaborate on the interaction between QEMU and the NFS server on the host ? What actually needed changing in nfs-ganesha to support virtio-vsock ? I thought that on the host side we wouldn't need any changes, because QEMU would just talk to a regular NFS server over TCP, and the only virtio-vsock changes would be in QEMU and the guest kernel.
The NFS protocol (and SUNRPC) is aware of the transport its running over. In order to fully support the protocol it needs to know about AF_VSOCK and addressing.
The NFS server changes allow specifying an AF_VSOCK listen port. The machine name format in /etc/exports or equivalent config also needs to support vsock.
So from host POV, in our current model of exposing host FS to the guest where libvirt wants control over managing exports, I don't think we would be inclined to use the in-kernel NFS server at all, nor would we likely use the main system ganesha server instance.
Instead what I think we'd want is to run an isolated instance of ganesha for each QEMU VM that requires filesystem passthrough. First of all this removes any unpredictability around setup, as arbitrary admin config changes to the default system ganesha server would not conflict with settings libvirt needed to make for QEMU. Second it would let us place the ganesha server associated with a VM in the same cgroup, so we can ensure resources limits associated with the VM get applied. Third it would let us apply the per-VM svirt MCS level to each ganesha, to ensure that there's no risk of cross-VM attack vectors via the ganesha services. Ideally QEMU would talk to ganesha over a private UNIX domain socket though it looks like ganesha only has the ability to use TCP or UDP right now, so that'd be something we need to add support for.
virtio-vsock uses a vhost kernel module so that traffic comes from the host's network stack, not from QEMU. So there wouldn't be a unix domain socket connecting the VM to ganesha (although something along those lines could be implemented in the future). One issue we need to consider is that the host's port namespace is global. If there is an isolated ganesha for each VM then either each ganesha must bind to a separate port (2049, 2050, etc) or there needs to be some namespace/firewall magic to send traffic for 2049 to different ganesha processes. Stefan
On Mon, May 16, 2016 at 04:58:48PM -0700, Stefan Hajnoczi wrote:
On Wed, May 11, 2016 at 03:42:34PM +0100, Daniel P. Berrange wrote:
On Tue, May 10, 2016 at 01:03:48PM +0100, Stefan Hajnoczi wrote:
On Mon, May 09, 2016 at 05:18:42PM +0100, Daniel P. Berrange wrote:
On Mon, May 09, 2016 at 04:57:17PM +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax.
The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down.
Can you elaborate on the interaction between QEMU and the NFS server on the host ? What actually needed changing in nfs-ganesha to support virtio-vsock ? I thought that on the host side we wouldn't need any changes, because QEMU would just talk to a regular NFS server over TCP, and the only virtio-vsock changes would be in QEMU and the guest kernel.
The NFS protocol (and SUNRPC) is aware of the transport its running over. In order to fully support the protocol it needs to know about AF_VSOCK and addressing.
The NFS server changes allow specifying an AF_VSOCK listen port. The machine name format in /etc/exports or equivalent config also needs to support vsock.
So from host POV, in our current model of exposing host FS to the guest where libvirt wants control over managing exports, I don't think we would be inclined to use the in-kernel NFS server at all, nor would we likely use the main system ganesha server instance.
Instead what I think we'd want is to run an isolated instance of ganesha for each QEMU VM that requires filesystem passthrough. First of all this removes any unpredictability around setup, as arbitrary admin config changes to the default system ganesha server would not conflict with settings libvirt needed to make for QEMU. Second it would let us place the ganesha server associated with a VM in the same cgroup, so we can ensure resources limits associated with the VM get applied. Third it would let us apply the per-VM svirt MCS level to each ganesha, to ensure that there's no risk of cross-VM attack vectors via the ganesha services. Ideally QEMU would talk to ganesha over a private UNIX domain socket though it looks like ganesha only has the ability to use TCP or UDP right now, so that'd be something we need to add support for.
virtio-vsock uses a vhost kernel module so that traffic comes from the host's network stack, not from QEMU. So there wouldn't be a unix domain socket connecting the VM to ganesha (although something along those lines could be implemented in the future).
Hmm, are there any docs explaining the virtio-vsock architecture/setup in a bit more detail ? It feels some undesirable for the vsock backend to be directly connected to the host network - feels like it would be opening avenues for attacking the host network services.
One issue we need to consider is that the host's port namespace is global. If there is an isolated ganesha for each VM then either each ganesha must bind to a separate port (2049, 2050, etc) or there needs to be some namespace/firewall magic to send traffic for 2049 to different ganesha processes.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Thu, May 19, 2016 at 10:27:17AM +0100, Daniel P. Berrange wrote:
On Mon, May 16, 2016 at 04:58:48PM -0700, Stefan Hajnoczi wrote:
On Wed, May 11, 2016 at 03:42:34PM +0100, Daniel P. Berrange wrote:
On Tue, May 10, 2016 at 01:03:48PM +0100, Stefan Hajnoczi wrote:
On Mon, May 09, 2016 at 05:18:42PM +0100, Daniel P. Berrange wrote:
On Mon, May 09, 2016 at 04:57:17PM +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax.
The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down.
Can you elaborate on the interaction between QEMU and the NFS server on the host ? What actually needed changing in nfs-ganesha to support virtio-vsock ? I thought that on the host side we wouldn't need any changes, because QEMU would just talk to a regular NFS server over TCP, and the only virtio-vsock changes would be in QEMU and the guest kernel.
The NFS protocol (and SUNRPC) is aware of the transport its running over. In order to fully support the protocol it needs to know about AF_VSOCK and addressing.
The NFS server changes allow specifying an AF_VSOCK listen port. The machine name format in /etc/exports or equivalent config also needs to support vsock.
So from host POV, in our current model of exposing host FS to the guest where libvirt wants control over managing exports, I don't think we would be inclined to use the in-kernel NFS server at all, nor would we likely use the main system ganesha server instance.
Instead what I think we'd want is to run an isolated instance of ganesha for each QEMU VM that requires filesystem passthrough. First of all this removes any unpredictability around setup, as arbitrary admin config changes to the default system ganesha server would not conflict with settings libvirt needed to make for QEMU. Second it would let us place the ganesha server associated with a VM in the same cgroup, so we can ensure resources limits associated with the VM get applied. Third it would let us apply the per-VM svirt MCS level to each ganesha, to ensure that there's no risk of cross-VM attack vectors via the ganesha services. Ideally QEMU would talk to ganesha over a private UNIX domain socket though it looks like ganesha only has the ability to use TCP or UDP right now, so that'd be something we need to add support for.
virtio-vsock uses a vhost kernel module so that traffic comes from the host's network stack, not from QEMU. So there wouldn't be a unix domain socket connecting the VM to ganesha (although something along those lines could be implemented in the future).
Hmm, are there any docs explaining the virtio-vsock architecture/setup in a bit more detail ? It feels some undesirable for the vsock backend to be directly connected to the host network - feels like it would be opening avenues for attacking the host network services.
Only AF_VSOCK host network services would be listening. Today there are none. There are no docs yet but it's pretty much the same as VMware vSockets (which shares the AF_VSOCK address family and some of the code with virtio-vsock). The VMware guide covering sockets usage is here: https://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/ws9_esx60_vmc... Stefan
On Tue, May 31, 2016 at 03:00:24PM -0700, Stefan Hajnoczi wrote:
On Thu, May 19, 2016 at 10:27:17AM +0100, Daniel P. Berrange wrote:
On Mon, May 16, 2016 at 04:58:48PM -0700, Stefan Hajnoczi wrote:
On Wed, May 11, 2016 at 03:42:34PM +0100, Daniel P. Berrange wrote:
On Tue, May 10, 2016 at 01:03:48PM +0100, Stefan Hajnoczi wrote:
On Mon, May 09, 2016 at 05:18:42PM +0100, Daniel P. Berrange wrote:
On Mon, May 09, 2016 at 04:57:17PM +0100, Stefan Hajnoczi wrote: > virtio-vsock support has been added to the nfs-ganesha NFS server. I'm > currently working on upstreaming virtio-vsock into Linux and QEMU. I > also have patches for the Linux NFS client and server. > > Users wishing to share a file system with the guest will need to > configure the NFS server. Perhaps libvirt could handle that given that > it already has <filesystem> syntax. > > The basic task is setting up either the kernel nfsd or nfs-ganesha for > the VM to access the NFS export(s). When the VM is destroy the NFS > server can be shut down.
Can you elaborate on the interaction between QEMU and the NFS server on the host ? What actually needed changing in nfs-ganesha to support virtio-vsock ? I thought that on the host side we wouldn't need any changes, because QEMU would just talk to a regular NFS server over TCP, and the only virtio-vsock changes would be in QEMU and the guest kernel.
The NFS protocol (and SUNRPC) is aware of the transport its running over. In order to fully support the protocol it needs to know about AF_VSOCK and addressing.
The NFS server changes allow specifying an AF_VSOCK listen port. The machine name format in /etc/exports or equivalent config also needs to support vsock.
So from host POV, in our current model of exposing host FS to the guest where libvirt wants control over managing exports, I don't think we would be inclined to use the in-kernel NFS server at all, nor would we likely use the main system ganesha server instance.
Instead what I think we'd want is to run an isolated instance of ganesha for each QEMU VM that requires filesystem passthrough. First of all this removes any unpredictability around setup, as arbitrary admin config changes to the default system ganesha server would not conflict with settings libvirt needed to make for QEMU. Second it would let us place the ganesha server associated with a VM in the same cgroup, so we can ensure resources limits associated with the VM get applied. Third it would let us apply the per-VM svirt MCS level to each ganesha, to ensure that there's no risk of cross-VM attack vectors via the ganesha services. Ideally QEMU would talk to ganesha over a private UNIX domain socket though it looks like ganesha only has the ability to use TCP or UDP right now, so that'd be something we need to add support for.
virtio-vsock uses a vhost kernel module so that traffic comes from the host's network stack, not from QEMU. So there wouldn't be a unix domain socket connecting the VM to ganesha (although something along those lines could be implemented in the future).
Hmm, are there any docs explaining the virtio-vsock architecture/setup in a bit more detail ? It feels some undesirable for the vsock backend to be directly connected to the host network - feels like it would be opening avenues for attacking the host network services.
Only AF_VSOCK host network services would be listening. Today there are none.
Ahhh, that makes more sense - I was misinterpreting what you meant, thinking that the host side would end up listening on AF_INET(6) sockets on the host interface(s).
There are no docs yet but it's pretty much the same as VMware vSockets (which shares the AF_VSOCK address family and some of the code with virtio-vsock). The VMware guide covering sockets usage is here: https://pubs.vmware.com/vsphere-60/topic/com.vmware.ICbase/PDF/ws9_esx60_vmc...
Thanks, that's helpful Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, May 09, 2016 at 16:57:17 +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Just out of curiosity are you planning to add support for virtio-sock to nfsroot? Jirka
On Tue, May 10, 2016 at 08:26:30AM +0200, Jiri Denemark wrote:
On Mon, May 09, 2016 at 16:57:17 +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Just out of curiosity are you planning to add support for virtio-sock to nfsroot?
I think nfsroot over AF_VSOCK should be possible but I haven't tried it. Why is nfsroot special? Stefan
On Tue, May 10, 2016 at 13:04:20 +0100, Stefan Hajnoczi wrote:
On Tue, May 10, 2016 at 08:26:30AM +0200, Jiri Denemark wrote:
On Mon, May 09, 2016 at 16:57:17 +0100, Stefan Hajnoczi wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Just out of curiosity are you planning to add support for virtio-sock to nfsroot?
I think nfsroot over AF_VSOCK should be possible but I haven't tried it.
Why is nfsroot special?
Well, after reading your reply to Daniel's email I think it's not special at all. Thanks for shedding some more light on this. Jirka
On Mon, May 9, 2016 at 4:57 PM, Stefan Hajnoczi <stefanha@redhat.com> wrote:
virtio-vsock support has been added to the nfs-ganesha NFS server. I'm currently working on upstreaming virtio-vsock into Linux and QEMU. I also have patches for the Linux NFS client and server.
Users wishing to share a file system with the guest will need to configure the NFS server. Perhaps libvirt could handle that given that it already has <filesystem> syntax.
The basic task is setting up either the kernel nfsd or nfs-ganesha for the VM to access the NFS export(s). When the VM is destroy the NFS server can be shut down.
For anyone interested in how configuring a running ganesha instance works in practice, here is how it is done by the glusterfs Manila driver: https://github.com/openstack/manila/blob/master/manila/share/drivers/ganesha... (The short answer is "DBUS" :-)) John
participants (4)
-
Daniel P. Berrange -
Jiri Denemark -
John Spray -
Stefan Hajnoczi