On 4/13/2016 10:00 AM, John Ferlan wrote:
On 04/13/2016 09:23 AM, TomK wrote:
On Tue, Apr 12, 2016 at 06:24:16PM -0400, TomK wrote:
On 4/12/2016 5:08 PM, John Ferlan wrote:
Having/using a root squash via an NFS pool is "easy" (famous last words)
Create some pool XML (taking the example I have)
% cat nfs.xml <pool type='netfs'> <name>rootsquash</name> <source> <host name='localhost'/> <dir path='/home/bzs/rootsquash/nfs'/> <format type='nfs'/> </source> <target> <path>/tmp/netfs-rootsquash-pool</path> <permissions> <mode>0755</mode> <owner>107</owner> <group>107</group> </permissions> </target> </pool>
In this case 107:107 is qemu:qemu and I used 'localhost' as the hostname, but that can be a fqdn or ip-addr to the NFS server.
You've already seen my /etc/exports
virsh pool-define nfs.xml virsh pool-build rootsquash virsh pool-start rootsquash virsh vol-list rootsquash
Now instead of
<disk type='file' device='disk'> <source file='/var/lib/one//datastores/0/38/disk.0'/> <target dev='hda'/> <driver name='qemu' type='qcow2' cache='none'/> </disk>
Something like:
<disk type='volume' device='disk'> <driver name='qemu' type='qemu' cache='none'/> <source pool='rootsquash' volume='disk.0'/> <target dev='hda'/> </disk>
The volume name may be off, but it's perhaps close. I forget how to do the readonly bit for a pool (again, my focus is elsewhere).
Of course you'd have to adjust the nfs.xml above to suit your environment and see what you see/get. The privileges for the pool and volumes in the pool become the key to how libvirt decides to "request access" to the volume. "disk.1" having read access is probably not an issue since you seem to be using it as a CDROM; however, "disk.0" is going to be used for read/write - thus would have to be appropriately configured...
Thanks John! Appreciated again.
No worries, handle what's on the plate now and earmark this for checking once you have some free cycles. I can temporarily hop on one leg by using Martin Kletzander's workaround (It's a POC at the moment).
I'll have a look at your instructions further but wanted to find out if that config nfs.xml is a one time thing correct? I'm spinning these up at will via the OpenNebula GUI and if I have update for each VM, that breaks the Cloud provisioning. I'll go over your notes again. I'm optimistic. :)
The more I'm thinking about it, the more I am convinced that the workaround is actually not a workaround. The only thing you need to do is having execute for others (precisely for 'nobody' on the nfs share) in the whole path on all directories. Without that even the pool won't be usable from libvirt. However it does not pose any security issue as it only allows others to check the path. When qemu is launched, it has the proper "label", meaning uid:gid to access the file so it will be able to read/write or whatever permissions you set there. It's just that libvirt does some checks that the path exists for example.
Hope that's understandable and it will resolve your issue permanently.
Have a nice day, Martin
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users The only reason I said that this might be a 'workaround' is due to John Farlan commenting that he'll look at this later on. Ideally the opennebula community keeps the other permissions to nill and presumably
On 4/13/2016 1:33 AM, Martin Kletzander wrote: they work on NFSv3 per the forum topic I included earlier from them. But if setting the permissions on nobody to allow for the functionality, I would be comfortable with that.
Martin and I were taking different paths... But yes, it certainly makes sense given your error message about canonical path and the need for eXecute permissions... I think I started wondering about that first, but then jumped into the NFS pool because that's what my reference point is for root-squash. Since root squash essentially sends root requests as "nfsnobody" (IOW: others not the user or group), then the "o+x" approach is the solution if you're going directly at the file.
John
Yes, appears the o+x is the only way right now. It definitely tries to access the share as root though, on CentOS 7 since I also tried to add nfsnobody and nobody to the oneadmin group and that did not work either. Seems OpenNebula doesn't have this issue with NFSv3 running on Ubuntu: [root@mdskvm-p01 ~]# rmdir /tmp/netfs-rootsquash-pool [root@mdskvm-p01 ~]# cat nfs.xml <pool type='netfs'> <name>rootsquash</name> <source> <host name='opennebula01'/> <dir path='/var/lib/one'/> <format type='nfs'/> </source> <target> <path>/tmp/netfs-rootsquash-pool</path> <permissions> <mode>0755</mode> <owner>9869</owner> <group>9869</group> </permissions> </target> </pool> [root@mdskvm-p01 ~]# [root@mdskvm-p01 ~]# [root@mdskvm-p01 ~]# virsh pool-define nfs.xml Pool rootsquash defined from nfs.xml [root@mdskvm-p01 ~]# virsh pool-build rootsquash Pool rootsquash built [root@mdskvm-p01 ~]# virsh pool-start rootsquash error: Failed to start pool rootsquash error: cannot open path '/tmp/netfs-rootsquash-pool': Permission denied [root@mdskvm-p01 ~]# virsh vol-list rootsquash error: Failed to list volumes error: Requested operation is not valid: storage pool 'rootsquash' is not active [root@mdskvm-p01 ~]# ls -altri /tmp/netfs-rootsquash-pool total 4 133 drwxrwxrwt. 14 root root 4096 Apr 14 00:05 .. 68785924 drwxr-xr-x 2 oneadmin oneadmin 6 Apr 14 00:05 . [root@mdskvm-p01 ~]# [root@mdskvm-p01 ~]# id oneadmin uid=9869(oneadmin) gid=9869(oneadmin) groups=9869(oneadmin),992(libvirt),36(kvm) [root@mdskvm-p01 ~]# id nobody uid=99(nobody) gid=99(nobody) groups=99(nobody),9869(oneadmin) [root@mdskvm-p01 ~]# id nfsnobody uid=65534(nfsnobody) gid=65534(nfsnobody) groups=65534(nfsnobody),9869(oneadmin) [root@mdskvm-p01 ~]# id root uid=0(root) gid=0(root) groups=0(root) [root@mdskvm-p01 ~]# [root@mdskvm-p01 ~]# ps -ef|grep -i libvirtd root 352 31058 0 00:31 pts/1 00:00:00 grep --color=auto -i libvirtd root 1459 1 0 Apr11 ? 00:07:40 /usr/sbin/libvirtd --listen --config /etc/libvirt/libvirtd.conf [root@mdskvm-p01 ~]# [root@mdskvm-p01 ~]# umount /var/lib/one [root@mdskvm-p01 ~]# mount --no-canonicalize /var/lib/one [root@mdskvm-p01 ~]# umount /var/lib/one [root@mdskvm-p01 ~]# mount /var/lib/one [root@mdskvm-p01 ~]# mount|tail -n 1 192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70) [root@mdskvm-p01 ~]# umount /var/lib/one [root@mdskvm-p01 ~]# mount --no-canonicalize /var/lib/one [root@mdskvm-p01 ~]# mount|tail -n 1 192.168.0.70:/var/lib/one on /var/lib/one type nfs4 (rw,relatime,vers=4.0,rsize=8192,wsize=8192,namlen=255,soft,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.0.60,local_lock=none,addr=192.168.0.70) [root@mdskvm-p01 ~]# su - oneadmin Last login: Thu Apr 14 00:27:59 EDT 2016 on pts/0 [oneadmin@mdskvm-p01 ~]$ virsh -d 1 --connect qemu:///system create /var/lib/one//datastores/0/47/deployment.0 create: file(optdata): /var/lib/one//datastores/0/47/deployment.0 error: Failed to create domain from /var/lib/one//datastores/0/47/deployment.0 error: can't canonicalize path '/var/lib/one//datastores/0/47/disk.1': Permission denied [oneadmin@mdskvm-p01 ~]$ CONTROLLER ( NFS Server ) [oneadmin@opennebula01 one]$ ls -ld /var{,/lib{,/one{,/datastores{,/0{,/47{,/disk.1}}}}}} drwxr-xr-x. 19 root root 4096 Apr 4 21:26 /var drwxr-xr-x. 28 root root 4096 Apr 13 03:30 /var/lib drwxr-x---. 12 oneadmin oneadmin 4096 Apr 14 00:40 /var/lib/one drwxrwxr-x 6 oneadmin oneadmin 46 Mar 31 02:44 /var/lib/one/datastores drwxrwxr-x 8 oneadmin oneadmin 60 Apr 13 23:31 /var/lib/one/datastores/0 drwxrwxr-x 2 oneadmin oneadmin 68 Apr 13 23:32 /var/lib/one/datastores/0/47 -rw-r--r-- 1 oneadmin oneadmin 372736 Apr 13 23:32 /var/lib/one/datastores/0/47/disk.1 [oneadmin@opennebula01 one]$ NODE ( NFS Client ) [oneadmin@mdskvm-p01 ~]$ ls -ld /var{,/lib{,/one{,/datastores{,/0{,/47{,/disk.1}}}}}} drwxr-xr-x. 21 root root 4096 Apr 11 07:10 /var drwxr-xr-x. 45 root root 4096 Apr 13 04:11 /var/lib drwxr-x--- 12 oneadmin oneadmin 4096 Apr 14 00:39 /var/lib/one drwxrwxr-x 6 oneadmin oneadmin 46 Mar 31 02:44 /var/lib/one/datastores drwxrwxr-x 8 oneadmin oneadmin 60 Apr 13 23:31 /var/lib/one/datastores/0 drwxrwxr-x 2 oneadmin oneadmin 68 Apr 13 23:32 /var/lib/one/datastores/0/47 -rw-r--r-- 1 oneadmin oneadmin 372736 Apr 13 23:32 /var/lib/one/datastores/0/47/disk.1 [oneadmin@mdskvm-p01 ~]$ Cheers, Tom K. ------------------------------------------------------------------------------------- Living on earth is expensive, but it includes a free trip around the sun.