Hello! I'm testing user namespaces and I have quite some problem getting them to work. First of all, I have user namespaces support enabled in kernel: offlinehacker:~/ $ uname -r 3.10.10 offlinehacker:~/ $ ls /proc/self/ns/ ipc@ mnt@ net@ pid@ user@ uts@ I created simple ubuntu rootfs and when I start container without idmap, so without user namespace mappings, it works just fine: Libivrt config: <domain type='lxc'> <name>helloworld</name> <memory>102400</memory> <os> <type>exe</type> <init>/bin/dash</init> </os> <!--<idmap> <uid start='0' target='499' count='10'/> <gid start='0' target='100' count='10'/> </idmap>--> <devices> <console type='pty'/> <filesystem type='mount'> <source dir='/home/offlinehacker/rootfs'/> <target dir='/'/> </filesystem> </devices> </domain>

This is how my rootfs looks like: offlinehacker:~/ $ ls -la rootfs drwxr-xr-x 23 offlinehacker users 4096 sep 5 19:06 ./ drwxr-xr-x 59 offlinehacker users 4096 sep 5 19:06 ../ drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 bin/ drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 boot/ drwxr-xr-x 4 offlinehacker users 4096 avg 27 14:11 dev/ drwxr-xr-x 86 offlinehacker users 4096 sep 5 18:20 etc/ drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 home/ lrwxrwxrwx 1 offlinehacker users 33 avg 27 14:10 initrd.img -> /boot/initrd.img-3.2.0-52-virtual drwxr-xr-x 18 offlinehacker users 4096 avg 27 14:10 lib/ drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:10 lib64/ drwx------ 2 offlinehacker users 4096 avg 27 14:11 lost+found/ drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 media/ drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 mnt/ drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 opt/ -rw-r--r-- 1 offlinehacker vboxusers 231671365 avg 27 14:12 precise-server-cloudimg-amd64-root.tar.gz drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 proc/ drwx------ 2 offlinehacker users 4096 sep 3 23:47 root/ drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 run/ drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 sbin/ drwxr-xr-x 2 offlinehacker users 4096 mar 5 2012 selinux/ drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 srv/ drwxr-xr-x 2 offlinehacker users 4096 apr 14 2012 sys/ drwxrwxrwt 2 offlinehacker users 4096 sep 4 01:39 tmp/ drwxr-xr-x 10 offlinehacker users 4096 avg 27 14:09 usr/ drwxr-xr-x 12 offlinehacker users 4096 sep 5 18:10 var/ lrwxrwxrwx 1 offlinehacker users 29 avg 27 14:10 vmlinuz -> boot/vmlinuz-3.2.0-52-virtual And this is who I am: offlinehacker:~/ $ id uid=499(offlinehacker) gid=67(libvirtd) groups=100(users),1(wheel),57(networkmanager),67(libvirtd)

When I create the container with idmap uncommented I get the following error: offlinehacker:~/ $ virsh -c lxc:/// create helloworld.xml error: Failed to create domain from helloworld.xml error: internal error: guest failed to start: 2013-09-05 19:08:57.781+0000: 19036: debug And these are the logs: sep 05 19:08:52 laptop libvirtd[1542]: server=0x7fc8a60ddd60 client=0x7fc8a60e68d0 msg=0x7fc8a60e9380 rerr=0x7fc89a32cd40 args=0x7fc88800b4a0 ret=0x7fc88800a1c0 sep 05 19:08:52 laptop libvirtd[1542]: priv=0x7fc8a60e91f0 conn=(nil) sep 05 19:08:52 laptop libvirtd[1542]: name=lxc:/// sep 05 19:08:57 laptop libvirtd[1542]: Cannot recv data: Connection reset by peer sep 05 19:08:58 laptop libvirtd[1542]: internal error: guest failed to start: 2013-09-05 19:08:57.781+0000: 19036: debug Looks like .oldroot, dev, proc and sys gets created with mapped permissions(499:100), but container fails to start.

Hello! Okay i tried again with only staticly linked busybox: offlinehacker:~/ $ /home/offlinehacker/busybox/busybox BusyBox v1.17.1 (Debian 1:1.17.1-8) multi-call binary. Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko and others. Licensed under GPLv2. See source distribution for full notice. .... Again my id: uid=499(offlinehacker) gid=100(users) groups=100(users),1(wheel),57(networkmanager) My rootfs tree(/home/offlinehacker/busybox): busybox ├── [offlineh users ] busybox └── [offlineh users ] busybox-static_1.17.1-8_amd64.deb It works just fine as root and these folders gets created: busybox ├── [offlineh users ] busybox ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb ├── [root root ] dev ├── [root root ] .oldroot ├── [root root ] proc └── [root root ] sys When i start it with idmap with clean rootfs(dev proc sys and .oldroot deleted) i get this error, and it is a little bit different now: error: Failed to create domain from helloworld.xml error: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC And log is pretty similar: sep 06 11:24:56 laptop libvirtd[1542]: EVENT_POLL_UPDATE_HANDLE: watch=241 events=1 sep 06 11:24:57 laptop libvirtd[1542]: Skip interrupt, 1 140499747788544 sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 sep 06 11:24:57 laptop libvirtd[1542]: server=0x7fc8a60ddd60 client=0x7fc8a60e8bb0 msg=0x7fc8a60e6970 rerr=0x7fc89a32cd40 args=0x7fc8880160a0 ret=0x7fc888016030 sep 06 11:24:57 laptop libvirtd[1542]: priv=0x7fc8a60ea3a0 conn=(nil) sep 06 11:24:57 laptop libvirtd[1542]: name=lxc:/// sep 06 11:24:57 laptop libvirtd[1542]: Cannot recv data: Connection reset by peer sep 06 11:24:57 laptop libvirtd[1542]: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC Rootfs after failed creation looks like this: busybox ├── [offlineh users ] busybox ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb ├── [offlineh users ] .oldroot ├── [offlineh users ] proc └── [offlineh users ] sys I have debugging enabled, at least LIBVIRT_DEBUG is set to 1 and i get much more messages. If there's any my granular debug please let me know. PS: I forgot to mention my version of libvirt is 1.1.2 Thanks, Jaka! On Fri, Sep 6, 2013 at 3:41 AM, Gao feng <gaofeng(a)cn.fujitsu.com&gt; wrote: > On 09/06/2013 03:15 AM, Jaka Hudoklin wrote: > > Hello! > > > > I'm testing user namespaces and I have quite some problem getting them > to work. > > > > First of all, I have user namespaces support enabled in kernel: > > > > offlinehacker:~/ $ uname -r > > 3.10.10 > > offlinehacker:~/ $ ls /proc/self/ns/ > > ipc@ mnt@ net@ pid@ user@ uts@ > > > > I created simple ubuntu rootfs and when I start container without > idmap, so without user namespace mappings, it works just fine: > > > > Libivrt config: > > > > <domain type='lxc'> > > <name>helloworld</name> > > <memory>102400</memory> > > <os> > > <type>exe</type> > > <init>/bin/dash</init> > > </os> > > <!--<idmap> > > <uid start='0' target='499' count='10'/> > > <gid start='0' target='100' count='10'/> > > </idmap>--> > > <devices> > > <console type='pty'/> > > <filesystem type='mount'> > > <source dir='/home/offlinehacker/rootfs'/> > > <target dir='/'/> > > </filesystem> > > </devices> > > </domain> > > > > Your configuration looks good. > > > This is how my rootfs looks like: > > > > offlinehacker:~/ $ ls -la rootfs > > drwxr-xr-x 23 offlinehacker users 4096 sep 5 19:06 ./ > > drwxr-xr-x 59 offlinehacker users 4096 sep 5 19:06 ../ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 bin/ > > drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 boot/ > > drwxr-xr-x 4 offlinehacker users 4096 avg 27 14:11 dev/ > > drwxr-xr-x 86 offlinehacker users 4096 sep 5 18:20 etc/ > > drwxr-xr-x 3 offlinehacker users 4096 avg 27 14:11 home/ > > lrwxrwxrwx 1 offlinehacker users 33 avg 27 14:10 initrd.img > -> /boot/initrd.img-3.2.0-52-virtual > > drwxr-xr-x 18 offlinehacker users 4096 avg 27 14:10 lib/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:10 lib64/ > > drwx------ 2 offlinehacker users 4096 avg 27 14:11 lost+found/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 media/ > > drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 mnt/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 opt/ > > -rw-r--r-- 1 offlinehacker vboxusers 231671365 avg 27 14:12 > precise-server-cloudimg-amd64-root.tar.gz > > drwxr-xr-x 2 offlinehacker users 4096 apr 19 2012 proc/ > > drwx------ 2 offlinehacker users 4096 sep 3 23:47 root/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 run/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:11 sbin/ > > drwxr-xr-x 2 offlinehacker users 4096 mar 5 2012 selinux/ > > drwxr-xr-x 2 offlinehacker users 4096 avg 27 14:09 srv/ > > drwxr-xr-x 2 offlinehacker users 4096 apr 14 2012 sys/ > > drwxrwxrwt 2 offlinehacker users 4096 sep 4 01:39 tmp/ > > drwxr-xr-x 10 offlinehacker users 4096 avg 27 14:09 usr/ > > drwxr-xr-x 12 offlinehacker users 4096 sep 5 18:10 var/ > > lrwxrwxrwx 1 offlinehacker users 29 avg 27 14:10 vmlinuz -> > boot/vmlinuz-3.2.0-52-virtual > > > > And this is who I am: > > offlinehacker:~/ $ id > > uid=499(offlinehacker) gid=67(libvirtd) > groups=100(users),1(wheel),57(networkmanager),67(libvirtd) > > Can this user exec //home/offlinehacker/rootfs/bin/dash successfully ? > > > > > When I create the container with idmap uncommented I get the following > error: > > > > offlinehacker:~/ $ virsh -c lxc:/// create helloworld.xml > > error: Failed to create domain from helloworld.xml > > error: internal error: guest failed to start: 2013-09-05 > 19:08:57.781+0000: 19036: debug > > > > And these are the logs: > > sep 05 19:08:52 laptop libvirtd[1542]: server=0x7fc8a60ddd60 > client=0x7fc8a60e68d0 msg=0x7fc8a60e9380 rerr=0x7fc89a32cd40 > args=0x7fc88800b4a0 ret=0x7fc88800a1c0 > > sep 05 19:08:52 laptop libvirtd[1542]: priv=0x7fc8a60e91f0 conn=(nil) > > sep 05 19:08:52 laptop libvirtd[1542]: name=lxc:/// > > sep 05 19:08:57 laptop libvirtd[1542]: Cannot recv data: Connection > reset by peer > > sep 05 19:08:58 laptop libvirtd[1542]: internal error: guest failed to > start: 2013-09-05 19:08:57.781+0000: 19036: debug > > > > Looks like .oldroot, dev, proc and sys gets created with mapped > permissions(499:100), but container fails to start. > > > > Please enable debug mode, I need more information. > > http://libvirt.org/logging.html > > Thansk >

On 09/06/2013 07:32 PM, Jaka Hudoklin wrote: > Hello! > > Okay i tried again with only staticly linked busybox: > offlinehacker:~/ $ /home/offlinehacker/busybox/busybox > BusyBox v1.17.1 (Debian 1:1.17.1-8) multi-call binary. > Copyright (C) 1998-2009 Erik Andersen, Rob Landley, Denys Vlasenko > and others. Licensed under GPLv2. > See source distribution for full notice. > .... > > Again my id: > uid=499(offlinehacker) gid=100(users) groups=100(users),1(wheel),57(networkmanager) > > My rootfs tree(/home/offlinehacker/busybox): > busybox > ├── [offlineh users ] busybox > └── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > > It works just fine as root and these folders gets created: > busybox > ├── [offlineh users ] busybox > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > ├── [root root ] dev > ├── [root root ] .oldroot > ├── [root root ] proc > └── [root root ] sys > > When i start it with idmap with clean rootfs(dev proc sys and .oldroot deleted) i get this error, and it is a little bit different now: > error: Failed to create domain from helloworld.xml > error: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC > > And log is pretty similar: > sep 06 11:24:56 laptop libvirtd[1542]: EVENT_POLL_UPDATE_HANDLE: watch=241 events=1 > sep 06 11:24:57 laptop libvirtd[1542]: Skip interrupt, 1 140499747788544 > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > sep 06 11:24:57 laptop libvirtd[1542]: OBJECT_REF: obj=0x7fc878000c90 > sep 06 11:24:57 laptop libvirtd[1542]: server=0x7fc8a60ddd60 client=0x7fc8a60e8bb0 msg=0x7fc8a60e6970 rerr=0x7fc89a32cd40 args=0x7fc8880160a0 ret=0x7fc888016030 > sep 06 11:24:57 laptop libvirtd[1542]: priv=0x7fc8a60ea3a0 conn=(nil) > sep 06 11:24:57 laptop libvirtd[1542]: name=lxc:/// > sep 06 11:24:57 laptop libvirtd[1542]: Cannot recv data: Connection reset by peer > sep 06 11:24:57 laptop libvirtd[1542]: internal error: guest failed to start: 2013-09-06 11:24:57.088+0000: 5794: debug : virFileC > > Rootfs after failed creation looks like this: > busybox > ├── [offlineh users ] busybox > ├── [offlineh users ] busybox-static_1.17.1-8_amd64.deb > ├── [offlineh users ] .oldroot > ├── [offlineh users ] proc > └── [offlineh users ] sys > > I have debugging enabled, at least LIBVIRT_DEBUG is set to 1 and i get much more messages. If there's any my granular debug please let me know. > > PS: I forgot to mention my version of libvirt is 1.1.2 > OK, I get it, Maybe you need this patch 1583dfda7c4e5ad71efe0615c06e5676528d8203 LXC: Don't mount securityfs when user namespace enabled Thanks

It seems to be working now, what I needed was libvirt built with libcap support and also securityfs patch. Thanks!