[libvirt-users] Permission problem with /dev/net/tun
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi lxc folks, the symptom my libvirt LXC container suffers from is: root@depot:/dev/net# ls -la total 0 drwxr-xr-x 2 root root 40 Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 .. root@depot:/dev/net# mknod tun c 10 200 mknod: `tun': Operation not permitted The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that was formerly installed from precise and then upgraded. The guest is Ubuntu precise; however, I see the same symptom in another raring container on the same host. What I tried to resolve this: 1) On the host, I echoed various stuff to the cgroup device files: cd /sys/fs/cgroup/devices/libvirt/lxc echo "c 10:200 rwm" > devices.allow echo "c 10:200 rwm" > depot/devices.allow echo a > depot/devices.allow ... and I see the successful results in depot/devices.list, but no success. 2) I inserted a line "/dev/net/tun rwk," into /etc/apparmor.d/abstractions/lxc/container-base - no change. (I know, it seems kind of pointless - because it's about permissions to a device, not a path. You may deduce my desperation from this ... SELinux is not active. Mounts on /dev look normal to me: devfs on /dev type tmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) devpts on /dev/ptmx type devpts (rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666) cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices) Now comes the weird part: Once I umount /dev inside the container, the "hidden" /dev appears which contains a usable /dev/net/tun. So the mknod problem is probably due to the dropped capabilities - but why/how mounts the container a more restricted /dev on top of the prepared one ...? Thanks & regards Thomas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHYdysACgkQiMyIQtYO79yUMQCfUrbaAWcZsqYkopRL6F2evWdq XnMAoOq6/wopgBGZniWDGnIiYBEsFX61 =BIr9 -----END PGP SIGNATURE-----
On Sat, Jul 06, 2013 at 09:59:39PM +0200, Thomas Karcher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi lxc folks,
the symptom my libvirt LXC container suffers from is:
root@depot:/dev/net# ls -la total 0 drwxr-xr-x 2 root root 40 Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
root@depot:/dev/net# mknod tun c 10 200 mknod: `tun': Operation not permitted
The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that was formerly installed from precise and then upgraded. The guest is Ubuntu precise; however, I see the same symptom in another raring container on the same host.
What I tried to resolve this:
1) On the host, I echoed various stuff to the cgroup device files:
cd /sys/fs/cgroup/devices/libvirt/lxc echo "c 10:200 rwm" > devices.allow echo "c 10:200 rwm" > depot/devices.allow echo a > depot/devices.allow
... and I see the successful results in depot/devices.list, but no success.
2) I inserted a line "/dev/net/tun rwk," into /etc/apparmor.d/abstractions/lxc/container-base - no change. (I know, it seems kind of pointless - because it's about permissions to a device, not a path. You may deduce my desperation from this ...
SELinux is not active. Mounts on /dev look normal to me:
devfs on /dev type tmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) devpts on /dev/ptmx type devpts (rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666) cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
Now comes the weird part: Once I umount /dev inside the container, the "hidden" /dev appears which contains a usable /dev/net/tun. So the mknod problem is probably due to the dropped capabilities - but why/how mounts the container a more restricted /dev on top of the prepared one ...?
Allowing the container direct access to the hosts' /dev would be a security flaw, so libvirt sets up a private /dev for the container. Allowing the container to use mknod would also be insecure, so we blocking mknod using both cgroups device ACL, and also droping the CAP_MKNOD capability. http://libvirt.org/drvlxc.html#devnodes Any device that the container is authorized to access per the XML configuration, will be pre-created in the container's /dev. To explicitly allow /dev/net/tun you need to tell libvirt about it. http://libvirt.org/formatdomain.html#elementsHostDevCaps Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Daniel, On 07/08/2013 11:41 AM, Daniel P. Berrange wrote:
the symptom my libvirt LXC container suffers from is: root@depot:/dev/net# ls -la total 0 drwxr-xr-x 2 root root 40 Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 .. root@depot:/dev/net# mknod tun c 10 200 mknod: `tun': Operation not permitted Allowing the container direct access to the hosts' /dev would be a security flaw, so libvirt sets up a private /dev for the container. Allowing the container to use mknod would also be insecure, so we blocking mknod using both cgroups device ACL, and also droping the CAP_MKNOD capability. http://libvirt.org/drvlxc.html#devnodes
Good to know.
Any device that the container is authorized to access per the XML configuration, will be pre-created in the container's /dev. To explicitly allow /dev/net/tun you need to tell libvirt about it. http://libvirt.org/formatdomain.html#elementsHostDevCaps
Thanks! I extended the 'devices' section as follows: <hostdev mode='capabilities' type='misc'> <source> <char>/dev/net/tun</char> </source> </hostdev> ... because even though /dev/net/tun is used for networking, it appears as a character device. (Btw: The documentation says in the hostdev section: ''For block/character device passthrough mode is always "capabilities" and type is "block" for a block device, "char" for a character device and "net" for a host network interface.'' When I specify type='char', I get an error from virsh.) With this XML, I can define the container. But upon start, I get the following error message: Fehler: internal error guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=f3602503-9603-24aa-7dd8-fccc830a802b LIBVIRT_LXC_UUID=f3602503-9603-24aa-7dd8-fccc830a802b LIBVIRT_LXC_NAME=depot /sbin/init 2013-07-08 21:36:50.735+0000: 1: info : libvirt version: 1.0.2 2013-07-08 21:36:50.735+0000: 1: error : lxcContainerSetupHostdevCapsMisc:1490 : Unable to create device /dev/net/tun: No such file or directory 2013-07-08 21:36:50.744+0000: 19537: info : libvirt version: 1.0.2 2013-07-08 21:36:50.744+0000: 19537: error : virCommandWait:2287 : internal error Child process (ip link set veth6 netns 19538) unexpected exit status 2: RTNETLINK answers: No such process 2013-07-08 21:36:50.786+0000: 19537: error : virCommandWait:2287 : internal error Child process (ip link del veth4) unexpected exit status 1: Cannot find device "veth4" On the host, /dev/net/tun exists as character device: root@main:~# ls -la /dev/net/tun crw-rw-rwT 1 root root 10, 200 Jul 8 23:45 /dev/net/tun What am I doing wrong ...? Thanks Thomas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlHbNGwACgkQiMyIQtYO79zOwgCdFVzn0JopHK+ZY2ZshgZnuz6L Yx8An3BL/2sfLTFSs39yNkB0FXzq9K/4 =xRtO -----END PGP SIGNATURE-----
On 07/09/2013 05:51 AM, Thomas Karcher wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi Daniel,
On 07/08/2013 11:41 AM, Daniel P. Berrange wrote:
the symptom my libvirt LXC container suffers from is: root@depot:/dev/net# ls -la total 0 drwxr-xr-x 2 root root 40 Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 .. root@depot:/dev/net# mknod tun c 10 200 mknod: `tun': Operation not permitted Allowing the container direct access to the hosts' /dev would be a security flaw, so libvirt sets up a private /dev for the container. Allowing the container to use mknod would also be insecure, so we blocking mknod using both cgroups device ACL, and also droping the CAP_MKNOD capability. http://libvirt.org/drvlxc.html#devnodes
Good to know.
Any device that the container is authorized to access per the XML configuration, will be pre-created in the container's /dev. To explicitly allow /dev/net/tun you need to tell libvirt about it. http://libvirt.org/formatdomain.html#elementsHostDevCaps
Thanks!
I extended the 'devices' section as follows:
<hostdev mode='capabilities' type='misc'> <source> <char>/dev/net/tun</char> </source> </hostdev>
... because even though /dev/net/tun is used for networking, it appears as a character device. (Btw: The documentation says in the hostdev section: ''For block/character device passthrough mode is always "capabilities" and type is "block" for a block device, "char" for a character device and "net" for a host network interface.'' When I specify type='char', I get an error from virsh.)
With this XML, I can define the container. But upon start, I get the following error message:
Fehler: internal error guest failed to start: PATH=/bin:/sbin TERM=linux container=lxc-libvirt container_uuid=f3602503-9603-24aa-7dd8-fccc830a802b LIBVIRT_LXC_UUID=f3602503-9603-24aa-7dd8-fccc830a802b LIBVIRT_LXC_NAME=depot /sbin/init 2013-07-08 21:36:50.735+0000: 1: info : libvirt version: 1.0.2 2013-07-08 21:36:50.735+0000: 1: error : lxcContainerSetupHostdevCapsMisc:1490 : Unable to create device /dev/net/tun: No such file or directory 2013-07-08 21:36:50.744+0000: 19537: info : libvirt version: 1.0.2 2013-07-08 21:36:50.744+0000: 19537: error : virCommandWait:2287 : internal error Child process (ip link set veth6 netns 19538) unexpected exit status 2: RTNETLINK answers: No such process
2013-07-08 21:36:50.786+0000: 19537: error : virCommandWait:2287 : internal error Child process (ip link del veth4) unexpected exit status 1: Cannot find device "veth4"
On the host, /dev/net/tun exists as character device:
root@main:~# ls -la /dev/net/tun crw-rw-rwT 1 root root 10, 200 Jul 8 23:45 /dev/net/tun
What am I doing wrong ...?
You are right, it should be char device. libvirt lxc should create "net" directory atomically for the tun device.
This helper function is used to create parent directroy for the hostdev which will be added to the container. if the parent directory of this hostdev doesn't exist, the mknod of the hostdev will fail. Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com> --- src/lxc/lxc_container.c | 17 +++++++++++++++++ src/lxc/lxc_container.h | 2 ++ 2 files changed, 19 insertions(+) diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c index c8420db..b954107 100644 --- a/src/lxc/lxc_container.c +++ b/src/lxc/lxc_container.c @@ -1544,6 +1544,23 @@ cleanup: } +int lxcContainerSetupHostdevCapsMakePath(char *dev) +{ + int ret = 0; + char *dir = NULL; + + if ((dir = strrchr(dev, '/'))) { + *dir = '\0'; + if ((virFileMakePath(dev) < 0) && (errno != EEXIST)) + ret = -1; + + *dir = '/'; + } + + return ret; +} + + static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr vmDef ATTRIBUTE_UNUSED, virDomainHostdevDefPtr def ATTRIBUTE_UNUSED, virSecurityManagerPtr securityDriver ATTRIBUTE_UNUSED) diff --git a/src/lxc/lxc_container.h b/src/lxc/lxc_container.h index ada72f7..f168703 100644 --- a/src/lxc/lxc_container.h +++ b/src/lxc/lxc_container.h @@ -63,6 +63,8 @@ int lxcContainerStart(virDomainDefPtr def, int lxcContainerAvailable(int features); +int lxcContainerSetupHostdevCapsMakePath(char *dev); + virArch lxcContainerGetAlt32bitArch(virArch arch); #endif /* LXC_CONTAINER_H */ -- 1.8.3.1
Create parent directroy for hostdev atomically when we start a lxc domain or attach a hostdev to a lxc domain. Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com> --- src/lxc/lxc_container.c | 42 ++++++++++++++++++++++++++++-------------- src/lxc/lxc_driver.c | 14 ++++++++++++++ 2 files changed, 42 insertions(+), 14 deletions(-) diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c index b954107..a204789 100644 --- a/src/lxc/lxc_container.c +++ b/src/lxc/lxc_container.c @@ -1569,14 +1569,15 @@ static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr vmDef ATTRIBUTE_U int ret = -1; struct stat sb; mode_t mode; + char *dev = def->source.caps.u.storage.block; - if (def->source.caps.u.storage.block == NULL) { + if (dev == NULL) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("Missing storage host block path")); goto cleanup; } - if (virAsprintf(&src, "/.oldroot/%s", def->source.caps.u.storage.block) < 0) { + if (virAsprintf(&src, "/.oldroot/%s", dev) < 0) { virReportOOMError(); goto cleanup; } @@ -1591,19 +1592,25 @@ static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr vmDef ATTRIBUTE_U if (!S_ISBLK(sb.st_mode)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Storage source %s must be a block device"), - def->source.caps.u.storage.block); + dev); + goto cleanup; + } + + if (lxcContainerSetupHostdevCapsMakePath(dev) < 0) { + virReportError(errno, + _("Failed to create directory for device %s"), + dev); goto cleanup; } mode = 0700 | S_IFBLK; - VIR_DEBUG("Creating dev %s (%d,%d)", - def->source.caps.u.storage.block, + VIR_DEBUG("Creating dev %s (%d,%d)", dev, major(sb.st_rdev), minor(sb.st_rdev)); - if (mknod(def->source.caps.u.storage.block, mode, sb.st_rdev) < 0) { + if (mknod(dev, mode, sb.st_rdev) < 0) { virReportSystemError(errno, _("Unable to create device %s"), - def->source.caps.u.storage.block); + dev); goto cleanup; } @@ -1626,14 +1633,15 @@ static int lxcContainerSetupHostdevCapsMisc(virDomainDefPtr vmDef ATTRIBUTE_UNUS int ret = -1; struct stat sb; mode_t mode; + char *dev = def->source.caps.u.misc.chardev; - if (def->source.caps.u.misc.chardev == NULL) { + if (dev == NULL) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s", _("Missing storage host block path")); goto cleanup; } - if (virAsprintf(&src, "/.oldroot/%s", def->source.caps.u.misc.chardev) < 0) { + if (virAsprintf(&src, "/.oldroot/%s", dev) < 0) { virReportOOMError(); goto cleanup; } @@ -1648,19 +1656,25 @@ static int lxcContainerSetupHostdevCapsMisc(virDomainDefPtr vmDef ATTRIBUTE_UNUS if (!S_ISCHR(sb.st_mode)) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("Storage source %s must be a character device"), - def->source.caps.u.misc.chardev); + dev); + goto cleanup; + } + + if (lxcContainerSetupHostdevCapsMakePath(dev) < 0) { + virReportError(errno, + _("Failed to create directory for device %s"), + dev); goto cleanup; } mode = 0700 | S_IFCHR; - VIR_DEBUG("Creating dev %s (%d,%d)", - def->source.caps.u.misc.chardev, + VIR_DEBUG("Creating dev %s (%d,%d)", dev, major(sb.st_rdev), minor(sb.st_rdev)); - if (mknod(def->source.caps.u.misc.chardev, mode, sb.st_rdev) < 0) { + if (mknod(dev, mode, sb.st_rdev) < 0) { virReportSystemError(errno, _("Unable to create device %s"), - def->source.caps.u.misc.chardev); + dev); goto cleanup; } diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c index 1a6d086..2661c1b 100644 --- a/src/lxc/lxc_driver.c +++ b/src/lxc/lxc_driver.c @@ -3616,6 +3616,13 @@ lxcDomainAttachDeviceHostdevStorageLive(virLXCDriverPtr driver, goto cleanup; } + if (lxcContainerSetupHostdevCapsMakePath(dst) < 0) { + virReportSystemError(errno, + _("Unable to create directroy for device %s"), + dst); + goto cleanup; + } + mode = 0700 | S_IFBLK; VIR_DEBUG("Creating dev %s (%d,%d)", @@ -3720,6 +3727,13 @@ lxcDomainAttachDeviceHostdevMiscLive(virLXCDriverPtr driver, goto cleanup; } + if (lxcContainerSetupHostdevCapsMakePath(dst) < 0) { + virReportSystemError(errno, + _("Unable to create directroy for device %s"), + dst); + goto cleanup; + } + mode = 0700 | S_IFCHR; VIR_DEBUG("Creating dev %s (%d,%d)", -- 1.8.3.1
On Tue, Jul 09, 2013 at 11:20:36AM +0800, Gao feng wrote:
Create parent directroy for hostdev atomically when we start a lxc domain or attach a hostdev to a lxc domain.
Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com> --- src/lxc/lxc_container.c | 42 ++++++++++++++++++++++++++++-------------- src/lxc/lxc_driver.c | 14 ++++++++++++++ 2 files changed, 42 insertions(+), 14 deletions(-)
ACK Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Guys, thank you for your help and your effort! (Didn't try the solution yet, but I will.) Thomas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) iEYEARECAAYFAlHepkcACgkQiMyIQtYO79zSLwCfdVfFwU5Tz5FYxmoYaKXTHOHT uAsAoM54vncy6HjftWl5mQMTOe+Rpw0Z =gems -----END PGP SIGNATURE-----
On Tue, Jul 09, 2013 at 11:20:35AM +0800, Gao feng wrote:
This helper function is used to create parent directroy for the hostdev which will be added to the container. if the parent directory of this hostdev doesn't exist, the mknod of the hostdev will fail.
Signed-off-by: Gao feng <gaofeng@cn.fujitsu.com> --- src/lxc/lxc_container.c | 17 +++++++++++++++++ src/lxc/lxc_container.h | 2 ++ 2 files changed, 19 insertions(+)
diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c index c8420db..b954107 100644 --- a/src/lxc/lxc_container.c +++ b/src/lxc/lxc_container.c @@ -1544,6 +1544,23 @@ cleanup: }
+int lxcContainerSetupHostdevCapsMakePath(char *dev) +{ + int ret = 0; + char *dir = NULL; + + if ((dir = strrchr(dev, '/'))) { + *dir = '\0';
Modifying arguments that are passed into a function is bad practice, so I'm changing this to strdup the arg thus: +int lxcContainerSetupHostdevCapsMakePath(const char *dev) +{ + int ret = -1; + char *dir, *tmp; + + if (VIR_STRDUP(dir, dev) < 0) + return -1; + + if ((tmp = strrchr(dir, '/'))) { + *tmp = '\0'; + if (virFileMakePath(dir) < 0) { + virReportSystemError(errno, + _("Failed to create directory for '%s' dev '%s'"), + dir, dev); + goto cleanup; + } + } + + ret = 0; + +cleanup: + VIR_FREE(dir); + return ret; +} Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (3)
-
Daniel P. Berrange -
Gao feng -
Thomas Karcher