3:59 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi lxc folks,
the symptom my libvirt LXC container suffers from is:
root@depot:/dev/net# ls -la
total 0
drwxr-xr-x 2 root root 40 Jun 29 16:26 .
drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
root@depot:/dev/net# mknod tun c 10 200
mknod: `tun': Operation not permitted
The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that
was formerly installed from precise and then upgraded. The guest is
Ubuntu precise; however, I see the same symptom in another raring
container on the same host.
What I tried to resolve this:
1) On the host, I echoed various stuff to the cgroup device files:
cd /sys/fs/cgroup/devices/libvirt/lxc
echo "c 10:200 rwm" > devices.allow
echo "c 10:200 rwm" > depot/devices.allow
echo a > depot/devices.allow
... and I see the successful results in depot/devices.list, but no
success.
2) I inserted a line "/dev/net/tun rwk," into
/etc/apparmor.d/abstractions/lxc/container-base - no change. (I know,
it seems kind of pointless - because it's about permissions to a
device, not a path. You may deduce my desperation from this ...
SELinux is not active. Mounts on /dev look normal to me:
devfs on /dev type tmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
devpts on /dev/ptmx type devpts
(rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
Now comes the weird part: Once I umount /dev inside the container, the
"hidden" /dev appears which contains a usable /dev/net/tun. So the
mknod problem is probably due to the dropped capabilities - but
why/how mounts the container a more restricted /dev on top of the
prepared one ...?
Thanks & regards
Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlHYdysACgkQiMyIQtYO79yUMQCfUrbaAWcZsqYkopRL6F2evWdq
XnMAoOq6/wopgBGZniWDGnIiYBEsFX61
=BIr9
-----END PGP SIGNATURE-----
5:41 p.m.
On Sat, Jul 06, 2013 at 09:59:39PM +0200, Thomas Karcher wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi lxc folks,
the symptom my libvirt LXC container suffers from is:
root@depot:/dev/net# ls -la
total 0
drwxr-xr-x 2 root root 40 Jun 29 16:26 .
drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
root@depot:/dev/net# mknod tun c 10 200
mknod: `tun': Operation not permitted
The host is an up-to-date AMD64 Ubuntu raring on 3.8.0-25-generic that
was formerly installed from precise and then upgraded. The guest is
Ubuntu precise; however, I see the same symptom in another raring
container on the same host.
What I tried to resolve this:
1) On the host, I echoed various stuff to the cgroup device files:
cd /sys/fs/cgroup/devices/libvirt/lxc
echo "c 10:200 rwm" > devices.allow
echo "c 10:200 rwm" > depot/devices.allow
echo a > depot/devices.allow
... and I see the successful results in depot/devices.list, but no
success.
2) I inserted a line "/dev/net/tun rwk," into
/etc/apparmor.d/abstractions/lxc/container-base - no change. (I know,
it seems kind of pointless - because it's about permissions to a
device, not a path. You may deduce my desperation from this ...
SELinux is not active. Mounts on /dev look normal to me:
devfs on /dev type tmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
devpts on /dev/ptmx type devpts
(rw,nosuid,relatime,gid=5,mode=620,ptmxmode=666)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,relatime,devices)
Now comes the weird part: Once I umount /dev inside the container, the
"hidden" /dev appears which contains a usable /dev/net/tun. So the
mknod problem is probably due to the dropped capabilities - but
why/how mounts the container a more restricted /dev on top of the
prepared one ...?
Allowing the container direct access to the hosts' /dev would be
a security flaw, so libvirt sets up a private /dev for the container.
Allowing the container to use mknod would also be insecure, so we
blocking mknod using both cgroups device ACL, and also droping the
CAP_MKNOD capability.
http://libvirt.org/drvlxc.html#devnodes
Any device that the container is authorized to access per the XML
configuration, will be pre-created in the container's /dev. To
explicitly allow /dev/net/tun you need to tell libvirt about it.
http://libvirt.org/formatdomain.html#elementsHostDevCaps
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
5:51 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Daniel,
On 07/08/2013 11:41 AM, Daniel P. Berrange wrote:
> the symptom my libvirt LXC container suffers from is:
> root@depot:/dev/net# ls -la total 0 drwxr-xr-x 2 root root 40
> Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
> root@depot:/dev/net# mknod tun c 10 200 mknod: `tun': Operation
> not permitted
Allowing the container direct access to the hosts' /dev would be a
security flaw, so libvirt sets up a private /dev for the
container. Allowing the container to use mknod would also be
insecure, so we blocking mknod using both cgroups device ACL, and
also droping the CAP_MKNOD capability.
http://libvirt.org/drvlxc.html#devnodes
Good to know.
Any device that the container is authorized to access per the XML
configuration, will be pre-created in the container's /dev. To
explicitly allow /dev/net/tun you need to tell libvirt about it.
http://libvirt.org/formatdomain.html#elementsHostDevCaps
Thanks!
I extended the 'devices' section as follows:
<hostdev mode='capabilities' type='misc'>
<source>
<char>/dev/net/tun</char>
</source>
</hostdev>
... because even though /dev/net/tun is used for networking, it
appears as a character device. (Btw: The documentation says in the
hostdev section: ''For block/character device passthrough mode is
always "capabilities" and type is "block" for a block device,
"char"
for a character device and "net" for a host network interface.'' When
I specify type='char', I get an error from virsh.)
With this XML, I can define the container. But upon start, I get the
following error message:
Fehler: internal error guest failed to start: PATH=/bin:/sbin
TERM=linux container=lxc-libvirt
container_uuid=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_UUID=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_NAME=depot /sbin/init
2013-07-08 21:36:50.735+0000: 1: info : libvirt version: 1.0.2
2013-07-08 21:36:50.735+0000: 1: error :
lxcContainerSetupHostdevCapsMisc:1490 : Unable to create device
/dev/net/tun: No such file or directory
2013-07-08 21:36:50.744+0000: 19537: info : libvirt version: 1.0.2
2013-07-08 21:36:50.744+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link set veth6 netns 19538)
unexpected exit status 2: RTNETLINK answers: No such process
2013-07-08 21:36:50.786+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link del veth4) unexpected exit
status 1: Cannot find device "veth4"
On the host, /dev/net/tun exists as character device:
root@main:~# ls -la /dev/net/tun
crw-rw-rwT 1 root root 10, 200 Jul 8 23:45 /dev/net/tun
What am I doing wrong ...?
Thanks
Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iEYEARECAAYFAlHbNGwACgkQiMyIQtYO79zOwgCdFVzn0JopHK+ZY2ZshgZnuz6L
Yx8An3BL/2sfLTFSs39yNkB0FXzq9K/4
=xRtO
-----END PGP SIGNATURE-----
10:05 a.m.
On 07/09/2013 05:51 AM, Thomas Karcher wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi Daniel,
On 07/08/2013 11:41 AM, Daniel P. Berrange wrote:
>> the symptom my libvirt LXC container suffers from is:
>> root@depot:/dev/net# ls -la total 0 drwxr-xr-x 2 root root 40
>> Jun 29 16:26 . drwxr-xr-x 5 root root 480 Jun 29 16:26 ..
>> root@depot:/dev/net# mknod tun c 10 200 mknod: `tun': Operation
>> not permitted
> Allowing the container direct access to the hosts' /dev would be a
> security flaw, so libvirt sets up a private /dev for the
> container. Allowing the container to use mknod would also be
> insecure, so we blocking mknod using both cgroups device ACL, and
> also droping the CAP_MKNOD capability.
> http://libvirt.org/drvlxc.html#devnodes
Good to know.
> Any device that the container is authorized to access per the XML
> configuration, will be pre-created in the container's /dev. To
> explicitly allow /dev/net/tun you need to tell libvirt about it.
> http://libvirt.org/formatdomain.html#elementsHostDevCaps
Thanks!
I extended the 'devices' section as follows:
<hostdev mode='capabilities' type='misc'>
<source>
<char>/dev/net/tun</char>
</source>
</hostdev>
... because even though /dev/net/tun is used for networking, it
appears as a character device. (Btw: The documentation says in the
hostdev section: ''For block/character device passthrough mode is
always "capabilities" and type is "block" for a block device,
"char"
for a character device and "net" for a host network interface.'' When
I specify type='char', I get an error from virsh.)
With this XML, I can define the container. But upon start, I get the
following error message:
Fehler: internal error guest failed to start: PATH=/bin:/sbin
TERM=linux container=lxc-libvirt
container_uuid=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_UUID=f3602503-9603-24aa-7dd8-fccc830a802b
LIBVIRT_LXC_NAME=depot /sbin/init
2013-07-08 21:36:50.735+0000: 1: info : libvirt version: 1.0.2
2013-07-08 21:36:50.735+0000: 1: error :
lxcContainerSetupHostdevCapsMisc:1490 : Unable to create device
/dev/net/tun: No such file or directory
2013-07-08 21:36:50.744+0000: 19537: info : libvirt version: 1.0.2
2013-07-08 21:36:50.744+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link set veth6 netns 19538)
unexpected exit status 2: RTNETLINK answers: No such process
2013-07-08 21:36:50.786+0000: 19537: error : virCommandWait:2287 :
internal error Child process (ip link del veth4) unexpected exit
status 1: Cannot find device "veth4"
On the host, /dev/net/tun exists as character device:
root@main:~# ls -la /dev/net/tun
crw-rw-rwT 1 root root 10, 200 Jul 8 23:45 /dev/net/tun
What am I doing wrong ...?
You are right, it should be char device.
libvirt lxc should create "net" directory atomically for the tun device.
11:20 a.m.
New subject: [libvirt-users] [PATCH 1/2] LXC: hostdev: introduce lxcContainerSetupHostdevCapsMakePath
This helper function is used to create parent directroy for
the hostdev which will be added to the container. if the
parent directory of this hostdev doesn't exist, the mknod of
the hostdev will fail.
Signed-off-by: Gao feng <gaofeng(a)cn.fujitsu.com>
---
src/lxc/lxc_container.c | 17 +++++++++++++++++
src/lxc/lxc_container.h | 2 ++
2 files changed, 19 insertions(+)
diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c
index c8420db..b954107 100644
--- a/src/lxc/lxc_container.c
+++ b/src/lxc/lxc_container.c
@@ -1544,6 +1544,23 @@ cleanup:
}
+int lxcContainerSetupHostdevCapsMakePath(char *dev)
+{
+ int ret = 0;
+ char *dir = NULL;
+
+ if ((dir = strrchr(dev, '/'))) {
+ *dir = '\0';
+ if ((virFileMakePath(dev) < 0) && (errno != EEXIST))
+ ret = -1;
+
+ *dir = '/';
+ }
+
+ return ret;
+}
+
+
static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr vmDef ATTRIBUTE_UNUSED,
virDomainHostdevDefPtr def
ATTRIBUTE_UNUSED,
virSecurityManagerPtr securityDriver
ATTRIBUTE_UNUSED)
diff --git a/src/lxc/lxc_container.h b/src/lxc/lxc_container.h
index ada72f7..f168703 100644
--- a/src/lxc/lxc_container.h
+++ b/src/lxc/lxc_container.h
@@ -63,6 +63,8 @@ int lxcContainerStart(virDomainDefPtr def,
int lxcContainerAvailable(int features);
+int lxcContainerSetupHostdevCapsMakePath(char *dev);
+
virArch lxcContainerGetAlt32bitArch(virArch arch);
#endif /* LXC_CONTAINER_H */
--
1.8.3.1
11:20 a.m.
New subject: [libvirt-users] [PATCH 2/2] LXC: hostdev: parent directroy for hostdev atomically
Create parent directroy for hostdev atomically when we
start a lxc domain or attach a hostdev to a lxc domain.
Signed-off-by: Gao feng <gaofeng(a)cn.fujitsu.com>
---
src/lxc/lxc_container.c | 42 ++++++++++++++++++++++++++++--------------
src/lxc/lxc_driver.c | 14 ++++++++++++++
2 files changed, 42 insertions(+), 14 deletions(-)
diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c
index b954107..a204789 100644
--- a/src/lxc/lxc_container.c
+++ b/src/lxc/lxc_container.c
@@ -1569,14 +1569,15 @@ static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr
vmDef ATTRIBUTE_U
int ret = -1;
struct stat sb;
mode_t mode;
+ char *dev = def->source.caps.u.storage.block;
- if (def->source.caps.u.storage.block == NULL) {
+ if (dev == NULL) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
_("Missing storage host block path"));
goto cleanup;
}
- if (virAsprintf(&src, "/.oldroot/%s",
def->source.caps.u.storage.block) < 0) {
+ if (virAsprintf(&src, "/.oldroot/%s", dev) < 0) {
virReportOOMError();
goto cleanup;
}
@@ -1591,19 +1592,25 @@ static int lxcContainerSetupHostdevCapsStorage(virDomainDefPtr
vmDef ATTRIBUTE_U
if (!S_ISBLK(sb.st_mode)) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
_("Storage source %s must be a block device"),
- def->source.caps.u.storage.block);
+ dev);
+ goto cleanup;
+ }
+
+ if (lxcContainerSetupHostdevCapsMakePath(dev) < 0) {
+ virReportError(errno,
+ _("Failed to create directory for device %s"),
+ dev);
goto cleanup;
}
mode = 0700 | S_IFBLK;
- VIR_DEBUG("Creating dev %s (%d,%d)",
- def->source.caps.u.storage.block,
+ VIR_DEBUG("Creating dev %s (%d,%d)", dev,
major(sb.st_rdev), minor(sb.st_rdev));
- if (mknod(def->source.caps.u.storage.block, mode, sb.st_rdev) < 0) {
+ if (mknod(dev, mode, sb.st_rdev) < 0) {
virReportSystemError(errno,
_("Unable to create device %s"),
- def->source.caps.u.storage.block);
+ dev);
goto cleanup;
}
@@ -1626,14 +1633,15 @@ static int lxcContainerSetupHostdevCapsMisc(virDomainDefPtr vmDef
ATTRIBUTE_UNUS
int ret = -1;
struct stat sb;
mode_t mode;
+ char *dev = def->source.caps.u.misc.chardev;
- if (def->source.caps.u.misc.chardev == NULL) {
+ if (dev == NULL) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED, "%s",
_("Missing storage host block path"));
goto cleanup;
}
- if (virAsprintf(&src, "/.oldroot/%s",
def->source.caps.u.misc.chardev) < 0) {
+ if (virAsprintf(&src, "/.oldroot/%s", dev) < 0) {
virReportOOMError();
goto cleanup;
}
@@ -1648,19 +1656,25 @@ static int lxcContainerSetupHostdevCapsMisc(virDomainDefPtr vmDef
ATTRIBUTE_UNUS
if (!S_ISCHR(sb.st_mode)) {
virReportError(VIR_ERR_CONFIG_UNSUPPORTED,
_("Storage source %s must be a character device"),
- def->source.caps.u.misc.chardev);
+ dev);
+ goto cleanup;
+ }
+
+ if (lxcContainerSetupHostdevCapsMakePath(dev) < 0) {
+ virReportError(errno,
+ _("Failed to create directory for device %s"),
+ dev);
goto cleanup;
}
mode = 0700 | S_IFCHR;
- VIR_DEBUG("Creating dev %s (%d,%d)",
- def->source.caps.u.misc.chardev,
+ VIR_DEBUG("Creating dev %s (%d,%d)", dev,
major(sb.st_rdev), minor(sb.st_rdev));
- if (mknod(def->source.caps.u.misc.chardev, mode, sb.st_rdev) < 0) {
+ if (mknod(dev, mode, sb.st_rdev) < 0) {
virReportSystemError(errno,
_("Unable to create device %s"),
- def->source.caps.u.misc.chardev);
+ dev);
goto cleanup;
}
diff --git a/src/lxc/lxc_driver.c b/src/lxc/lxc_driver.c
index 1a6d086..2661c1b 100644
--- a/src/lxc/lxc_driver.c
+++ b/src/lxc/lxc_driver.c
@@ -3616,6 +3616,13 @@ lxcDomainAttachDeviceHostdevStorageLive(virLXCDriverPtr driver,
goto cleanup;
}
+ if (lxcContainerSetupHostdevCapsMakePath(dst) < 0) {
+ virReportSystemError(errno,
+ _("Unable to create directroy for device %s"),
+ dst);
+ goto cleanup;
+ }
+
mode = 0700 | S_IFBLK;
VIR_DEBUG("Creating dev %s (%d,%d)",
@@ -3720,6 +3727,13 @@ lxcDomainAttachDeviceHostdevMiscLive(virLXCDriverPtr driver,
goto cleanup;
}
+ if (lxcContainerSetupHostdevCapsMakePath(dst) < 0) {
+ virReportSystemError(errno,
+ _("Unable to create directroy for device %s"),
+ dst);
+ goto cleanup;
+ }
+
mode = 0700 | S_IFCHR;
VIR_DEBUG("Creating dev %s (%d,%d)",
--
1.8.3.1
6:19 p.m.
New subject: [libvirt-users] [PATCH 2/2] LXC: hostdev: parent directroy for hostdev atomically
On Tue, Jul 09, 2013 at 11:20:36AM +0800, Gao feng wrote:
Create parent directroy for hostdev atomically when we
start a lxc domain or attach a hostdev to a lxc domain.
Signed-off-by: Gao feng <gaofeng(a)cn.fujitsu.com>
---
src/lxc/lxc_container.c | 42 ++++++++++++++++++++++++++++--------------
src/lxc/lxc_driver.c | 14 ++++++++++++++
2 files changed, 42 insertions(+), 14 deletions(-)
ACK
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
8:34 p.m.
New subject: [libvirt-users] [PATCH 2/2] LXC: hostdev: parent directroy for hostdev atomically
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Guys,
thank you for your help and your effort! (Didn't try the solution yet,
but I will.)
Thomas
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (GNU/Linux)
iEYEARECAAYFAlHepkcACgkQiMyIQtYO79zSLwCfdVfFwU5Tz5FYxmoYaKXTHOHT
uAsAoM54vncy6HjftWl5mQMTOe+Rpw0Z
=gems
-----END PGP SIGNATURE-----
6:19 p.m.
New subject: [libvirt-users] [PATCH 1/2] LXC: hostdev: introduce lxcContainerSetupHostdevCapsMakePath
On Tue, Jul 09, 2013 at 11:20:35AM +0800, Gao feng wrote:
This helper function is used to create parent directroy for
the hostdev which will be added to the container. if the
parent directory of this hostdev doesn't exist, the mknod of
the hostdev will fail.
Signed-off-by: Gao feng <gaofeng(a)cn.fujitsu.com>
---
src/lxc/lxc_container.c | 17 +++++++++++++++++
src/lxc/lxc_container.h | 2 ++
2 files changed, 19 insertions(+)
diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c
index c8420db..b954107 100644
--- a/src/lxc/lxc_container.c
+++ b/src/lxc/lxc_container.c
@@ -1544,6 +1544,23 @@ cleanup:
}
+int lxcContainerSetupHostdevCapsMakePath(char *dev)
+{
+ int ret = 0;
+ char *dir = NULL;
+
+ if ((dir = strrchr(dev, '/'))) {
+ *dir = '\0';
Modifying arguments that are passed into a function is bad
practice, so I'm changing this to strdup the arg thus:
+int lxcContainerSetupHostdevCapsMakePath(const char *dev)
+{
+ int ret = -1;
+ char *dir, *tmp;
+
+ if (VIR_STRDUP(dir, dev) < 0)
+ return -1;
+
+ if ((tmp = strrchr(dir, '/'))) {
+ *tmp = '\0';
+ if (virFileMakePath(dir) < 0) {
+ virReportSystemError(errno,
+ _("Failed to create directory for '%s' dev
'%s'"),
+ dir, dev);
+ goto cleanup;
+ }
+ }
+
+ ret = 0;
+
+cleanup:
+ VIR_FREE(dir);
+ return ret;
+}
Daniel
--
|: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org -o- http://virt-manager.org :|
|: http://autobuild.org -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
4076
days inactive
4081
days old
8 comments
3 participants
participants (3)
-
Daniel P. Berrange -
Gao feng -
Thomas Karcher