[libvirt-users] LXC + USB passthrough = Operation not permitted
Hi! First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D I'm running ubuntu 13.10 with LXC 1.0.1 and tried both libvirt 1.1.1 and 1.2.2 (backported from ubuntu-trusty), but with either version of libvirt am getting issues as soon as I try to get access to USB devices inside the container. Relevant versions of stuff: $ dpkg -l | grep -i lxc ii liblxc1 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools (library) ii lxc 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools ii lxc-templates 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools (templates) ii python3-lxc 1.0.1-0ubuntu1~ubuntu13.10.1~ppa1 i386 Linux Containers userspace tools (Python 3.x bindings) $ dpkg -l | grep libvirt ii libvirt-bin 1.2.2-0ubuntu12 i386 programs for the libvirt library ii libvirt0 1.2.2-0ubuntu12 i386 library for interfacing with different virtualization systems ii python-libvirt 1.2.2-0ubuntu1 i386 libvirt Python bindings Here's my entire domain definition: <domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain> Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device connected to the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log: 2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted I get the same output above when I use libvirt 1.1.1. Tracing down through the libvirt code it looks like it fails when calling `mknod`. It's clear that libvirt converts the product/vendor ids I specified in the domain definition file to the appropriate USB bus (/dev/bus/usb/002/003, verified by comparing against output of `usb-devices`). In a parallel effort, a colleague of mine set up a vanilla LXContainer on a new install of the same version of ubuntu, and successfully was able to communicate with the Android device from a container. The only configuration tweaks he made was make sure to whitelist the cgroup with the appropriate major/minor device number (associated with the Android device) inside the LXC configuration file. From there the container was able to appropriately create the device file as you attach/detach the Android device. Hopefully someone can shed some light, and appreciate your patience with me as I am learning a lot of this stuff as I go :) Thanks in advance for any help! Fil
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
Hi!
First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D Here's my entire domain definition:
<domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain>
Your config looks fine here.
Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device connected to the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log:
2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
Do you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode: $ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626) And still get issues. From libvirtd.log: 2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted 2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices Thanks in advance for any help, Daniel! Cheers, Fil On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
Hi!
First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D Here's my entire domain definition:
<domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain>
Your config looks fine here.
Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device connected
to
the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log:
2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
Do you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/:| |: http://libvirt.org -o- http://virt-manager.org:| |: http://autobuild.org -o- http://search.cpan.org/~danberr/:| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc:|
To follow up on this a little bit, tail'ing kern.log while trying to get our little container up doesn't yield anything with apparmor complaining, so, unless I'm looking in the wrong spots for apparmor logs (which I don't think so, as I see other apparmor-related log entries in kern.log), I am not entirely sure this is an apparmor issue at this point. On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil@saucelabs.com> wrote:
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode:
$ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626)
And still get issues. From libvirtd.log:
2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices
Thanks in advance for any help, Daniel!
Cheers, Fil
On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
Hi!
First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D Here's my entire domain definition:
<domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain>
Your config looks fine here.
Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device connected
to
the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log:
2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
Do you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Further followups! We are correlating DEBUG-level output from libvirt with the libvirt 1.2.2 code to try to figure out what libvirt is doing under the hood. Even though we have the log level set to 1 (info) in our libvirtd.conf, we are not seeing the VIR_DEBUG() [1] statements being printed out. There are tons of other presumably-debug lines of output showing up in our log. We are sort of expecting to see the output from [1] in our logs somewhere, given the type of usb mounting we are trying to do? [1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c0... On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <fil@saucelabs.com> wrote:
To follow up on this a little bit, tail'ing kern.log while trying to get our little container up doesn't yield anything with apparmor complaining, so, unless I'm looking in the wrong spots for apparmor logs (which I don't think so, as I see other apparmor-related log entries in kern.log), I am not entirely sure this is an apparmor issue at this point.
On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil@saucelabs.com> wrote:
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode:
$ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626)
And still get issues. From libvirtd.log:
2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices
Thanks in advance for any help, Daniel!
Cheers, Fil
On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com>wrote:
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
Hi!
First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D Here's my entire domain definition:
<domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain>
Your config looks fine here.
Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device
connected to
the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log:
2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
Do you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Ah, nevermind, think my last post may not have been entirely correct. We've spent some more time correlating the log output from our failed LXC startup via libvirt [1] with libvirt code from the 1.2.2 tag. Interestingly, the error we get back is different, and digging through the multitudes of logs, nothing comes up about 'operating not permitted' regarding the USB bus that I saw earlier. If we turn off the verbose logging, we get the original error back. The more-verbose error is a bit more cryptic: libvirt LXC Driver error: internal error: guest failed to start: 2014- Anyways, here's a rough overview of the logs: 1. it starts to check/detect for cgroup stuff [2] 2. starts to 'prepare host devices' and logs out that the appropriate usb address is initialized (002/003) [3] 3. sets the appropriate usb bus to 'active host devs' [4] 4. seems to then switch to doing network-related things, and stop talking about any hostdev/usb stuff [5] 5. eventually invokes `libvirt_lxc` [6] 6. does the cgroup song-n-dance again for some reason? [7]. not sure if this is critical or not. 7. eventually some kind of socket read wire error comes up [8] 8. and this leads to a 'guest failed to start error' [9], which it probably truncates to return the error i mentioned above. What is also confusing is, looking through the libvirt lxc code, shouldn't we be seeing a series of "Allowing any <blah>" log debug outputs in our logs? This is part of the virLXCCgroupSetupDeviceACL part of the lxc_cgroup.c file [10]. I imagine that we'd eventually want the virCgroupAllowDeviceMajor method [11] invoked, no? I realize that you are the author of most of this code, Daniel, so appreciate your patience with me and hope you are willing to continue trying to get to the bottom of this with us! Thanks a lot in advance! Cheers, Fil [1] https://gist.github.com/filmaj/207ab3c09f5b881d83b1 [2] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L... [3] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L... [4] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296 [5] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302 [6] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685 [7] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L... [8] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075 [9] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090 [10] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955... [11] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955... On Wed, Apr 16, 2014 at 5:11 PM, Filip Maj <fil@saucelabs.com> wrote:
Further followups!
We are correlating DEBUG-level output from libvirt with the libvirt 1.2.2 code to try to figure out what libvirt is doing under the hood.
Even though we have the log level set to 1 (info) in our libvirtd.conf, we are not seeing the VIR_DEBUG() [1] statements being printed out. There are tons of other presumably-debug lines of output showing up in our log. We are sort of expecting to see the output from [1] in our logs somewhere, given the type of usb mounting we are trying to do?
[1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c0...
On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <fil@saucelabs.com> wrote:
To follow up on this a little bit, tail'ing kern.log while trying to get our little container up doesn't yield anything with apparmor complaining, so, unless I'm looking in the wrong spots for apparmor logs (which I don't think so, as I see other apparmor-related log entries in kern.log), I am not entirely sure this is an apparmor issue at this point.
On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil@saucelabs.com> wrote:
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode:
$ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626)
And still get issues. From libvirtd.log:
2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices
Thanks in advance for any help, Daniel!
Cheers, Fil
On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange <berrange@redhat.com
wrote:
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
Hi!
First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D Here's my entire domain definition:
<domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain>
Your config looks fine here.
Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device
connected to
the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log:
2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
Do you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Me again! Think we've found it. By diving into the LXC logs for the specific container, we found this: 2014-04-17 21:07:06.066+0000: 2861: debug : virCgroupSetValueStr:678 : Set value '/sys/fs/cgroup/devices/machine/oshi32134.libvirt-lxc/devices.allow' to 'c 189:130 rw' Looks like libvirt the permission to 'rw', not 'rmw' [1], so no surprise that when it goes to call mknod [2], it can't do so. We are going to patch this so it gives 'rwm' access in the cgroup whitelist, try to compile and replace libvirt to see if that works out for us. Will report back :) [1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955... [2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=5c... On Thu, Apr 17, 2014 at 11:26 AM, Filip Maj <fil@saucelabs.com> wrote:
Ah, nevermind, think my last post may not have been entirely correct. We've spent some more time correlating the log output from our failed LXC startup via libvirt [1] with libvirt code from the 1.2.2 tag. Interestingly, the error we get back is different, and digging through the multitudes of logs, nothing comes up about 'operating not permitted' regarding the USB bus that I saw earlier. If we turn off the verbose logging, we get the original error back. The more-verbose error is a bit more cryptic:
libvirt LXC Driver error: internal error: guest failed to start: 2014-
Anyways, here's a rough overview of the logs:
1. it starts to check/detect for cgroup stuff [2] 2. starts to 'prepare host devices' and logs out that the appropriate usb address is initialized (002/003) [3] 3. sets the appropriate usb bus to 'active host devs' [4] 4. seems to then switch to doing network-related things, and stop talking about any hostdev/usb stuff [5] 5. eventually invokes `libvirt_lxc` [6] 6. does the cgroup song-n-dance again for some reason? [7]. not sure if this is critical or not. 7. eventually some kind of socket read wire error comes up [8] 8. and this leads to a 'guest failed to start error' [9], which it probably truncates to return the error i mentioned above.
What is also confusing is, looking through the libvirt lxc code, shouldn't we be seeing a series of "Allowing any <blah>" log debug outputs in our logs? This is part of the virLXCCgroupSetupDeviceACL part of the lxc_cgroup.c file [10]. I imagine that we'd eventually want the virCgroupAllowDeviceMajor method [11] invoked, no?
I realize that you are the author of most of this code, Daniel, so appreciate your patience with me and hope you are willing to continue trying to get to the bottom of this with us! Thanks a lot in advance!
Cheers, Fil
[1] https://gist.github.com/filmaj/207ab3c09f5b881d83b1 [2] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L... [3] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L... [4] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296 [5] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302 [6] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685 [7] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L... [8] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075 [9] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090 [10] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955... [11] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955...
On Wed, Apr 16, 2014 at 5:11 PM, Filip Maj <fil@saucelabs.com> wrote:
Further followups!
We are correlating DEBUG-level output from libvirt with the libvirt 1.2.2 code to try to figure out what libvirt is doing under the hood.
Even though we have the log level set to 1 (info) in our libvirtd.conf, we are not seeing the VIR_DEBUG() [1] statements being printed out. There are tons of other presumably-debug lines of output showing up in our log. We are sort of expecting to see the output from [1] in our logs somewhere, given the type of usb mounting we are trying to do?
[1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c0...
On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <fil@saucelabs.com> wrote:
To follow up on this a little bit, tail'ing kern.log while trying to get our little container up doesn't yield anything with apparmor complaining, so, unless I'm looking in the wrong spots for apparmor logs (which I don't think so, as I see other apparmor-related log entries in kern.log), I am not entirely sure this is an apparmor issue at this point.
On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil@saucelabs.com> wrote:
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode:
$ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626)
And still get issues. From libvirtd.log:
2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices
Thanks in advance for any help, Daniel!
Cheers, Fil
On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange < berrange@redhat.com> wrote:
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote:
Hi!
First post, kind of a noobie. I've been working with LXC and libvirt for a few months now. Trying to do some interesting things with containers and Android devices :D Here's my entire domain definition:
<domain type='lxc'> <name>oshi32134</name> <uuid>xxxxx</uuid> <memory unit='KiB'>3145728</memory> <currentMemory unit='KiB'>3145728</currentMemory> <vcpu placement='static'>1</vcpu> <resource> <partition>/machine</partition> </resource> <os> <type arch='i686'>exe</type> <init>/sbin/init</init> </os> <clock offset='utc'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> <filesystem type='mount' accessmode='passthrough'> <source dir='/some/valid/filesystem/location'/> <target dir='/'/> </filesystem> <filesystem type='mount' accessmode='passthrough'> <source dir='/another/valid/filesystem/location'/> <target dir='/mnt/android'/> </filesystem> <interface type='bridge'> <mac address='xx:xx:xx:xx:xx:xx'/> <source bridge='br1'/> </interface> <console type='pty'> <target type='lxc' port='0'/> </console> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/kvm</char> </source> </hostdev> <hostdev mode='subsystem' type='usb' managed='yes'> <source> <vendor id='0x04e8'/> <product id='0x6860'/> </source> </hostdev> </devices> </domain>
Your config looks fine here.
Everything worked fine until I added the USB <hostdev> element. I'm essentially trying to get access to a physical Android device
connected to
the host from inside a container. When I go to start the container, I get an error about Operation not permitted. Here's the relevant bits from /var/log/libvirt/lxc/machine.log:
2014-04-11 22:46:40.491+0000: starting up PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc --name oshi32134 --console 24 --security=none --handshake 27 --background --veth vnet1 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 2014-04-11 22:46:40.597+0000: 685: error : virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
Do you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Indeed, we've applied that code change, compiled and installed the patched source, and things work great! That was a fun code dive :D On Thu, Apr 17, 2014 at 2:44 PM, Filip Maj <fil@saucelabs.com> wrote:
Me again!
Think we've found it.
By diving into the LXC logs for the specific container, we found this:
2014-04-17 21:07:06.066+0000: 2861: debug : virCgroupSetValueStr:678 : Set value '/sys/fs/cgroup/devices/machine/oshi32134.libvirt-lxc/devices.allow' to 'c 189:130 rw'
Looks like libvirt the permission to 'rw', not 'rmw' [1], so no surprise that when it goes to call mknod [2], it can't do so.
We are going to patch this so it gives 'rwm' access in the cgroup whitelist, try to compile and replace libvirt to see if that works out for us. Will report back :)
[1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955... [2] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=5c...
On Thu, Apr 17, 2014 at 11:26 AM, Filip Maj <fil@saucelabs.com> wrote:
Ah, nevermind, think my last post may not have been entirely correct. We've spent some more time correlating the log output from our failed LXC startup via libvirt [1] with libvirt code from the 1.2.2 tag. Interestingly, the error we get back is different, and digging through the multitudes of logs, nothing comes up about 'operating not permitted' regarding the USB bus that I saw earlier. If we turn off the verbose logging, we get the original error back. The more-verbose error is a bit more cryptic:
libvirt LXC Driver error: internal error: guest failed to start: 2014-
Anyways, here's a rough overview of the logs:
1. it starts to check/detect for cgroup stuff [2] 2. starts to 'prepare host devices' and logs out that the appropriate usb address is initialized (002/003) [3] 3. sets the appropriate usb bus to 'active host devs' [4] 4. seems to then switch to doing network-related things, and stop talking about any hostdev/usb stuff [5] 5. eventually invokes `libvirt_lxc` [6] 6. does the cgroup song-n-dance again for some reason? [7]. not sure if this is critical or not. 7. eventually some kind of socket read wire error comes up [8] 8. and this leads to a 'guest failed to start error' [9], which it probably truncates to return the error i mentioned above.
What is also confusing is, looking through the libvirt lxc code, shouldn't we be seeing a series of "Allowing any <blah>" log debug outputs in our logs? This is part of the virLXCCgroupSetupDeviceACL part of the lxc_cgroup.c file [10]. I imagine that we'd eventually want the virCgroupAllowDeviceMajor method [11] invoked, no?
I realize that you are the author of most of this code, Daniel, so appreciate your patience with me and hope you are willing to continue trying to get to the bottom of this with us! Thanks a lot in advance!
Cheers, Fil
[1] https://gist.github.com/filmaj/207ab3c09f5b881d83b1 [2] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L226-L... [3] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L257-L... [4] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L296 [5] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L302 [6] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L685 [7] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L752-L... [8] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1075 [9] https://gist.github.com/filmaj/207ab3c09f5b881d83b1#file-libvirtd-log-L1090 [10] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955... [11] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_cgroup.c;h=39d955...
On Wed, Apr 16, 2014 at 5:11 PM, Filip Maj <fil@saucelabs.com> wrote:
Further followups!
We are correlating DEBUG-level output from libvirt with the libvirt 1.2.2 code to try to figure out what libvirt is doing under the hood.
Even though we have the log level set to 1 (info) in our libvirtd.conf, we are not seeing the VIR_DEBUG() [1] statements being printed out. There are tons of other presumably-debug lines of output showing up in our log. We are sort of expecting to see the output from [1] in our logs somewhere, given the type of usb mounting we are trying to do?
[1] http://libvirt.org/git/?p=libvirt.git;a=blob;f=src/lxc/lxc_controller.c;h=c0...
On Wed, Apr 16, 2014 at 3:58 PM, Filip Maj <fil@saucelabs.com> wrote:
To follow up on this a little bit, tail'ing kern.log while trying to get our little container up doesn't yield anything with apparmor complaining, so, unless I'm looking in the wrong spots for apparmor logs (which I don't think so, as I see other apparmor-related log entries in kern.log), I am not entirely sure this is an apparmor issue at this point.
On Wed, Apr 16, 2014 at 3:25 PM, Filip Maj <fil@saucelabs.com> wrote:
Yeah, AppArmor is enabled, but I put everything (that I could find) into complain mode:
$ sudo apparmor_status apparmor module is loaded. 12 profiles are loaded. 3 profiles are in enforce mode. lxc-container-default lxc-container-default-with-mounting lxc-container-default-with-nesting 9 profiles are in complain mode. /sbin/dhclient /usr/bin/lxc-start /usr/lib/NetworkManager/nm-dhcp-client.action /usr/lib/connman/scripts/dhclient-script /usr/lib/libvirt/virt-aa-helper /usr/sbin/libvirtd /usr/sbin/ntpd /usr/sbin/rsyslogd /usr/sbin/tcpdump 3 processes have profiles defined. 0 processes are in enforce mode. 2 processes are in complain mode. /usr/sbin/libvirtd (30419) /usr/sbin/ntpd (3418) 1 processes are unconfined but have a profile defined. /usr/sbin/rsyslogd (626)
And still get issues. From libvirtd.log:
2014-04-16 22:19:10.855+0000: 30419: info : libvirt version: 1.2.2 2014-04-16 22:19:10.855+0000: 30419: error : virNetSocketReadWire:1446 : Cannot recv data: Connection reset by peer 2014-04-16 22:19:10.940+0000: 30420: error : virLXCProcessStart:1299 : internal error: guest failed to start: Unable to create device //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not permitted
2014-04-16 22:19:10.964+0000: 30420: warning : virLXCDomainReAttachHostUsbDevices:388 : Unable to find device 000.000 in list of active USB devices
Thanks in advance for any help, Daniel!
Cheers, Fil
On Tue, Apr 15, 2014 at 1:33 AM, Daniel P. Berrange < berrange@redhat.com> wrote:
On Fri, Apr 11, 2014 at 05:32:28PM -0700, Filip Maj wrote: > Hi! > > First post, kind of a noobie. I've been working with LXC and libvirt for a > few months now. Trying to do some interesting things with containers and > Android devices :D > Here's my entire domain definition: > > <domain type='lxc'> > <name>oshi32134</name> > <uuid>xxxxx</uuid> > <memory unit='KiB'>3145728</memory> > <currentMemory unit='KiB'>3145728</currentMemory> > <vcpu placement='static'>1</vcpu> > <resource> > <partition>/machine</partition> > </resource> > <os> > <type arch='i686'>exe</type> > <init>/sbin/init</init> > </os> > <clock offset='utc'/> > <on_poweroff>destroy</on_poweroff> > <on_reboot>restart</on_reboot> > <on_crash>destroy</on_crash> > <devices> > <emulator>/usr/lib/libvirt/libvirt_lxc</emulator> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/some/valid/filesystem/location'/> > <target dir='/'/> > </filesystem> > <filesystem type='mount' accessmode='passthrough'> > <source dir='/another/valid/filesystem/location'/> > <target dir='/mnt/android'/> > </filesystem> > <interface type='bridge'> > <mac address='xx:xx:xx:xx:xx:xx'/> > <source bridge='br1'/> > </interface> > <console type='pty'> > <target type='lxc' port='0'/> > </console> > <hostdev mode='capabilities' type='misc'> > <source> > <char>/dev/kvm</char> > </source> > </hostdev> > <hostdev mode='subsystem' type='usb' managed='yes'> > <source> > <vendor id='0x04e8'/> > <product id='0x6860'/> > </source> > </hostdev> > </devices> > </domain>
Your config looks fine here.
> > Everything worked fine until I added the USB <hostdev> element. I'm > essentially trying to get access to a physical Android device connected to > the host from inside a container. When I go to start the container, I get > an error about Operation not permitted. Here's the relevant bits from > /var/log/libvirt/lxc/machine.log: > > 2014-04-11 22:46:40.491+0000: starting up > PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin > LIBVIRT_DEBUG=3 LIBVIRT_LOG_OUTPUTS=3:stderr /usr/lib/libvirt/libvirt_lxc > --name oshi32134 --console 24 --security=none --handshake 27 --background > --veth vnet1 > 2014-04-11 22:46:40.597+0000: 685: info : libvirt version: 1.2.2 > 2014-04-11 22:46:40.597+0000: 685: error : > virLXCControllerSetupHostdevSubsysUSB:1390 : Unable to create device > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > permitted > Unable to create device > //var/run/libvirt/lxc/oshi32134.dev/bus/usb//002//003: Operation not > permitted
Do you have AppArmour enabled on the machine. That seems like the most likely thing that would result in libvirt getting that permission error.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (2)
-
Daniel P. Berrange -
Filip Maj