Re: [libvirt-users] LXC + USB passthrough = Operation not permitted
Dear Fil, I am not sure if my answer can help you. I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn't make sense. Maybe you can try to use hostdev mode='capabilities'. Please see http://libvirt.org/formatdomain.html#elementsHostDevCaps Hope this helps Cheng Wang
Hi Chang Weng, Thanks for answering! Could you explain why subsystem=USB does not make sense? I was simply following the documentation for USB devices (right above the section you linked to). For what it's worth, I've also tried mounting the specific character device related to the appropriate android device into the container in the way you suggest, with no luck; `adb` still does not recognize the device. On Apr 13, 2014 7:10 PM, "WANG Cheng D" <Cheng.d.Wang@alcatel-sbell.com.cn> wrote:
Dear Fil,
I am not sure if my answer can help you.
I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn't make sense. Maybe you can try to use hostdev mode='capabilities'. Please see http://libvirt.org/formatdomain.html#elementsHostDevCaps
Hope this helps
Cheng Wang
Can you see the USB device in the lxc? My answering assumes you cannot see the USB device in the container. If you can see it, please ignore my answer. According to my understanding, “hostdev mode='subsystem' “ is used with KVM only. We use this if we want to passthrough a device to a KVM virtual machine. After doing that, the host machine will lose control of the device. If container is used, the container is not a real virtual machine, it’s just a group of processes and it shares the OS kernel with the host. If a device cannot be seen by the host machine, I guess it also cannot been seen by the container. I used the following XML scripts to expose a third-party PCIe card to the container successfully, and I can access the card in the lxc. <hostdev mode=’capabilities’ type=’misc’> <source> <char/dev/mydevice</char> </source> </hostdev> Although I am not sure if my card is a char device, but it really works. If I use “ hostdev mode=’subsystem’”, it will fail to expose the PCIe card. From: Filip Maj [mailto:fil@saucelabs.com] Sent: 2014年4月14日 11:26 To: WANG Cheng D Cc: libvirt-users@redhat.com Subject: RE: [libvirt-users] LXC + USB passthrough = Operation not permitted Hi Chang Weng, Thanks for answering! Could you explain why subsystem=USB does not make sense? I was simply following the documentation for USB devices (right above the section you linked to). For what it's worth, I've also tried mounting the specific character device related to the appropriate android device into the container in the way you suggest, with no luck; `adb` still does not recognize the device. On Apr 13, 2014 7:10 PM, "WANG Cheng D" <Cheng.d.Wang@alcatel-sbell.com.cn<mailto:Cheng.d.Wang@alcatel-sbell.com.cn>> wrote: Dear Fil, I am not sure if my answer can help you. I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn’t make sense. Maybe you can try to use hostdev mode='capabilities’. Please see http://libvirt.org/formatdomain.html#elementsHostDevCaps Hope this helps Cheng Wang
So, if I remove the hostdev element related to the USB device, running `lsusb` and `usb-devices` inside the container lists the proper device, but I do not see the appropriate character device bus for the Android device under /dev/bus/usb. We are not entirely sure that `lsusb` and `usb-devices` queries the appropriate location - it somehow gets information from the host, not the guest container, as far as we can tell. If we specify the appropriate usb bus character device path (like you suggest, Cheng), then it does appear inside the container, but the permissions and ownership on the device seem to be incorrect. Here are the details of the appropriate device busses in the host (002 and 003 are the specific Android devices): $ ll /dev/bus/usb/002 total 0 drwxr-xr-x 2 root root 100 Apr 9 18:14 ./ drwxr-xr-x 10 root root 200 Apr 9 18:14 ../ crw-rw-r-- 1 root root 189, 128 Apr 9 18:14 001 crw-rw-r-- 1 root plugdev 189, 129 Apr 9 18:14 002 crw-rw-r-- 1 root plugdev 189, 130 Apr 10 15:25 003 Then, I removed the old mode="subsystem" hostdev element and added the following hostdev elements: <hostdev mode='capabilities' type='misc'> <source> <char>/dev/bus/usb/002/002</char> </source> </hostdev> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/bus/usb/002/003</char> </source> </hostdev> Then I get inside the guest container and list out the usb busses we just mounted: $ ll /dev/bus/usb/002/ total 0 drwxr-xr-x 2 root root 80 Apr 14 17:43 ./ drwxr-xr-x 3 root root 60 Apr 14 17:43 ../ crwx------ 1 root root 189, 129 Apr 14 17:43 002 crwx------ 1 root root 189, 130 Apr 14 17:43 003 You can see the group that owns the character devices is different, and the read/write permissions are different, too. Not sure if this is the root of the issue, but, I noticed that difference. In any case, the Android devices are still not recognized by the container. Any help is appreciated! Cheers, Fil On Sun, Apr 13, 2014 at 8:50 PM, WANG Cheng D < Cheng.d.Wang@alcatel-sbell.com.cn> wrote:
Can you see the USB device in the lxc?
My answering assumes you cannot see the USB device in the container. If you can see it, please ignore my answer.
According to my understanding, “hostdev mode='subsystem' “ is used with KVM only. We use this if we want to passthrough a device to a KVM virtual machine. After doing that, the host machine will lose control of the device. If container is used, the container is not a real virtual machine, it’s just a group of processes and it shares the OS kernel with the host. If a device cannot be seen by the host machine, I guess it also cannot been seen by the container.
I used the following XML scripts to expose a third-party PCIe card to the container successfully, and I can access the card in the lxc.
<hostdev mode=’capabilities’ type=’misc’>
<source>
<char/dev/mydevice</char>
</source>
</hostdev>
Although I am not sure if my card is a char device, but it really works.
If I use “ hostdev mode=’subsystem’”, it will fail to expose the PCIe card.
*From:* Filip Maj [mailto:fil@saucelabs.com] *Sent:* 2014年4月14日 11:26 *To:* WANG Cheng D *Cc:* libvirt-users@redhat.com *Subject:* RE: [libvirt-users] LXC + USB passthrough = Operation not permitted
Hi Chang Weng,
Thanks for answering!
Could you explain why subsystem=USB does not make sense? I was simply following the documentation for USB devices (right above the section you linked to).
For what it's worth, I've also tried mounting the specific character device related to the appropriate android device into the container in the way you suggest, with no luck; `adb` still does not recognize the device.
On Apr 13, 2014 7:10 PM, "WANG Cheng D" <Cheng.d.Wang@alcatel-sbell.com.cn> wrote:
Dear Fil,
I am not sure if my answer can help you.
I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn’t make sense. Maybe you can try to use hostdev mode='capabilities’. Please see http://libvirt.org/formatdomain.html#elementsHostDevCaps
Hope this helps
Cheng Wang
I'd also like to point out that, when setting up a linux container with _just_ the lxc tools available in ubuntu, all that was needed to be done was whitelist the appropriate cgroups for the Android devices. The lxc tools seem to create the appropriate character devices when you detach/attach the Android device via USB, and the Android `adb` tool recognized the devices. So, a tangential question would be, what libvirt domain XML ends up "converting" into a cgroup whitelist line in the LXC configuration file? (see https://wiki.archlinux.org/index.php/Linux_Containers#Host_device_access_set... examples of cgroups configuration in a vanilla LXC config file). On Mon, Apr 14, 2014 at 10:44 AM, Filip Maj <fil@saucelabs.com> wrote:
So, if I remove the hostdev element related to the USB device, running `lsusb` and `usb-devices` inside the container lists the proper device, but I do not see the appropriate character device bus for the Android device under /dev/bus/usb. We are not entirely sure that `lsusb` and `usb-devices` queries the appropriate location - it somehow gets information from the host, not the guest container, as far as we can tell.
If we specify the appropriate usb bus character device path (like you suggest, Cheng), then it does appear inside the container, but the permissions and ownership on the device seem to be incorrect. Here are the details of the appropriate device busses in the host (002 and 003 are the specific Android devices):
$ ll /dev/bus/usb/002 total 0 drwxr-xr-x 2 root root 100 Apr 9 18:14 ./ drwxr-xr-x 10 root root 200 Apr 9 18:14 ../ crw-rw-r-- 1 root root 189, 128 Apr 9 18:14 001 crw-rw-r-- 1 root plugdev 189, 129 Apr 9 18:14 002 crw-rw-r-- 1 root plugdev 189, 130 Apr 10 15:25 003
Then, I removed the old mode="subsystem" hostdev element and added the following hostdev elements:
<hostdev mode='capabilities' type='misc'> <source> <char>/dev/bus/usb/002/002</char> </source> </hostdev> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/bus/usb/002/003</char> </source> </hostdev>
Then I get inside the guest container and list out the usb busses we just mounted:
$ ll /dev/bus/usb/002/ total 0 drwxr-xr-x 2 root root 80 Apr 14 17:43 ./ drwxr-xr-x 3 root root 60 Apr 14 17:43 ../ crwx------ 1 root root 189, 129 Apr 14 17:43 002 crwx------ 1 root root 189, 130 Apr 14 17:43 003
You can see the group that owns the character devices is different, and the read/write permissions are different, too. Not sure if this is the root of the issue, but, I noticed that difference. In any case, the Android devices are still not recognized by the container.
Any help is appreciated!
Cheers, Fil
On Sun, Apr 13, 2014 at 8:50 PM, WANG Cheng D < Cheng.d.Wang@alcatel-sbell.com.cn> wrote:
Can you see the USB device in the lxc?
My answering assumes you cannot see the USB device in the container. If you can see it, please ignore my answer.
According to my understanding, “hostdev mode='subsystem' “ is used with KVM only. We use this if we want to passthrough a device to a KVM virtual machine. After doing that, the host machine will lose control of the device. If container is used, the container is not a real virtual machine, it’s just a group of processes and it shares the OS kernel with the host. If a device cannot be seen by the host machine, I guess it also cannot been seen by the container.
I used the following XML scripts to expose a third-party PCIe card to the container successfully, and I can access the card in the lxc.
<hostdev mode=’capabilities’ type=’misc’>
<source>
<char/dev/mydevice</char>
</source>
</hostdev>
Although I am not sure if my card is a char device, but it really works.
If I use “ hostdev mode=’subsystem’”, it will fail to expose the PCIe card.
*From:* Filip Maj [mailto:fil@saucelabs.com] *Sent:* 2014年4月14日 11:26 *To:* WANG Cheng D *Cc:* libvirt-users@redhat.com *Subject:* RE: [libvirt-users] LXC + USB passthrough = Operation not permitted
Hi Chang Weng,
Thanks for answering!
Could you explain why subsystem=USB does not make sense? I was simply following the documentation for USB devices (right above the section you linked to).
For what it's worth, I've also tried mounting the specific character device related to the appropriate android device into the container in the way you suggest, with no luck; `adb` still does not recognize the device.
On Apr 13, 2014 7:10 PM, "WANG Cheng D" < Cheng.d.Wang@alcatel-sbell.com.cn> wrote:
Dear Fil,
I am not sure if my answer can help you.
I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn’t make sense. Maybe you can try to use hostdev mode='capabilities’. Please see http://libvirt.org/formatdomain.html#elementsHostDevCaps
Hope this helps
Cheng Wang
Hi Filip, I just checked about my machine. 1) Just like you said, in my container, the read/write permission of the exposed PCIe device is also different from the host machine. Only the root can read/write the PCIe card. Other users/groups are not allowed to access the card. 2) Both my host machine and lxc are fedora16. As I login in the lxc, I became root automatically, so I can access the card without problem. As I know, if Ubuntu is used, the root might be treated differently from Fedora. I am not sure if this is the reason. For next step 1) You can try fedora 2) Open a new post to see if lxc experts can help you Good luck! Cheng From: Filip Maj [mailto:fil@saucelabs.com] Sent: 2014年4月15日 1:57 To: WANG Cheng D Cc: libvirt-users@redhat.com Subject: Re: [libvirt-users] LXC + USB passthrough = Operation not permitted I'd also like to point out that, when setting up a linux container with _just_ the lxc tools available in ubuntu, all that was needed to be done was whitelist the appropriate cgroups for the Android devices. The lxc tools seem to create the appropriate character devices when you detach/attach the Android device via USB, and the Android `adb` tool recognized the devices. So, a tangential question would be, what libvirt domain XML ends up "converting" into a cgroup whitelist line in the LXC configuration file? (see https://wiki.archlinux.org/index.php/Linux_Containers#Host_device_access_set... for examples of cgroups configuration in a vanilla LXC config file). On Mon, Apr 14, 2014 at 10:44 AM, Filip Maj <fil@saucelabs.com<mailto:fil@saucelabs.com>> wrote: So, if I remove the hostdev element related to the USB device, running `lsusb` and `usb-devices` inside the container lists the proper device, but I do not see the appropriate character device bus for the Android device under /dev/bus/usb. We are not entirely sure that `lsusb` and `usb-devices` queries the appropriate location - it somehow gets information from the host, not the guest container, as far as we can tell. If we specify the appropriate usb bus character device path (like you suggest, Cheng), then it does appear inside the container, but the permissions and ownership on the device seem to be incorrect. Here are the details of the appropriate device busses in the host (002 and 003 are the specific Android devices): $ ll /dev/bus/usb/002 total 0 drwxr-xr-x 2 root root 100 Apr 9 18:14 ./ drwxr-xr-x 10 root root 200 Apr 9 18:14 ../ crw-rw-r-- 1 root root 189, 128 Apr 9 18:14 001 crw-rw-r-- 1 root plugdev 189, 129 Apr 9 18:14 002 crw-rw-r-- 1 root plugdev 189, 130 Apr 10 15:25 003 Then, I removed the old mode="subsystem" hostdev element and added the following hostdev elements: <hostdev mode='capabilities' type='misc'> <source> <char>/dev/bus/usb/002/002</char> </source> </hostdev> <hostdev mode='capabilities' type='misc'> <source> <char>/dev/bus/usb/002/003</char> </source> </hostdev> Then I get inside the guest container and list out the usb busses we just mounted: $ ll /dev/bus/usb/002/ total 0 drwxr-xr-x 2 root root 80 Apr 14 17:43 ./ drwxr-xr-x 3 root root 60 Apr 14 17:43 ../ crwx------ 1 root root 189, 129 Apr 14 17:43 002 crwx------ 1 root root 189, 130 Apr 14 17:43 003 You can see the group that owns the character devices is different, and the read/write permissions are different, too. Not sure if this is the root of the issue, but, I noticed that difference. In any case, the Android devices are still not recognized by the container. Any help is appreciated! Cheers, Fil On Sun, Apr 13, 2014 at 8:50 PM, WANG Cheng D <Cheng.d.Wang@alcatel-sbell.com.cn<mailto:Cheng.d.Wang@alcatel-sbell.com.cn>> wrote: Can you see the USB device in the lxc? My answering assumes you cannot see the USB device in the container. If you can see it, please ignore my answer. According to my understanding, “hostdev mode='subsystem' “ is used with KVM only. We use this if we want to passthrough a device to a KVM virtual machine. After doing that, the host machine will lose control of the device. If container is used, the container is not a real virtual machine, it’s just a group of processes and it shares the OS kernel with the host. If a device cannot be seen by the host machine, I guess it also cannot been seen by the container. I used the following XML scripts to expose a third-party PCIe card to the container successfully, and I can access the card in the lxc. <hostdev mode=’capabilities’ type=’misc’> <source> <char/dev/mydevice</char> </source> </hostdev> Although I am not sure if my card is a char device, but it really works. If I use “ hostdev mode=’subsystem’”, it will fail to expose the PCIe card. From: Filip Maj [mailto:fil@saucelabs.com<mailto:fil@saucelabs.com>] Sent: 2014年4月14日 11:26 To: WANG Cheng D Cc: libvirt-users@redhat.com<mailto:libvirt-users@redhat.com> Subject: RE: [libvirt-users] LXC + USB passthrough = Operation not permitted Hi Chang Weng, Thanks for answering! Could you explain why subsystem=USB does not make sense? I was simply following the documentation for USB devices (right above the section you linked to). For what it's worth, I've also tried mounting the specific character device related to the appropriate android device into the container in the way you suggest, with no luck; `adb` still does not recognize the device. On Apr 13, 2014 7:10 PM, "WANG Cheng D" <Cheng.d.Wang@alcatel-sbell.com.cn<mailto:Cheng.d.Wang@alcatel-sbell.com.cn>> wrote: Dear Fil, I am not sure if my answer can help you. I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn’t make sense. Maybe you can try to use hostdev mode='capabilities’. Please see http://libvirt.org/formatdomain.html#elementsHostDevCaps Hope this helps Cheng Wang
On Mon, Apr 14, 2014 at 10:56:42AM -0700, Filip Maj wrote:
I'd also like to point out that, when setting up a linux container with _just_ the lxc tools available in ubuntu, all that was needed to be done was whitelist the appropriate cgroups for the Android devices. The lxc tools seem to create the appropriate character devices when you detach/attach the Android device via USB, and the Android `adb` tool recognized the devices.
So, a tangential question would be, what libvirt domain XML ends up "converting" into a cgroup whitelist line in the LXC configuration file? (see https://wiki.archlinux.org/index.php/Linux_Containers#Host_device_access_set... examples of cgroups configuration in a vanilla LXC config file).
You don't need to specify anything related to cgroups in libvirt. We will automatically set the device ACL correctly to allow any devices listed in the XML config. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On Mon, Apr 14, 2014 at 02:10:14AM +0000, WANG Cheng D wrote:
Dear Fil, I am not sure if my answer can help you. I had ever asked a similar question to Daniel and I was using a thirty-party card. As a container uses a shared kernel with the host, so hostdev mode='subsystem' doesn't make sense. Maybe you can try to use hostdev mode='capabilities'. Please see http://libvirt.org/formatdomain.html#elementsHostDevCaps
Actually mode='subsystem' is *IS* fully supported for USB devices. It is only PCI devices where it is not available with LXC. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
participants (3)
-
Daniel P. Berrange -
Filip Maj -
WANG Cheng D