USB-hotplugging fails with "failed to load cgroup BPF prog: Operation not permitted" on cgroups v2
Hi all, I've disabled cgroups v1 on my system with the kernel boot option "systemd.unified_cgroup_hierarchy=1". Since doing so, USB hotplugging fails to work, seemingly due to a permissions problem with BPF. Please note that the technique I'm going to describe worked just fine for hotplugging USB devices to running domains until this change. Attaching / detaching USB devices when the domain is down still works as expected. I get the same error when attaching a device in virt-manager, as I do when running the following command: sudo virsh attach-device wenger /dev/stdin --persistent <<END <hostdev mode='subsystem' type='usb' managed='yes'> <source startupPolicy='optional'> <vendor id='0x046d' /> <product id='0xc215' /> </source> </hostdev> END This returns error: Failed to attach device from /dev/stdin error: failed to load cgroup BPF prog: Operation not permitted virt-manager returns basically the same error, but for completeness' sake, here it is: failed to load cgroup BPF prog: Operation not permitted Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/addhardware.py", line 1327, in _add_device self.vm.attach_device(dev) File "/usr/share/virt-manager/virtManager/object/domain.py", line 920, in attach_device self._backend.attachDevice(devxml) File "/usr/lib/python3.8/site-packages/libvirt.py", line 590, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirt.libvirtError: failed to load cgroup BPF prog: Operation not permitted Now, libvirtd is running as root, so I don't understand why any operation on BPF programs is not permitted. I've dug into libvirt's code a bit to see what is throwing this error and it boils down to <https://github.com/libvirt/libvirt/blob/7d608469621a3fda72dff2a89308e68cc9fb4c9a/src/util/vircgroupv2devices.c#L292-L296> and <https://github.com/libvirt/libvirt/blob/02bf7cc68bfc76242f02d23e73cad36618f3f790/src/util/virbpf.c#L54> but I have no clue what that syscall is doing, so that's where my debugging capability basically ends. Maybe this is something as simple as setting the right ACL somewhere. I haven't touched /etc/libvirt/qemu.conf except for setting nvram. There *is* something about cgroup_device_acl there but afaict that's for cgroups v1, when there was still a device cgroup controller. Any help would be greatly appreciated. Domain log files: Upon execution of the above commands, nothing gets added to the domain log in /var/log/qemu/wenger.log, so I've decided they're likely irrelevant to the issue. Please ask for any additional info required. System information: Arch Linux, (normal) kernel 5.4.11 libvirt 5.10.0 qemu 4.2.0, using KVM. Host system is x86_64 on an intel 5820k. Guest system is probably irrelevant, but is Windows 10 on the same. Possibly relevant kernel build options: $ zgrep BPF /proc/config.gz [22:55:52]: zgrep BPF /proc/config.gz CONFIG_CGROUP_BPF=y CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_BPF_JIT_ALWAYS_ON=y CONFIG_IPV6_SEG6_BPF=y CONFIG_NETFILTER_XT_MATCH_BPF=m # CONFIG_BPFILTER is not set CONFIG_NET_CLS_BPF=m CONFIG_NET_ACT_BPF=m CONFIG_BPF_JIT=y CONFIG_BPF_STREAM_PARSER=y CONFIG_LWTUNNEL_BPF=y CONFIG_HAVE_EBPF_JIT=y CONFIG_BPF_EVENTS=y # CONFIG_BPF_KPROBE_OVERRIDE is not set # CONFIG_TEST_BPF is not set Regards, Pol Van Aubel
On Sat, Jan 18, 2020 at 11:17:11PM +0100, Pol Van Aubel wrote:
Hi all,
I've disabled cgroups v1 on my system with the kernel boot option "systemd.unified_cgroup_hierarchy=1". Since doing so, USB hotplugging fails to work, seemingly due to a permissions problem with BPF. Please note that the technique I'm going to describe worked just fine for hotplugging USB devices to running domains until this change. Attaching / detaching USB devices when the domain is down still works as expected.
I get the same error when attaching a device in virt-manager, as I do when running the following command:
sudo virsh attach-device wenger /dev/stdin --persistent <<END <hostdev mode='subsystem' type='usb' managed='yes'> <source startupPolicy='optional'> <vendor id='0x046d' /> <product id='0xc215' /> </source> </hostdev> END
This returns error: Failed to attach device from /dev/stdin error: failed to load cgroup BPF prog: Operation not permitted
virt-manager returns basically the same error, but for completeness' sake, here it is:
failed to load cgroup BPF prog: Operation not permitted
Traceback (most recent call last): File "/usr/share/virt-manager/virtManager/addhardware.py", line 1327, in _add_device self.vm.attach_device(dev) File "/usr/share/virt-manager/virtManager/object/domain.py", line 920, in attach_device self._backend.attachDevice(devxml) File "/usr/lib/python3.8/site-packages/libvirt.py", line 590, in attachDevice if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) libvirt.libvirtError: failed to load cgroup BPF prog: Operation not permitted
Now, libvirtd is running as root, so I don't understand why any operation on BPF programs is not permitted. I've dug into libvirt's code a bit to see what is throwing this error and it boils down to <https://github.com/libvirt/libvirt/blob/7d608469621a3fda72dff2a89308e68cc9fb4c9a/src/util/vircgroupv2devices.c#L292-L296> and <https://github.com/libvirt/libvirt/blob/02bf7cc68bfc76242f02d23e73cad36618f3f790/src/util/virbpf.c#L54> but I have no clue what that syscall is doing, so that's where my debugging capability basically ends.
Maybe this is something as simple as setting the right ACL somewhere. I haven't touched /etc/libvirt/qemu.conf except for setting nvram. There *is* something about cgroup_device_acl there but afaict that's for cgroups v1, when there was still a device cgroup controller. Any help would be greatly appreciated.
Domain log files: Upon execution of the above commands, nothing gets added to the domain log in /var/log/qemu/wenger.log, so I've decided they're likely irrelevant to the issue. Please ask for any additional info required.
System information: Arch Linux, (normal) kernel 5.4.11 libvirt 5.10.0 qemu 4.2.0, using KVM. Host system is x86_64 on an intel 5820k. Guest system is probably irrelevant, but is Windows 10 on the same.
Possibly relevant kernel build options: $ zgrep BPF /proc/config.gz [22:55:52]: zgrep BPF /proc/config.gz
CONFIG_CGROUP_BPF=y CONFIG_BPF=y CONFIG_BPF_SYSCALL=y CONFIG_BPF_JIT_ALWAYS_ON=y CONFIG_IPV6_SEG6_BPF=y CONFIG_NETFILTER_XT_MATCH_BPF=m # CONFIG_BPFILTER is not set CONFIG_NET_CLS_BPF=m CONFIG_NET_ACT_BPF=m CONFIG_BPF_JIT=y CONFIG_BPF_STREAM_PARSER=y CONFIG_LWTUNNEL_BPF=y CONFIG_HAVE_EBPF_JIT=y CONFIG_BPF_EVENTS=y # CONFIG_BPF_KPROBE_OVERRIDE is not set # CONFIG_TEST_BPF is not set
Hi I've installed clean archlinux to try this out and it works as expected, I'm able to attach USB device into a VM. My system env is mostly the same as yours except for kernel version: kernel 5.4.13 libvirt 5.10.0 qemu 4.2.0, using KVM. Please enable libvirt debug logs [1] and share the output with us. Pavel [1] <https://wiki.libvirt.org/page/DebugLogs>
participants (2)
-
Pavel Hrdina -
Pol Van Aubel