Re: USB-hotplugging fails with "failed to load cgroup BPF prog: Operation not permitted" on cgroups v2
by Pavel Hrdina
On Mon, Jan 20, 2020 at 09:00:15PM +0100, Pol Van Aubel wrote:
> Hi,
>
> Quoting Pavel Hrdina (2020-01-20 14:29:36)
> > On Sat, Jan 18, 2020 at 11:17:11PM +0100, Pol Van Aubel wrote:
> > > Hi all,
> > >
> > > I've disabled cgroups v1 on my system with the kernel boot option
> > > "systemd.unified_cgroup_hierarchy=1". Since doing so, USB hotplugging
> > > fails to work, seemingly due to a permissions problem with BPF. Please
> > > note that the technique I'm going to describe worked just fine for
> > > hotplugging USB devices to running domains until this change.
> > > Attaching / detaching USB devices when the domain is down still works as
> > > expected.
> > >
> > > I get the same error when attaching a device in virt-manager, as I do
> > > when running the following command:
> > >
> > > sudo virsh attach-device wenger /dev/stdin --persistent <<END
> > > <hostdev mode='subsystem' type='usb' managed='yes'>
> > > <source startupPolicy='optional'>
> > > <vendor id='0x046d' />
> > > <product id='0xc215' />
> > > </source>
> > > </hostdev>
> > > END
> > >
> > > This returns
> > > error: Failed to attach device from /dev/stdin
> > > error: failed to load cgroup BPF prog: Operation not permitted
> > >
> > >
> > > virt-manager returns basically the same error, but for completeness'
> > > sake, here it is:
> > >
> > > failed to load cgroup BPF prog: Operation not permitted
> > >
> > > Traceback (most recent call last):
> > > File "/usr/share/virt-manager/virtManager/addhardware.py", line 1327, in _add_device
> > > self.vm.attach_device(dev)
> > > File "/usr/share/virt-manager/virtManager/object/domain.py", line 920, in attach_device
> > > self._backend.attachDevice(devxml)
> > > File "/usr/lib/python3.8/site-packages/libvirt.py", line 590, in attachDevice
> > > if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self)
> > > libvirt.libvirtError: failed to load cgroup BPF prog: Operation not permitted
> > >
> > >
> > > Now, libvirtd is running as root, so I don't understand why any
> > > operation on BPF programs is not permitted. I've dug into libvirt's code
> > > a bit to see what is throwing this error and it boils down to
> > > <https://github.com/libvirt/libvirt/blob/7d608469621a3fda72dff2a89308e68cc...>
> > > and
> > > <https://github.com/libvirt/libvirt/blob/02bf7cc68bfc76242f02d23e73cad3661...>
> > > but I have no clue what that syscall is doing, so that's where my
> > > debugging capability basically ends.
> > >
> > > Maybe this is something as simple as setting the right ACL somewhere. I
> > > haven't touched /etc/libvirt/qemu.conf except for setting nvram. There
> > > *is* something about cgroup_device_acl there but afaict that's for
> > > cgroups v1, when there was still a device cgroup controller. Any help
> > > would be greatly appreciated.
> > >
> > >
> > > Domain log files:
> > > Upon execution of the above commands, nothing gets added to the domain
> > > log in /var/log/qemu/wenger.log, so I've decided they're likely
> > > irrelevant to the issue. Please ask for any additional info required.
> > >
> > >
> > > System information:
> > > Arch Linux, (normal) kernel 5.4.11
> > > libvirt 5.10.0
> > > qemu 4.2.0, using KVM.
> > > Host system is x86_64 on an intel 5820k.
> > > Guest system is probably irrelevant, but is Windows 10 on the same.
> > >
> > >
> > > Possibly relevant kernel build options:
> > > $ zgrep BPF /proc/config.gz
> > > [22:55:52]: zgrep BPF /proc/config.gz
> > >
> > > CONFIG_CGROUP_BPF=y
> > > CONFIG_BPF=y
> > > CONFIG_BPF_SYSCALL=y
> > > CONFIG_BPF_JIT_ALWAYS_ON=y
> > > CONFIG_IPV6_SEG6_BPF=y
> > > CONFIG_NETFILTER_XT_MATCH_BPF=m
> > > # CONFIG_BPFILTER is not set
> > > CONFIG_NET_CLS_BPF=m
> > > CONFIG_NET_ACT_BPF=m
> > > CONFIG_BPF_JIT=y
> > > CONFIG_BPF_STREAM_PARSER=y
> > > CONFIG_LWTUNNEL_BPF=y
> > > CONFIG_HAVE_EBPF_JIT=y
> > > CONFIG_BPF_EVENTS=y
> > > # CONFIG_BPF_KPROBE_OVERRIDE is not set
> > > # CONFIG_TEST_BPF is not set
> >
> > Hi
> >
> > I've installed clean archlinux to try this out and it works as expected,
> > I'm able to attach USB device into a VM.
> >
> > My system env is mostly the same as yours except for kernel version:
> >
> > kernel 5.4.13
> > libvirt 5.10.0
> > qemu 4.2.0, using KVM.
> >
> > Please enable libvirt debug logs [1] and share the output with us.
>
> I've updated to 5.4.13 and created a barebones VM without storage to
> reproduce the behaviour. libvirtd debug logs are attached. There appear
> to be two BPF failures of the same BPF program (?). The first is on line
> 23209, which appears to be part of machine startup, and which I don't
> actually notice. The second one is where I manually add the USB device,
> on line 30599.
>
> Thanks,
Thanks for the logs, but it did not help to figure out where the issue
is. I was hoping to see some error output from the syscall but the line
that should contain it is empty:
2020-01-20 19:47:15.589+0000: 8579: debug : virBPFLoadProg:78 :
Can you please check system logs and output of dmesg?
I've managed to run into this article [1] that explains that even if you
have all permissions and no SELinux you can still be blocked by
something called kernel_lockdown and it should appear in dmesg.
Pavel
[1] <https://gehrcke.de/2019/09/running-an-ebpf-program-may-require-lifting-th...>
4 years, 6 months
Libvirt APIs for creating virtual networks
by Santhosh Kumar Gunturu
I am using the Libvirt APIs to create the virtual networks to use the
dnsmasq capabilities.
I want to understand the APIs how to set the lease time in XML to control
the dnsmasq lease time.
Also, I need some DHCP statistics on the virtual network.
Is there any API available to fetch the statistics ?
Please let me know if any ?
Thanks & Regards
Santhosh Kumar Gunturu
4 years, 6 months
sync guest time
by Miguel Duarte de Mora Barroso
Hi,
I'm seeing the following issue when attempting to update the guest's
clock on a running fc32 guest (using guest agent):
```
[root@virt-launcher-vmi-masquerade-mh2xm /]# virsh domtime 1 --pretty
Time: 2020-04-30 23:27:29
[root@virt-launcher-vmi-masquerade-mh2xm /]# virsh domtime 1 --sync
error: internal error: unable to execute QEMU agent command
'guest-set-time': hwclock failed to set hardware clock to system time
# now, this one works.
[root@virt-launcher-vmi-masquerade-mh2xm /]# virsh domtime 1 --now
[root@virt-launcher-vmi-masquerade-44v2x /]# virsh domtime 1 --pretty
Time: 2020-04-30 11:15:45
```
This is the simplest reproducer I could come with; the original issue
is this call to libvirt's `setTIme` in [0].
Is there any workaround I could try ? Am I doing something wrong here ?
[0] - https://github.com/kubevirt/kubevirt/blob/6bb516148ce4c29825ae74f473e0220...
4 years, 6 months
Re: [Query] Facing issue in build libvirt6.0 on ubuntu 1804
by Michal Privoznik
[Please keep the list CCed]
On 4/30/20 12:02 PM, Ramesh B wrote:
> Hi Michal,
>
> Thanks for quick response.
>
> Ubuntu: 18.04.3 LTS
> QEMU emulator version 4.2.0
> libvirtd (libvirt) 4.0.0
>
> I have lunched the guest os using QEMU.
> I would like to use the virsh commands to get the status / control of
> guest OS.
> when i run any virsh commands not getting any response from QEMU.
> When i tried to explore, found the solution saying use of QEMU monitor
> will help.
> added the below command while lunch,
> -chardev
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-freebsd11.0/monitor.sock,server,nowait
> -mon chardev=charmonitor,id=monitor,mode=control
>
> but result in below mentioed error:
> Failed to bind socket to
> /var/lib/libvirt/qemu/domain-freebsd11.0/monitor.sock
>
> Then thought of building libvirt 6.0 version, will help to solve the
> problem.
Ah, so you wanted to use 'virsh qemu-attach' to attach libvirt to a
running qemu. Yeah, it needs a monitor. However, the functionality was
not maintained and probably not functional anyway so we've removed in
the 5.5.0 release.
>
> Please guide me to solve this problem.
>
> Regards,
> Babu B
>
But what's interesting is that you are unable to build. Which package is
providing the rst2* binaries? Are their up to date?
Michal
4 years, 6 months
[Query] Facing issue in build libvirt6.0 on ubuntu 1804
by Ramesh B
Hi all,
I am trying to build libvirt 6.0 on ubuntu OS 18.04.
Facing one issue libvirt source compilation.
============================================
<stdin>:215: (ERROR/3) Error in "code-block" directive:
1 argument(s) required, 0 supplied.
.. code-block::
help [command-or-group]
<stdin>:229: (ERROR/3) Error in "code-block" directive:
1 argument(s) required, 0 supplied.
.. code-block::
===========================================
*Source:* https://libvirt.org/sources/libvirt-6.0.0.tar.xz
*Build Command*:
$ mkdir build && cd build
$ ../configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
$ make
$ sudo make install
Need help in solving the build error.
is it possible to install libvirt package without build
if yes please share the command/procedure/link.
Thanks,
Ramesh
4 years, 6 months
why libvirtd can not list the running domain?
by 崔涛的个人邮箱
1. I run a user domain like this:
[root@kvm1 ~]# qemu-system-x86_64 -enable-kvm -m 1G -smp 4 /images/test1.img
WARNING: Image format was not specified for '/images/test1.img' and probing
guessed raw.
Automatically detecting the format is dangerous for raw images,
write operations on block 0 will be restricted.
Specify the 'raw' format explicitly to remove the restrictions.
VNC server running on ::1:5900
2. Then, I install and using my user domain.
3. Then, I login to libvirtd on my hosts,but can not get the user
domain using list command.
[root@kvm1 ~]# virsh -c qemu:///system
Welcome to virsh, the virtualization interactive terminal.
Type: 'help' for help with commands
'quit' to quit
virsh #
virsh #
virsh # list
Id Name State
----------------------------------------------------
virsh #
4 years, 6 months
"failed to setup INTx fd: Operation not permitted" error when using PCI passthrough
by Riccardo Ravaioli
Hi,
I'm on a Dell VEP 1405 running Debian 9.11 and I'm running a few tests with
various interfaces given in PCI passthrough to a qemu/KVM Virtual Machine
also running Debian 9.11.
I noticed that only one of the four I350 network controllers can be used in
PCI passthrough. The available interfaces are:
*# dpdk-devbind.py --status Network devices using kernel driver
==============================*
*===== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eth2 drv=igb
unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eth3
drv=igb unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:02:00.2 'I350 Gigabit Network Connection 1521' if=eth0
drv=igb unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:02:00.3 'I350 Gigabit Network Connection 1521' if=eth1
drv=igb unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:04:00.0 'QCA986x/988x 802.11ac Wireless Network Adapter
003c' if= drv=ath10k_pci unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:05:00.0 'Device 15c4' if=eth7 drv=ixgbe
unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:05:00.1 'Device 15c4' if=eth6 drv=ixgbe
unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:07:00.0 'Device 15e5' if=eth5 drv=ixgbe
unused=igb_uio,vfio-pci,uio_*
*pci_generic 0000:07:00.1 'Device 15e5' if=eth4 drv=ixgbe
unused=igb_uio,vfio-pci,uio_**pci_generic*
If I try PCI passthrough on 02:00.2 (eth0), it works fine. With any of the
remaining three interfaces, libvirt fails with this error:
*# virsh create vnf.xml error: Failed to create domain from vnf.xml error:
internal error: process exited while connecting to monitor:
2020-04-06T16:08:47.048266Z qemu-system-x86_64: -device
vfio-pci,host=02:00.1,id=**hostdev0,bus=pci.0,addr=0x5: vfio 0000:02:00.1:
failed to setup INTx fd: Operation not permitted*
The contents of vnf.xml are available here: https://pastebin.com/rT3RmAi5
This is what happened in *dmesg* when I tried to start the VM:
*[ 7305.371730] igb 0000:02:00.1: removed PHC on eth3 [ 7307.085618] ACPI
Warning: \_SB.PCI0.PEX2._PRT: Return Package has no elements (empty)
(20160831/nsprepkg-130) [ 7307.085717] pcieport 0000:00:0b.0: can't derive
routing for PCI INT B [ 7307.085719] vfio-pci 0000:02:00.1: PCI INT B: no
GSI [ 7307.369611] igb 0000:02:00.1: enabling device (0400 -> 0402) [
7307.369668] ACPI Warning: \_SB.PCI0.PEX2._PRT: Return Package has no
elements (empty) (20160831/nsprepkg-130) [ 7307.369764] pcieport
0000:00:0b.0: can't derive routing for PCI INT B [ 7307.369766] igb
0000:02:00.1: PCI INT B: no GSI [ 7307.426266] igb 0000:02:00.1: added PHC
on eth3 [ 7307.426269] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network
Connection [ 7307.426271] igb 0000:02:00.1: eth3: (PCIe:5.0Gb/s:Width x2)
50:9a:4c:ee:9f:b1 [ 7307.426350] igb 0000:02:00.1: eth3: PBA No: 106300-000
[ 7307.426352] igb 0000:02:00.1: Using MSI-X interrupts. 4 rx queue(s), 4
tx queue(s)*
These are all the messages related to that device in dmesg before I tried
to start the VM:
*# dmesg | grep 02:00.1 [ 0.185301] pci 0000:02:00.1: [8086:1521] type
00 class 0x020000 [ 0.185317] pci 0000:02:00.1: reg 0x10: [mem
0xdfd40000-0xdfd5ffff] [ 0.185334] pci 0000:02:00.1: reg 0x18: [io
0xd040-0xd05f] [ 0.185343] pci 0000:02:00.1: reg 0x1c: [mem
0xdfd88000-0xdfd8bfff] [ 0.185434] pci 0000:02:00.1: PME# supported from
D0 D3hot D3cold [ 0.185464] pci 0000:02:00.1: reg 0x184: [mem
0xdeea0000-0xdeea3fff 64bit pref] [ 0.185467] pci 0000:02:00.1: VF(n)
BAR0 space: [mem 0xdeea0000-0xdeebffff 64bit pref] (contains BAR0 for 8
VFs) [ 0.185486] pci 0000:02:00.1: reg 0x190: [mem 0xdee80000-0xdee83fff
64bit pref] [ 0.185488] pci 0000:02:00.1: VF(n) BAR3 space: [mem
0xdee80000-0xdee9ffff 64bit pref] (contains BAR3 for 8 VFs) [ 0.334021]
DMAR: Hardware identity mapping for device 0000:02:00.1 [ 0.334463]
iommu: Adding device 0000:02:00.1 to group 16 [ 0.398809] pci
0000:02:00.1: Signaling PME through PCIe PME interrupt [ 2.588049] igb
0000:02:00.1: PCI INT B: not connected [ 2.643900] igb 0000:02:00.1:
added PHC on eth1 [ 2.643903] igb 0000:02:00.1: Intel(R) Gigabit
Ethernet Network Connection [ 2.643905] igb 0000:02:00.1: eth1:
(PCIe:5.0Gb/s:Width x2) 50:9a:4c:ee:9f:b1 [ 2.643984] igb 0000:02:00.1:
eth1: PBA No: 106300-000 [ 2.643986] igb 0000:02:00.1: Using MSI-X
interrupts. 4 rx queue(s), 4 tx queue(s) [ 2.873544] igb 0000:02:00.1
rename3: renamed from eth1 [ 2.939352] igb 0000:02:00.1 eth3: renamed
from rename3*
In particular this looks suspicious: *igb 0000:02:00.1: PCI INT B: not
connected*
The full dmesg is available here: https://pastebin.com/kPbUAKCi
This is the PCI bus structure:
*# lspci -tv -[0000:00]-+-00.0 Intel Corporation Device 1980
+-04.0 Intel Corporation Device 19a1 +-05.0 Intel Corporation
Device 19a2 +-06.0-[01]----00.0 Intel Corporation Device 19e2
+-0b.0-[02-03]--+-00.0 Intel Corporation I350 Gigabit Network
Connection | +-00.1 Intel Corporation I350
Gigabit Network Connection | +-00.2 Intel
Corporation I350 Gigabit Network Connection |
\-00.3 Intel Corporation I350 Gigabit Network Connection
+-0f.0-[04]----00.0 Qualcomm Atheros QCA986x/988x 802.11ac Wireless
Network Adapter +-12.0 Intel Corporation DNV SMBus Contoller -
Host +-13.0 Intel Corporation DNV SATA Controller 0
+-15.0 Intel Corporation Device 19d0 +-16.0-[05-06]--+-00.0
Intel Corporation Device 15c4 | \-00.1 Intel
Corporation Device 15c4 +-17.0-[07-08]--+-00.0 Intel
Corporation Device 15e5 | \-00.1 Intel
Corporation Device 15e5 +-18.0 Intel Corporation Device 19d3
+-1c.0 Intel Corporation Device 19db +-1f.0 Intel
Corporation DNV LPC or eSPI +-1f.2 Intel Corporation Device
19de +-1f.4 Intel Corporation DNV SMBus controller
\-1f.5 Intel Corporation DNV SPI Controller*
By looking at lspci -v, there's something going on with the IRQ field
exactly in three devices I can't use in PCI passthrough ("IRQ -2147483648"):
*# lspci -v|grep -A1 I350 02:00.0 Ethernet controller: Intel Corporation
I350 Gigabit Network Connection (rev 01) Flags: bus master, fast
devsel, latency 0, IRQ -2147483648 -- 02:00.1 Ethernet controller: Intel
Corporation I350 Gigabit Network Connection (rev 01) Flags: bus master,
fast devsel, latency 0, IRQ -2147483648 -- 02:00.2 Ethernet controller:
Intel Corporation I350 Gigabit Network Connection (rev 01) Flags: bus
master, fast devsel, latency 0, IRQ 18 -- 02:00.3 Ethernet controller:
Intel Corporation I350 Gigabit Network Connection (rev 01) Flags: bus
master, fast devsel, latency 0, IRQ -2147483648*
Finally, every i350 interface has its own IOMMU group in
/sys/kernel/iommu_groups/.
The kernel I'm using in the host machine is 4.9.189 and my libvirt version
is 4.3.0.
Any thoughts on this?
Is there something I should enable in the BIOS or in the kernel to make
this work?
Thanks!
Regards,
Riccardo Ravaioli
4 years, 6 months
Attaching DVD-writer
by Andreas Thörn
I want to add my dvd and be able to burn dvd’s. I’ve tried every solution I could find, but nothing works.
In my xml i have:
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<source dev='/dev/sr0'/>
<target dev='sdb' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
In my guest (windows 10) it’s shown as "QEMU QEMU DVD-ROM” .. which of course means I can’t burn.
If anyone has a solution it would be this mailing list
Thanks
Andreas
4 years, 6 months
domain: how long is new xml in saved file
by Vincent Wu
Hi administrator,
I am a cloud compute developer. I need some help from you about libvirt.
I have a work to modify a image file which is saved by virDomainSave() or virDomainSaveFlags(). So virDomainSaveImageGetXMLDesc() and virDomainSaveImageDefineXML() are APIs I choosed to do.
Because I found a sentence: A save file can be inspected or modified slightly with virDomainSaveImageGetXMLDesc() and virDomainSaveImageDefineXML().
But an error is happened when I do like that. libvirt: QEMU Driver error: operation failed: new xml too large to fit in file.
I found that if I increase strlen(xml_old) to strlen(xml_new).
if (strlen(xml_new) - strlen(xml_old) <= 29) {
this is right;
}
but if (strlen(xml_new) - strlen(xml_old) >= 50) {
this is error;
}
But I don't choose to find an accurate number. I think this value will be affected by some factors.
For example: memory alignment, range safety or other rules.
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ strlen(xml_old) + free space +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ strlen(xml_new) +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
I really want to know how long free space I can use. Can you convert slightly to a precise number?
Thank you for taking so long to read my Email.
Looking forward to your reply.
ps:
OS: CentOS7.4
libvirt: 4.5.0
hypervisor: KVM
Sincerely,
Vincent Wu
4 years, 7 months
Not able to add pcie card to guest: Operation not permitted
by Mauricio Tavares
Let's say I have libvirt
[root@vmhost2 ~]# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 2.12.0
[root@vmhost2 ~]#
running on centos 8 and then I have this card
[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
<name>pci_0000_01_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
<parent>pci_0000_00_01_0</parent>
<driver>
<name>mlx4_core</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>1</bus>
<slot>0</slot>
<function>0</function>
<product id='0x1003'>MT27500 Family [ConnectX-3]</product>
<vendor id='0x15b3'>Mellanox Technologies</vendor>
<iommuGroup number='1'>
<address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</iommuGroup>
<pci-express>
<link validity='cap' port='8' speed='8' width='8'/>
<link validity='sta' speed='5' width='8'/>
</pci-express>
</capability>
</device>
which I added to the guest (arch='x86_64' machine='pc-q35-rhel7.6.0') as
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00'
function='0x0'/>
</hostdev>
When I try to start the guest I get the following error message:
[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-24T20:01:35.341020Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
Why is it telling me that is not permitted?
4 years, 7 months
