April 2020 - Users - libvirt List Archives

Re: USB-hotplugging fails with "failed to load cgroup BPF prog: Operation not permitted" on cgroups v2
by Pavel Hrdina 13 May '20

13 May '20

On Mon, Jan 20, 2020 at 09:00:15PM +0100, Pol Van Aubel wrote: > Hi, > > Quoting Pavel Hrdina (2020-01-20 14:29:36) > > On Sat, Jan 18, 2020 at 11:17:11PM +0100, Pol Van Aubel wrote: > > > Hi all, > > > > > > I've disabled cgroups v1 on my system with the kernel boot option > > > "systemd.unified_cgroup_hierarchy=1". Since doing so, USB hotplugging > > > fails to work, seemingly due to a permissions problem with BPF. Please > > > note that the technique I'm going to describe worked just fine for > > > hotplugging USB devices to running domains until this change. > > > Attaching / detaching USB devices when the domain is down still works as > > > expected. > > > > > > I get the same error when attaching a device in virt-manager, as I do > > > when running the following command: > > > > > > sudo virsh attach-device wenger /dev/stdin --persistent <<END > > > <hostdev mode='subsystem' type='usb' managed='yes'> > > > <source startupPolicy='optional'> > > > <vendor id='0x046d' /> > > > <product id='0xc215' /> > > > </source> > > > </hostdev> > > > END > > > > > > This returns > > > error: Failed to attach device from /dev/stdin > > > error: failed to load cgroup BPF prog: Operation not permitted > > > > > > > > > virt-manager returns basically the same error, but for completeness' > > > sake, here it is: > > > > > > failed to load cgroup BPF prog: Operation not permitted > > > > > > Traceback (most recent call last): > > > File "/usr/share/virt-manager/virtManager/addhardware.py", line 1327, in _add_device > > > self.vm.attach_device(dev) > > > File "/usr/share/virt-manager/virtManager/object/domain.py", line 920, in attach_device > > > self._backend.attachDevice(devxml) > > > File "/usr/lib/python3.8/site-packages/libvirt.py", line 590, in attachDevice > > > if ret == -1: raise libvirtError ('virDomainAttachDevice() failed', dom=self) > > > libvirt.libvirtError: failed to load cgroup BPF prog: Operation not permitted > > > > > > > > > Now, libvirtd is running as root, so I don't understand why any > > > operation on BPF programs is not permitted. I've dug into libvirt's code > > > a bit to see what is throwing this error and it boils down to > > > <https://github.com/libvirt/libvirt/blob/7d608469621a3fda72dff2a89308e68cc9f…> > > > and > > > <https://github.com/libvirt/libvirt/blob/02bf7cc68bfc76242f02d23e73cad36618f…> > > > but I have no clue what that syscall is doing, so that's where my > > > debugging capability basically ends. > > > > > > Maybe this is something as simple as setting the right ACL somewhere. I > > > haven't touched /etc/libvirt/qemu.conf except for setting nvram. There > > > *is* something about cgroup_device_acl there but afaict that's for > > > cgroups v1, when there was still a device cgroup controller. Any help > > > would be greatly appreciated. > > > > > > > > > Domain log files: > > > Upon execution of the above commands, nothing gets added to the domain > > > log in /var/log/qemu/wenger.log, so I've decided they're likely > > > irrelevant to the issue. Please ask for any additional info required. > > > > > > > > > System information: > > > Arch Linux, (normal) kernel 5.4.11 > > > libvirt 5.10.0 > > > qemu 4.2.0, using KVM. > > > Host system is x86_64 on an intel 5820k. > > > Guest system is probably irrelevant, but is Windows 10 on the same. > > > > > > > > > Possibly relevant kernel build options: > > > $ zgrep BPF /proc/config.gz > > > [22:55:52]: zgrep BPF /proc/config.gz > > > > > > CONFIG_CGROUP_BPF=y > > > CONFIG_BPF=y > > > CONFIG_BPF_SYSCALL=y > > > CONFIG_BPF_JIT_ALWAYS_ON=y > > > CONFIG_IPV6_SEG6_BPF=y > > > CONFIG_NETFILTER_XT_MATCH_BPF=m > > > # CONFIG_BPFILTER is not set > > > CONFIG_NET_CLS_BPF=m > > > CONFIG_NET_ACT_BPF=m > > > CONFIG_BPF_JIT=y > > > CONFIG_BPF_STREAM_PARSER=y > > > CONFIG_LWTUNNEL_BPF=y > > > CONFIG_HAVE_EBPF_JIT=y > > > CONFIG_BPF_EVENTS=y > > > # CONFIG_BPF_KPROBE_OVERRIDE is not set > > > # CONFIG_TEST_BPF is not set > > > > Hi > > > > I've installed clean archlinux to try this out and it works as expected, > > I'm able to attach USB device into a VM. > > > > My system env is mostly the same as yours except for kernel version: > > > > kernel 5.4.13 > > libvirt 5.10.0 > > qemu 4.2.0, using KVM. > > > > Please enable libvirt debug logs [1] and share the output with us. > > I've updated to 5.4.13 and created a barebones VM without storage to > reproduce the behaviour. libvirtd debug logs are attached. There appear > to be two BPF failures of the same BPF program (?). The first is on line > 23209, which appears to be part of machine startup, and which I don't > actually notice. The second one is where I manually add the USB device, > on line 30599. > > Thanks, Thanks for the logs, but it did not help to figure out where the issue is. I was hoping to see some error output from the syscall but the line that should contain it is empty: 2020-01-20 19:47:15.589+0000: 8579: debug : virBPFLoadProg:78 : Can you please check system logs and output of dmesg? I've managed to run into this article [1] that explains that even if you have all permissions and no SELinux you can still be blocked by something called kernel_lockdown and it should appear in dmesg. Pavel [1] <https://gehrcke.de/2019/09/running-an-ebpf-program-may-require-lifting-the-…>

2 3

Libvirt APIs for creating virtual networks
by Santhosh Kumar Gunturu 08 May '20

08 May '20

I am using the Libvirt APIs to create the virtual networks to use the dnsmasq capabilities. I want to understand the APIs how to set the lease time in XML to control the dnsmasq lease time. Also, I need some DHCP statistics on the virtual network. Is there any API available to fetch the statistics ? Please let me know if any ? Thanks & Regards Santhosh Kumar Gunturu

3 6

sync guest time
by Miguel Duarte de Mora Barroso 01 May '20

01 May '20

Hi, I'm seeing the following issue when attempting to update the guest's clock on a running fc32 guest (using guest agent): ``` [root@virt-launcher-vmi-masquerade-mh2xm /]# virsh domtime 1 --pretty Time: 2020-04-30 23:27:29 [root@virt-launcher-vmi-masquerade-mh2xm /]# virsh domtime 1 --sync error: internal error: unable to execute QEMU agent command 'guest-set-time': hwclock failed to set hardware clock to system time # now, this one works. [root@virt-launcher-vmi-masquerade-mh2xm /]# virsh domtime 1 --now [root@virt-launcher-vmi-masquerade-44v2x /]# virsh domtime 1 --pretty Time: 2020-04-30 11:15:45 ``` This is the simplest reproducer I could come with; the original issue is this call to libvirt's `setTIme` in [0]. Is there any workaround I could try ? Am I doing something wrong here ? [0] - https://github.com/kubevirt/kubevirt/blob/6bb516148ce4c29825ae74f473e022075…

2 3

Re: [Query] Facing issue in build libvirt6.0 on ubuntu 1804
by Michal Privoznik 30 Apr '20

30 Apr '20

[Please keep the list CCed] On 4/30/20 12:02 PM, Ramesh B wrote: > Hi Michal, > > Thanks for quick response. > > Ubuntu: 18.04.3 LTS > QEMU emulator version 4.2.0 > libvirtd (libvirt) 4.0.0 > > I have lunched the guest os using QEMU. > I would like to use the virsh commands to get the status / control of > guest OS. > when i run any virsh commands not getting any response from QEMU. > When i tried to explore, found the solution saying use of QEMU monitor > will help. > added the below command while lunch, > -chardev > socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-freebsd11.0/monitor.sock,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control > > but result in below mentioed error: > Failed to bind socket to > /var/lib/libvirt/qemu/domain-freebsd11.0/monitor.sock > > Then thought of building libvirt 6.0 version, will help to solve the > problem. Ah, so you wanted to use 'virsh qemu-attach' to attach libvirt to a running qemu. Yeah, it needs a monitor. However, the functionality was not maintained and probably not functional anyway so we've removed in the 5.5.0 release. > > Please guide me to solve this problem. > > Regards, > Babu B > But what's interesting is that you are unable to build. Which package is providing the rst2* binaries? Are their up to date? Michal

2 5

[Query] Facing issue in build libvirt6.0 on ubuntu 1804
by Ramesh B 30 Apr '20

30 Apr '20

Hi all, I am trying to build libvirt 6.0 on ubuntu OS 18.04. Facing one issue libvirt source compilation. ============================================ <stdin>:215: (ERROR/3) Error in "code-block" directive: 1 argument(s) required, 0 supplied. .. code-block:: help [command-or-group] <stdin>:229: (ERROR/3) Error in "code-block" directive: 1 argument(s) required, 0 supplied. .. code-block:: =========================================== *Source:* https://libvirt.org/sources/libvirt-6.0.0.tar.xz *Build Command*: $ mkdir build && cd build $ ../configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var $ make $ sudo make install Need help in solving the build error. is it possible to install libvirt package without build if yes please share the command/procedure/link. Thanks, Ramesh

3 6

why libvirtd can not list the running domain?
by 崔涛的个人邮箱 30 Apr '20

30 Apr '20

1. I run a user domain like this: [root@kvm1 ~]# qemu-system-x86_64 -enable-kvm -m 1G -smp 4 /images/test1.img WARNING: Image format was not specified for '/images/test1.img' and probing guessed raw. Automatically detecting the format is dangerous for raw images, write operations on block 0 will be restricted. Specify the 'raw' format explicitly to remove the restrictions. VNC server running on ::1:5900 2. Then, I install and using my user domain. 3. Then, I login to libvirtd on my hosts,but can not get the user domain using list command. [root@kvm1 ~]# virsh -c qemu:///system Welcome to virsh, the virtualization interactive terminal. Type: 'help' for help with commands 'quit' to quit virsh # virsh # virsh # list Id Name State ---------------------------------------------------- virsh #

2 2

"failed to setup INTx fd: Operation not permitted" error when using PCI passthrough
by Riccardo Ravaioli 30 Apr '20

30 Apr '20

Hi, I'm on a Dell VEP 1405 running Debian 9.11 and I'm running a few tests with various interfaces given in PCI passthrough to a qemu/KVM Virtual Machine also running Debian 9.11. I noticed that only one of the four I350 network controllers can be used in PCI passthrough. The available interfaces are: *# dpdk-devbind.py --status Network devices using kernel driver ==============================* *===== 0000:02:00.0 'I350 Gigabit Network Connection 1521' if=eth2 drv=igb unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:02:00.1 'I350 Gigabit Network Connection 1521' if=eth3 drv=igb unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:02:00.2 'I350 Gigabit Network Connection 1521' if=eth0 drv=igb unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:02:00.3 'I350 Gigabit Network Connection 1521' if=eth1 drv=igb unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:04:00.0 'QCA986x/988x 802.11ac Wireless Network Adapter 003c' if= drv=ath10k_pci unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:05:00.0 'Device 15c4' if=eth7 drv=ixgbe unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:05:00.1 'Device 15c4' if=eth6 drv=ixgbe unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:07:00.0 'Device 15e5' if=eth5 drv=ixgbe unused=igb_uio,vfio-pci,uio_* *pci_generic 0000:07:00.1 'Device 15e5' if=eth4 drv=ixgbe unused=igb_uio,vfio-pci,uio_**pci_generic* If I try PCI passthrough on 02:00.2 (eth0), it works fine. With any of the remaining three interfaces, libvirt fails with this error: *# virsh create vnf.xml error: Failed to create domain from vnf.xml error: internal error: process exited while connecting to monitor: 2020-04-06T16:08:47.048266Z qemu-system-x86_64: -device vfio-pci,host=02:00.1,id=**hostdev0,bus=pci.0,addr=0x5: vfio 0000:02:00.1: failed to setup INTx fd: Operation not permitted* The contents of vnf.xml are available here: https://pastebin.com/rT3RmAi5 This is what happened in *dmesg* when I tried to start the VM: *[ 7305.371730] igb 0000:02:00.1: removed PHC on eth3 [ 7307.085618] ACPI Warning: \_SB.PCI0.PEX2._PRT: Return Package has no elements (empty) (20160831/nsprepkg-130) [ 7307.085717] pcieport 0000:00:0b.0: can't derive routing for PCI INT B [ 7307.085719] vfio-pci 0000:02:00.1: PCI INT B: no GSI [ 7307.369611] igb 0000:02:00.1: enabling device (0400 -> 0402) [ 7307.369668] ACPI Warning: \_SB.PCI0.PEX2._PRT: Return Package has no elements (empty) (20160831/nsprepkg-130) [ 7307.369764] pcieport 0000:00:0b.0: can't derive routing for PCI INT B [ 7307.369766] igb 0000:02:00.1: PCI INT B: no GSI [ 7307.426266] igb 0000:02:00.1: added PHC on eth3 [ 7307.426269] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection [ 7307.426271] igb 0000:02:00.1: eth3: (PCIe:5.0Gb/s:Width x2) 50:9a:4c:ee:9f:b1 [ 7307.426350] igb 0000:02:00.1: eth3: PBA No: 106300-000 [ 7307.426352] igb 0000:02:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)* These are all the messages related to that device in dmesg before I tried to start the VM: *# dmesg | grep 02:00.1 [ 0.185301] pci 0000:02:00.1: [8086:1521] type 00 class 0x020000 [ 0.185317] pci 0000:02:00.1: reg 0x10: [mem 0xdfd40000-0xdfd5ffff] [ 0.185334] pci 0000:02:00.1: reg 0x18: [io 0xd040-0xd05f] [ 0.185343] pci 0000:02:00.1: reg 0x1c: [mem 0xdfd88000-0xdfd8bfff] [ 0.185434] pci 0000:02:00.1: PME# supported from D0 D3hot D3cold [ 0.185464] pci 0000:02:00.1: reg 0x184: [mem 0xdeea0000-0xdeea3fff 64bit pref] [ 0.185467] pci 0000:02:00.1: VF(n) BAR0 space: [mem 0xdeea0000-0xdeebffff 64bit pref] (contains BAR0 for 8 VFs) [ 0.185486] pci 0000:02:00.1: reg 0x190: [mem 0xdee80000-0xdee83fff 64bit pref] [ 0.185488] pci 0000:02:00.1: VF(n) BAR3 space: [mem 0xdee80000-0xdee9ffff 64bit pref] (contains BAR3 for 8 VFs) [ 0.334021] DMAR: Hardware identity mapping for device 0000:02:00.1 [ 0.334463] iommu: Adding device 0000:02:00.1 to group 16 [ 0.398809] pci 0000:02:00.1: Signaling PME through PCIe PME interrupt [ 2.588049] igb 0000:02:00.1: PCI INT B: not connected [ 2.643900] igb 0000:02:00.1: added PHC on eth1 [ 2.643903] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection [ 2.643905] igb 0000:02:00.1: eth1: (PCIe:5.0Gb/s:Width x2) 50:9a:4c:ee:9f:b1 [ 2.643984] igb 0000:02:00.1: eth1: PBA No: 106300-000 [ 2.643986] igb 0000:02:00.1: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s) [ 2.873544] igb 0000:02:00.1 rename3: renamed from eth1 [ 2.939352] igb 0000:02:00.1 eth3: renamed from rename3* In particular this looks suspicious: *igb 0000:02:00.1: PCI INT B: not connected* The full dmesg is available here: https://pastebin.com/kPbUAKCi This is the PCI bus structure: *# lspci -tv -[0000:00]-+-00.0 Intel Corporation Device 1980 +-04.0 Intel Corporation Device 19a1 +-05.0 Intel Corporation Device 19a2 +-06.0-[01]----00.0 Intel Corporation Device 19e2 +-0b.0-[02-03]--+-00.0 Intel Corporation I350 Gigabit Network Connection | +-00.1 Intel Corporation I350 Gigabit Network Connection | +-00.2 Intel Corporation I350 Gigabit Network Connection | \-00.3 Intel Corporation I350 Gigabit Network Connection +-0f.0-[04]----00.0 Qualcomm Atheros QCA986x/988x 802.11ac Wireless Network Adapter +-12.0 Intel Corporation DNV SMBus Contoller - Host +-13.0 Intel Corporation DNV SATA Controller 0 +-15.0 Intel Corporation Device 19d0 +-16.0-[05-06]--+-00.0 Intel Corporation Device 15c4 | \-00.1 Intel Corporation Device 15c4 +-17.0-[07-08]--+-00.0 Intel Corporation Device 15e5 | \-00.1 Intel Corporation Device 15e5 +-18.0 Intel Corporation Device 19d3 +-1c.0 Intel Corporation Device 19db +-1f.0 Intel Corporation DNV LPC or eSPI +-1f.2 Intel Corporation Device 19de +-1f.4 Intel Corporation DNV SMBus controller \-1f.5 Intel Corporation DNV SPI Controller* By looking at lspci -v, there's something going on with the IRQ field exactly in three devices I can't use in PCI passthrough ("IRQ -2147483648"): *# lspci -v|grep -A1 I350 02:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Flags: bus master, fast devsel, latency 0, IRQ -2147483648 -- 02:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Flags: bus master, fast devsel, latency 0, IRQ -2147483648 -- 02:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Flags: bus master, fast devsel, latency 0, IRQ 18 -- 02:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) Flags: bus master, fast devsel, latency 0, IRQ -2147483648* Finally, every i350 interface has its own IOMMU group in /sys/kernel/iommu_groups/. The kernel I'm using in the host machine is 4.9.189 and my libvirt version is 4.3.0. Any thoughts on this? Is there something I should enable in the BIOS or in the kernel to make this work? Thanks! Regards, Riccardo Ravaioli

1 1

Attaching DVD-writer
by Andreas Thörn 28 Apr '20

28 Apr '20

I want to add my dvd and be able to burn dvd’s. I’ve tried every solution I could find, but nothing works. In my xml i have: <disk type='block' device='cdrom'> <driver name='qemu' type='raw'/> <source dev='/dev/sr0'/> <target dev='sdb' bus='sata'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> In my guest (windows 10) it’s shown as "QEMU QEMU DVD-ROM” .. which of course means I can’t burn. If anyone has a solution it would be this mailing list Thanks Andreas

4 5

domain: how long is new xml in saved file
by Vincent Wu 27 Apr '20

27 Apr '20

Hi administrator, I am a cloud compute developer. I need some help from you about libvirt. I have a work to modify a image file which is saved by virDomainSave() or virDomainSaveFlags(). So virDomainSaveImageGetXMLDesc() and virDomainSaveImageDefineXML() are APIs I choosed to do. Because I found a sentence: A save file can be inspected or modified slightly with virDomainSaveImageGetXMLDesc() and virDomainSaveImageDefineXML(). But an error is happened when I do like that.  libvirt: QEMU Driver error: operation failed: new xml too large to fit in file. I found that if I increase strlen(xml_old) to strlen(xml_new). if (strlen(xml_new) - strlen(xml_old) <= 29) { this is right; } but if (strlen(xml_new) - strlen(xml_old) >= 50) { this is error; } But I don't choose to find an accurate number.  I think this value will be affected by some factors. For example: memory alignment, range safety or other rules. ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +                                                           strlen(xml_old)                                                  +  free space  + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ +                                                             strlen(xml_new)                                                                     + ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ I really want to know how long free space I can use. Can you convert slightly to a precise number? Thank you for taking so long to read my Email. Looking forward to your reply. ps: OS: CentOS7.4 libvirt: 4.5.0 hypervisor: KVM Sincerely,  Vincent Wu

5 5

Not able to add pcie card to guest: Operation not permitted
by Mauricio Tavares 25 Apr '20

25 Apr '20

Let's say I have libvirt [root@vmhost2 ~]# virsh version Compiled against library: libvirt 4.5.0 Using library: libvirt 4.5.0 Using API: QEMU 4.5.0 Running hypervisor: QEMU 2.12.0 [root@vmhost2 ~]# running on centos 8 and then I have this card [root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0 <device> <name>pci_0000_01_00_0</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path> <parent>pci_0000_00_01_0</parent> <driver> <name>mlx4_core</name> </driver> <capability type='pci'> <domain>0</domain> <bus>1</bus> <slot>0</slot> <function>0</function> <product id='0x1003'>MT27500 Family [ConnectX-3]</product> <vendor id='0x15b3'>Mellanox Technologies</vendor> <iommuGroup number='1'> <address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </iommuGroup> <pci-express> <link validity='cap' port='8' speed='8' width='8'/> <link validity='sta' speed='5' width='8'/> </pci-express> </capability> </device> which I added to the guest (arch='x86_64' machine='pc-q35-rhel7.6.0') as <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> When I try to start the guest I get the following error message: [root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-24T20:01:35.341020Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted [root@vmhost2 ~]# Why is it telling me that is not permitted?

2 3