Not able to add pcie card to guest: Operation not permitted
Let's say I have libvirt [root@vmhost2 ~]# virsh version Compiled against library: libvirt 4.5.0 Using library: libvirt 4.5.0 Using API: QEMU 4.5.0 Running hypervisor: QEMU 2.12.0 [root@vmhost2 ~]# running on centos 8 and then I have this card [root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0 <device> <name>pci_0000_01_00_0</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path> <parent>pci_0000_00_01_0</parent> <driver> <name>mlx4_core</name> </driver> <capability type='pci'> <domain>0</domain> <bus>1</bus> <slot>0</slot> <function>0</function> <product id='0x1003'>MT27500 Family [ConnectX-3]</product> <vendor id='0x15b3'>Mellanox Technologies</vendor> <iommuGroup number='1'> <address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </iommuGroup> <pci-express> <link validity='cap' port='8' speed='8' width='8'/> <link validity='sta' speed='5' width='8'/> </pci-express> </capability> </device> which I added to the guest (arch='x86_64' machine='pc-q35-rhel7.6.0') as <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/> </hostdev> When I try to start the guest I get the following error message: [root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-24T20:01:35.341020Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted [root@vmhost2 ~]# Why is it telling me that is not permitted?
On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel@gmail.com> wrote:
Let's say I have libvirt
[root@vmhost2 ~]# virsh version [...]
Running hypervisor: QEMU 2.12.0
[root@vmhost2 ~]# [...]
When I try to start the guest I get the following error message:
[root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-24T20:01:35.341020Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
Why is it telling me that is not permitted?
The guest will be running as qemu on the host. Does qemu have appropriate
permissions in the host, and does that include in any hardening like SElinux that you're running? Cheers, - Peter
On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther <peter.crowther@melandra.com> wrote:
On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel@gmail.com> wrote:
Let's say I have libvirt
[root@vmhost2 ~]# virsh version [...]
Running hypervisor: QEMU 2.12.0 [root@vmhost2 ~]# [...]
When I try to start the guest I get the following error message:
[root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-24T20:01:35.341020Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
Why is it telling me that is not permitted?
The guest will be running as qemu on the host. Does qemu have appropriate permissions in the host, and does that include in any hardening like SElinux that you're running?
I tried with selinux in permissive mode to see if it made a difference. Not much. [root@vmhost2 ~]# getenforce Permissive [root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-25T00:43:36.621246Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted [root@vmhost2 ~]# For the fun of it, I swapped that card with another one (same speed, number of ports, diff brand), so it is on th every sam epci slot: [root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0 <device> <name>pci_0000_01_00_0</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path> <parent>pci_0000_00_01_0</parent> <driver> <name>vfio-pci</name> </driver> <capability type='pci'> <domain>0</domain> <bus>1</bus> <slot>0</slot> <function>0</function> <product id='0x4000' /> <vendor id='0x19ee'>Netronome Systems, Inc.</vendor> <capability type='virt_functions' maxCount='64'/> <iommuGroup number='1'> <address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='8' width='8'/> <link validity='sta' speed='2.5' width='8'/> </pci-express> </capability> </device> [root@vmhost2 ~]# And it starts without an issue: [root@vmhost2 ~]# virsh start testfedora Domain testfedora started [root@vmhost2 ~]# Inside the guest: [root@testfedora ~]# dmesg |grep -i netronome [ 12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems [ 12.335036] nfp 0000:07:00.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe [root@testfedora ~]# so I do not know what is going on.
Cheers,
- Peter
On Fri, Apr 24, 2020 at 9:30 PM Mauricio Tavares <raubvogel@gmail.com> wrote:
On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther <peter.crowther@melandra.com> wrote:
On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel@gmail.com> wrote:
Let's say I have libvirt
[root@vmhost2 ~]# virsh version [...]
Running hypervisor: QEMU 2.12.0 [root@vmhost2 ~]# [...]
When I try to start the guest I get the following error message:
[root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-24T20:01:35.341020Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
Why is it telling me that is not permitted?
The guest will be running as qemu on the host. Does qemu have appropriate permissions in the host, and does that include in any hardening like SElinux that you're running?
I tried with selinux in permissive mode to see if it made a difference. Not much.
[root@vmhost2 ~]# getenforce Permissive [root@vmhost2 ~]# virsh start testfedora error: Failed to start domain testfedora error: internal error: qemu unexpectedly closed the monitor: 2020-04-25T00:43:36.621246Z qemu-kvm: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error: 0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
For the fun of it, I swapped that card with another one (same speed, number of ports, diff brand), so it is on th every sam epci slot:
[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0 <device> <name>pci_0000_01_00_0</name> <path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path> <parent>pci_0000_00_01_0</parent> <driver> <name>vfio-pci</name> </driver> <capability type='pci'> <domain>0</domain> <bus>1</bus> <slot>0</slot> <function>0</function> <product id='0x4000' /> <vendor id='0x19ee'>Netronome Systems, Inc.</vendor> <capability type='virt_functions' maxCount='64'/> <iommuGroup number='1'> <address domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </iommuGroup> <pci-express> <link validity='cap' port='0' speed='8' width='8'/> <link validity='sta' speed='2.5' width='8'/> </pci-express> </capability> </device>
[root@vmhost2 ~]#
And it starts without an issue:
[root@vmhost2 ~]# virsh start testfedora Domain testfedora started
[root@vmhost2 ~]#
Inside the guest:
[root@testfedora ~]# dmesg |grep -i netronome [ 12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems [ 12.335036] nfp 0000:07:00.0: Netronome Flow Processor NFP4000/NFP5000/NFP6000 PCIe Card Probe [root@testfedora ~]#
so I do not know what is going on.
My last statement can be translated to "I am doing PCI passthrough, which to me means I am passing the entire card -- whatever it is -- to the guest." Just for the sake of argument, I also created a centos8 guest and had the same outcome. Who is doing the pci passthrough thingie: libvirt or qemu? Just want to see where I should put my efforts on. I have the Also, I decided for the fun of it to create a docker container in the same host and pass the card to it. That seemed to have worked better [root@ce3077ee015c /]# ls /sys/bus/pci/devices/0000\:01\:00.0/ aer_dev_correctable infiniband_srp/ pools aer_dev_fatal infiniband_verbs/ power/ aer_dev_nonfatal iommu/ remove ari_enabled iommu_group/ rescan broken_parity_status irq reset class local_cpulist resource config local_cpus resource0 consistent_dma_mask_bits max_link_speed resource2 current_link_speed max_link_width resource2_wc current_link_width mlx4_port1 revision d3cold_allowed mlx4_port1_mtu rom device mlx4_port2 subsystem/ dma_mask_bits mlx4_port2_mtu subsystem_device driver/ modalias subsystem_vendor driver_override msi_bus uevent enable msi_irqs/ vendor infiniband/ net/ vpd infiniband_mad/ numa_node [root@ce3077ee015c /]# cat /sys/bus/pci/devices/0000\:01\:00.0/device 0x1003 [root@ce3077ee015c /]# Of course the test is to configure card to use or to program it, but I have so far been more successful. Which makes me even more confused.
Cheers,
- Peter
participants (2)
-
Mauricio Tavares -
Peter Crowther