3:09 p.m.
Let's say I have libvirt
[root@vmhost2 ~]# virsh version
Compiled against library: libvirt 4.5.0
Using library: libvirt 4.5.0
Using API: QEMU 4.5.0
Running hypervisor: QEMU 2.12.0
[root@vmhost2 ~]#
running on centos 8 and then I have this card
[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
<name>pci_0000_01_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
<parent>pci_0000_00_01_0</parent>
<driver>
<name>mlx4_core</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>1</bus>
<slot>0</slot>
<function>0</function>
<product id='0x1003'>MT27500 Family [ConnectX-3]</product>
<vendor id='0x15b3'>Mellanox Technologies</vendor>
<iommuGroup number='1'>
<address domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</iommuGroup>
<pci-express>
<link validity='cap' port='8' speed='8'
width='8'/>
<link validity='sta' speed='5' width='8'/>
</pci-express>
</capability>
</device>
which I added to the guest (arch='x86_64' machine='pc-q35-rhel7.6.0') as
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x08'
slot='0x00'
function='0x0'/>
</hostdev>
When I try to start the guest I get the following error message:
[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-24T20:01:35.341020Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
Why is it telling me that is not permitted?
3:34 p.m.
On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel(a)gmail.com> wrote:
Let's say I have libvirt
[root@vmhost2 ~]# virsh version
[...]
Running hypervisor: QEMU 2.12.0
[root@vmhost2 ~]#
[...]
When I try to start the guest I get the following error message:
[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-24T20:01:35.341020Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
Why is it telling me that is not permitted?
The guest will be running as qemu on the host. Does qemu have appropriate
permissions in the host, and does that include in any hardening like
SElinux that you're running?
Cheers,
- Peter
8:30 p.m.
On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther
<peter.crowther(a)melandra.com> wrote:
On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel(a)gmail.com> wrote:
>
> Let's say I have libvirt
>
> [root@vmhost2 ~]# virsh version
> [...]
>
> Running hypervisor: QEMU 2.12.0
> [root@vmhost2 ~]#
> [...]
When I try to start the guest I get the following error message:
>
>
> [root@vmhost2 ~]# virsh start testfedora
> error: Failed to start domain testfedora
> error: internal error: qemu unexpectedly closed the monitor:
> 2020-04-24T20:01:35.341020Z qemu-kvm: -device
> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
> 0000:01:00.0: failed to setup INTx fd: Operation not permitted
>
> [root@vmhost2 ~]#
>
> Why is it telling me that is not permitted?
>
The guest will be running as qemu on the host. Does qemu have appropriate permissions in
the host, and does that include in any hardening like SElinux that you're running?
I tried with selinux in permissive mode to see if it made a
difference. Not much.
[root@vmhost2 ~]# getenforce
Permissive
[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-25T00:43:36.621246Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
For the fun of it, I swapped that card with another one (same speed,
number of ports, diff brand), so it is on th every sam epci slot:
[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
<name>pci_0000_01_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
<parent>pci_0000_00_01_0</parent>
<driver>
<name>vfio-pci</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>1</bus>
<slot>0</slot>
<function>0</function>
<product id='0x4000' />
<vendor id='0x19ee'>Netronome Systems, Inc.</vendor>
<capability type='virt_functions' maxCount='64'/>
<iommuGroup number='1'>
<address domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</iommuGroup>
<pci-express>
<link validity='cap' port='0' speed='8'
width='8'/>
<link validity='sta' speed='2.5' width='8'/>
</pci-express>
</capability>
</device>
[root@vmhost2 ~]#
And it starts without an issue:
[root@vmhost2 ~]# virsh start testfedora
Domain testfedora started
[root@vmhost2 ~]#
Inside the guest:
[root@testfedora ~]# dmesg |grep -i netronome
[ 12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
[ 12.335036] nfp 0000:07:00.0: Netronome Flow Processor
NFP4000/NFP5000/NFP6000 PCIe Card Probe
[root@testfedora ~]#
so I do not know what is going on.
Cheers,
- Peter
6:27 a.m.
On Fri, Apr 24, 2020 at 9:30 PM Mauricio Tavares <raubvogel(a)gmail.com> wrote:
On Fri, Apr 24, 2020 at 4:35 PM Peter Crowther
<peter.crowther(a)melandra.com> wrote:
>
> On Fri, 24 Apr 2020 at 21:10, Mauricio Tavares <raubvogel(a)gmail.com> wrote:
>>
>> Let's say I have libvirt
>>
>> [root@vmhost2 ~]# virsh version
>> [...]
>>
>> Running hypervisor: QEMU 2.12.0
>> [root@vmhost2 ~]#
>> [...]
>
> When I try to start the guest I get the following error message:
>>
>>
>> [root@vmhost2 ~]# virsh start testfedora
>> error: Failed to start domain testfedora
>> error: internal error: qemu unexpectedly closed the monitor:
>> 2020-04-24T20:01:35.341020Z qemu-kvm: -device
>> vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
>> 0000:01:00.0: failed to setup INTx fd: Operation not permitted
>>
>> [root@vmhost2 ~]#
>>
>> Why is it telling me that is not permitted?
>>
> The guest will be running as qemu on the host. Does qemu have appropriate
permissions in the host, and does that include in any hardening like SElinux that
you're running?
>
I tried with selinux in permissive mode to see if it made a
difference. Not much.
[root@vmhost2 ~]# getenforce
Permissive
[root@vmhost2 ~]# virsh start testfedora
error: Failed to start domain testfedora
error: internal error: qemu unexpectedly closed the monitor:
2020-04-25T00:43:36.621246Z qemu-kvm: -device
vfio-pci,host=01:00.0,id=hostdev0,bus=pci.8,addr=0x0: vfio error:
0000:01:00.0: failed to setup INTx fd: Operation not permitted
[root@vmhost2 ~]#
For the fun of it, I swapped that card with another one (same speed,
number of ports, diff brand), so it is on th every sam epci slot:
[root@vmhost2 ~]# virsh nodedev-dumpxml pci_0000_01_00_0
<device>
<name>pci_0000_01_00_0</name>
<path>/sys/devices/pci0000:00/0000:00:01.0/0000:01:00.0</path>
<parent>pci_0000_00_01_0</parent>
<driver>
<name>vfio-pci</name>
</driver>
<capability type='pci'>
<domain>0</domain>
<bus>1</bus>
<slot>0</slot>
<function>0</function>
<product id='0x4000' />
<vendor id='0x19ee'>Netronome Systems, Inc.</vendor>
<capability type='virt_functions' maxCount='64'/>
<iommuGroup number='1'>
<address domain='0x0000' bus='0x00' slot='0x01'
function='0x0'/>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</iommuGroup>
<pci-express>
<link validity='cap' port='0' speed='8'
width='8'/>
<link validity='sta' speed='2.5' width='8'/>
</pci-express>
</capability>
</device>
[root@vmhost2 ~]#
And it starts without an issue:
[root@vmhost2 ~]# virsh start testfedora
Domain testfedora started
[root@vmhost2 ~]#
Inside the guest:
[root@testfedora ~]# dmesg |grep -i netronome
[ 12.327316] nfp: NFP PCIe Driver, Copyright (C) 2014-2017 Netronome Systems
[ 12.335036] nfp 0000:07:00.0: Netronome Flow Processor
NFP4000/NFP5000/NFP6000 PCIe Card Probe
[root@testfedora ~]#
so I do not know what is going on.
My last statement can be translated to "I am doing PCI
passthrough, which to me means I am passing the entire card --
whatever it is -- to the guest." Just for the sake of argument, I also
created a centos8 guest and had the same outcome. Who is doing the pci
passthrough thingie: libvirt or qemu? Just want to see where I should
put my efforts on.
I have the
Also, I decided for the fun of it to create a docker container in the
same host and pass the card to it. That seemed to have worked better
[root@ce3077ee015c /]# ls /sys/bus/pci/devices/0000\:01\:00.0/
aer_dev_correctable infiniband_srp/ pools
aer_dev_fatal infiniband_verbs/ power/
aer_dev_nonfatal iommu/ remove
ari_enabled iommu_group/ rescan
broken_parity_status irq reset
class local_cpulist resource
config local_cpus resource0
consistent_dma_mask_bits max_link_speed resource2
current_link_speed max_link_width resource2_wc
current_link_width mlx4_port1 revision
d3cold_allowed mlx4_port1_mtu rom
device mlx4_port2 subsystem/
dma_mask_bits mlx4_port2_mtu subsystem_device
driver/ modalias subsystem_vendor
driver_override msi_bus uevent
enable msi_irqs/ vendor
infiniband/ net/ vpd
infiniband_mad/ numa_node
[root@ce3077ee015c /]# cat /sys/bus/pci/devices/0000\:01\:00.0/device
0x1003
[root@ce3077ee015c /]#
Of course the test is to configure card to use or to program it, but I
have so far been more successful. Which makes me even more confused.
> Cheers,
>
> - Peter
>
1672
days inactive
1673
days old
3 comments
2 participants
participants (2)
-
Mauricio Tavares -
Peter Crowther