[libvirt] vfio: Error: Failed to setup INTx fd: Device or resource busy when PF pass-thought duel port NIC

9 Aug 2016

      Hi,

I am trying to PF pass-thought  with Mellanox NIC CX4 with duel port card. 

I know the currently it doesn't support ACS, therefore we can't allocate 2 PF to 2 guest. (both pci device are in the same IOMMU group)

But if I understand accoriding to [1] this currently there should be a problem to pass-thought both PFs to the same guest. 

before we boot the guest I unbind the pci device with the following commands. 
echo 0000:03:00.00>  /sys/bus/pci/devices/0000:03:00.00/driver/unbind
echo 0000:03:00.01 >  /sys/bus/pci/devices/0000:03:00.1/driver/unbind

I tried it worked and it was working on one  environment , but on other environment  we are getting the
following error  vfio: Error: Failed to setup INTx fd: Device or resource busy. See detail error [2].

Can anyone explain when/why this error occurs? And how can we fix it? 

[1]  - http://vfio.blogspot.co.il/2014/08/iommu-groups-inside-and-out.html 

[2] - Nova-compute.log error:

        2016-08-08 21:20:14.213 20236 ERROR nova.virt.libvirt.guest [req-5a6532d4-317f-4165-855e-37d8b36c71ec 295fe519b3864de5894fa0219362c26f 2e4cb36a0a3645e7b924deecea3bd210 - - -] Error launching a defined domain with XML: <domain type='kvm'>
  <name>instance-00000040</name>
  <uuid>74abfd25-aab6-4362-ab28-49c3bbd5e508</uuid>
  <metadata>
    <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
      <nova:package version="12.0.3-1.el7ost"/>
      <nova:name>Test</nova:name>
      <nova:creationTime>2016-08-08 21:20:08</nova:creationTime>
      <nova:flavor name="Mellanox_PT_2">
        <nova:memory>16384</nova:memory>
        <nova:disk>0</nova:disk>
        <nova:swap>0</nova:swap>
        <nova:ephemeral>0</nova:ephemeral>
        <nova:vcpus>16</nova:vcpus>
      </nova:flavor>
      <nova:owner>
        <nova:user uuid="295fe519b3864de5894fa0219362c26f">admin</nova:user>
        <nova:project uuid="2e4cb36a0a3645e7b924deecea3bd210">openstack</nova:project>
      </nova:owner>
      <nova:root type="image" uuid="49d20d6e-e247-43ff-aeec-84ba53d6b439"/>
    </nova:instance>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static' cpuset='3-23,27-47'>16</vcpu>
 <cputune>
    <shares>16384</shares>
  </cputune>
  <sysinfo type='smbios'>
    <system>
      <entry name='manufacturer'>Red Hat</entry>
      <entry name='product'>OpenStack Compute</entry>
      <entry name='version'>12.0.3-1.el7ost</entry>
      <entry name='serial'>d336b2a1-4abd-4602-9bc4-4aeb20f778e7</entry>
      <entry name='uuid'>74abfd25-aab6-4362-ab28-49c3bbd5e508</entry>
      <entry name='family'>Virtual Machine</entry>
    </system>
  </sysinfo>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <boot dev='hd'/>
    <smbios mode='sysinfo'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
    <topology sockets='16' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/disk'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='fa:16:3e:14:ec:96'/>
      <source bridge='qbre5243bb0-8e'/>
      <target dev='tape5243bb0-8e'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='file'>
      <source path='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/console.log'/>
      <target port='0'/>
    </serial>
    <serial type='pty'>
      <target port='1'/>
    </serial>
    <console type='file'>
      <source path='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/console.log'/>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='cirrus' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <stats period='10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [req-5a6532d4-317f-4165-855e-37d8b36c71ec 295fe519b3864de5894fa0219362c26f 2e4cb36a0a3645e7b924deecea3bd210 - - -] [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] Instance failed to spawn
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] Traceback (most recent call last):
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2156, in _build_resources
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     yield resources
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     block_device_info=block_device_info)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2585, in spawn
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     block_device_info=block_device_info)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4699, in _create_domain_and_network
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     xml, pause=pause, power_on=power_on)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4629, in _create_domain
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     guest.launch(pause=pause)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 142, in launch
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     self._encoded_xml, errors='ignore')
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     six.reraise(self.type_, self.value, self.tb)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 137, in launch
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     return self._domain.createWithFlags(flags)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     result = proxy_call(self._autowrap, f, *args, **kwargs)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     rv = execute(f, *args, **kwargs)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     six.reraise(c, e, tb)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     rv = meth(*args, **kwargs)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]   File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] libvirtError: internal error: early end of file from monitor: possible problem:
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.577395Z qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: vfio: Error: Failed to setup INTx fd: Device or resource busy
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.578216Z qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device initialization failed
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.578231Z qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 
2016-08-08 21:20:54.118 20236 WARNING nova.compute.manager [req-d1bd757e-ab71-472b-8555-935bfbecbe89 - - - - -] Bandwidth usage not supported by hypervisor.

Moshe Levi

tags

participants (1)