Hi, I am trying to PF pass-thought with Mellanox NIC CX4 with duel port card. I know the currently it doesn't support ACS, therefore we can't allocate 2 PF to 2 guest. (both pci device are in the same IOMMU group) But if I understand accoriding to [1] this currently there should be a problem to pass-thought both PFs to the same guest. before we boot the guest I unbind the pci device with the following commands. echo 0000:03:00.00> /sys/bus/pci/devices/0000:03:00.00/driver/unbind echo 0000:03:00.01 > /sys/bus/pci/devices/0000:03:00.1/driver/unbind I tried it worked and it was working on one environment , but on other environment we are getting the following error vfio: Error: Failed to setup INTx fd: Device or resource busy. See detail error [2]. Can anyone explain when/why this error occurs? And how can we fix it? [1] - http://vfio.blogspot.co.il/2014/08/iommu-groups-inside-and-out.html [2] - Nova-compute.log error: 2016-08-08 21:20:14.213 20236 ERROR nova.virt.libvirt.guest [req-5a6532d4-317f-4165-855e-37d8b36c71ec 295fe519b3864de5894fa0219362c26f 2e4cb36a0a3645e7b924deecea3bd210 - - -] Error launching a defined domain with XML: <domain type='kvm'> <name>instance-00000040</name> <uuid>74abfd25-aab6-4362-ab28-49c3bbd5e508</uuid> <metadata> <nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0"> <nova:package version="12.0.3-1.el7ost"/> <nova:name>Test</nova:name> <nova:creationTime>2016-08-08 21:20:08</nova:creationTime> <nova:flavor name="Mellanox_PT_2"> <nova:memory>16384</nova:memory> <nova:disk>0</nova:disk> <nova:swap>0</nova:swap> <nova:ephemeral>0</nova:ephemeral> <nova:vcpus>16</nova:vcpus> </nova:flavor> <nova:owner> <nova:user uuid="295fe519b3864de5894fa0219362c26f">admin</nova:user> <nova:project uuid="2e4cb36a0a3645e7b924deecea3bd210">openstack</nova:project> </nova:owner> <nova:root type="image" uuid="49d20d6e-e247-43ff-aeec-84ba53d6b439"/> </nova:instance> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <vcpu placement='static' cpuset='3-23,27-47'>16</vcpu> <cputune> <shares>16384</shares> </cputune> <sysinfo type='smbios'> <system> <entry name='manufacturer'>Red Hat</entry> <entry name='product'>OpenStack Compute</entry> <entry name='version'>12.0.3-1.el7ost</entry> <entry name='serial'>d336b2a1-4abd-4602-9bc4-4aeb20f778e7</entry> <entry name='uuid'>74abfd25-aab6-4362-ab28-49c3bbd5e508</entry> <entry name='family'>Virtual Machine</entry> </system> </sysinfo> <os> <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type> <boot dev='hd'/> <smbios mode='sysinfo'/> </os> <features> <acpi/> <apic/> </features> <cpu mode='host-model'> <model fallback='allow'/> <topology sockets='16' cores='1' threads='1'/> </cpu> <clock offset='utc'> <timer name='pit' tickpolicy='delay'/> <timer name='rtc' tickpolicy='catchup'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/libexec/qemu-kvm</emulator> <disk type='file' device='disk'> <driver name='qemu' type='qcow2' cache='none'/> <source file='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/disk'/> <target dev='vda' bus='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </disk> <controller type='usb' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='0' model='pci-root'/> <interface type='bridge'> <mac address='fa:16:3e:14:ec:96'/> <source bridge='qbre5243bb0-8e'/> <target dev='tape5243bb0-8e'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </interface> <serial type='file'> <source path='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/console.log'/> <target port='0'/> </serial> <serial type='pty'> <target port='1'/> </serial> <console type='file'> <source path='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/console.log'/> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='-1' autoport='yes' listen='0.0.0.0' keymap='en-us'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='cirrus' vram='16384' heads='1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <memballoon model='virtio'> <stats period='10'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </memballoon> </devices> </domain> 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [req-5a6532d4-317f-4165-855e-37d8b36c71ec 295fe519b3864de5894fa0219362c26f 2e4cb36a0a3645e7b924deecea3bd210 - - -] [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] Instance failed to spawn 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] Traceback (most recent call last): 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2156, in _build_resources 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] yield resources 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] block_device_info=block_device_info) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2585, in spawn 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] block_device_info=block_device_info) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4699, in _create_domain_and_network 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] xml, pause=pause, power_on=power_on) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4629, in _create_domain 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] guest.launch(pause=pause) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 142, in launch 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] self._encoded_xml, errors='ignore') 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in __exit__ 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] six.reraise(self.type_, self.value, self.tb) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 137, in launch 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] return self._domain.createWithFlags(flags) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] result = proxy_call(self._autowrap, f, *args, **kwargs) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] rv = execute(f, *args, **kwargs) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] six.reraise(c, e, tb) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] rv = meth(*args, **kwargs) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] File "/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self) 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] libvirtError: internal error: early end of file from monitor: possible problem: 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.577395Z qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: vfio: Error: Failed to setup INTx fd: Device or resource busy 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.578216Z qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device initialization failed 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.578231Z qemu-kvm: -device vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device 'vfio-pci' could not be initialized 2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08 21:20:54.118 20236 WARNING nova.compute.manager [req-d1bd757e-ab71-472b-8555-935bfbecbe89 - - - - -] Bandwidth usage not supported by hypervisor.