1:22 a.m.
Hi,
I am trying to PF pass-thought with Mellanox NIC CX4 with duel port card.
I know the currently it doesn't support ACS, therefore we can't allocate 2 PF to 2
guest. (both pci device are in the same IOMMU group)
But if I understand accoriding to [1] this currently there should be a problem to
pass-thought both PFs to the same guest.
before we boot the guest I unbind the pci device with the following commands.
echo 0000:03:00.00> /sys/bus/pci/devices/0000:03:00.00/driver/unbind
echo 0000:03:00.01 > /sys/bus/pci/devices/0000:03:00.1/driver/unbind
I tried it worked and it was working on one environment , but on other environment we
are getting the
following error vfio: Error: Failed to setup INTx fd: Device or resource busy. See detail
error [2].
Can anyone explain when/why this error occurs? And how can we fix it?
[1] - http://vfio.blogspot.co.il/2014/08/iommu-groups-inside-and-out.html
[2] - Nova-compute.log error:
2016-08-08 21:20:14.213 20236 ERROR nova.virt.libvirt.guest
[req-5a6532d4-317f-4165-855e-37d8b36c71ec 295fe519b3864de5894fa0219362c26f
2e4cb36a0a3645e7b924deecea3bd210 - - -] Error launching a defined domain with XML:
<domain type='kvm'>
<name>instance-00000040</name>
<uuid>74abfd25-aab6-4362-ab28-49c3bbd5e508</uuid>
<metadata>
<nova:instance
xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.0">
<nova:package version="12.0.3-1.el7ost"/>
<nova:name>Test</nova:name>
<nova:creationTime>2016-08-08 21:20:08</nova:creationTime>
<nova:flavor name="Mellanox_PT_2">
<nova:memory>16384</nova:memory>
<nova:disk>0</nova:disk>
<nova:swap>0</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>16</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user
uuid="295fe519b3864de5894fa0219362c26f">admin</nova:user>
<nova:project
uuid="2e4cb36a0a3645e7b924deecea3bd210">openstack</nova:project>
</nova:owner>
<nova:root type="image"
uuid="49d20d6e-e247-43ff-aeec-84ba53d6b439"/>
</nova:instance>
</metadata>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<vcpu placement='static' cpuset='3-23,27-47'>16</vcpu>
<cputune>
<shares>16384</shares>
</cputune>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Red Hat</entry>
<entry name='product'>OpenStack Compute</entry>
<entry name='version'>12.0.3-1.el7ost</entry>
<entry
name='serial'>d336b2a1-4abd-4602-9bc4-4aeb20f778e7</entry>
<entry name='uuid'>74abfd25-aab6-4362-ab28-49c3bbd5e508</entry>
<entry name='family'>Virtual Machine</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64'
machine='pc-i440fx-rhel7.2.0'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model'>
<model fallback='allow'/>
<topology sockets='16' cores='1' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source
file='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/disk'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x04' function='0x0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<interface type='bridge'>
<mac address='fa:16:3e:14:ec:96'/>
<source bridge='qbre5243bb0-8e'/>
<target dev='tape5243bb0-8e'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0'/>
</interface>
<serial type='file'>
<source
path='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/console.log'/>
<target port='0'/>
</serial>
<serial type='pty'>
<target port='1'/>
</serial>
<console type='file'>
<source
path='/var/lib/nova/instances/74abfd25-aab6-4362-ab28-49c3bbd5e508/console.log'/>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'
listen='0.0.0.0' keymap='en-us'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='16384' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x03' slot='0x00'
function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x05' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<source>
<address domain='0x0000' bus='0x03' slot='0x00'
function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x06' function='0x0'/>
</hostdev>
<memballoon model='virtio'>
<stats period='10'/>
<address type='pci' domain='0x0000' bus='0x00'
slot='0x07' function='0x0'/>
</memballoon>
</devices>
</domain>
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager
[req-5a6532d4-317f-4165-855e-37d8b36c71ec 295fe519b3864de5894fa0219362c26f
2e4cb36a0a3645e7b924deecea3bd210 - - -] [instance: 74abfd25-aab6-4362-ab28-49c3bbd5e508]
Instance failed to spawn
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] Traceback (most recent call last):
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2156, in
_build_resources
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] yield resources
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in
_build_and_run_instance
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] block_device_info=block_device_info)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2585, in
spawn
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] block_device_info=block_device_info)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4699, in
_create_domain_and_network
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] xml, pause=pause, power_on=power_on)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4629, in
_create_domain
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] guest.launch(pause=pause)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 142, in
launch
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] self._encoded_xml, errors='ignore')
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 204, in
__exit__
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] six.reraise(self.type_, self.value, self.tb)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/nova/virt/libvirt/guest.py", line 137, in
launch
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] return self._domain.createWithFlags(flags)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 183, in doit
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] result = proxy_call(self._autowrap, f, *args,
**kwargs)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 141, in proxy_call
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] rv = execute(f, *args, **kwargs)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 122, in execute
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] six.reraise(c, e, tb)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib/python2.7/site-packages/eventlet/tpool.py", line 80, in tworker
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] rv = meth(*args, **kwargs)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] File
"/usr/lib64/python2.7/site-packages/libvirt.py", line 1059, in createWithFlags
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] if ret == -1: raise libvirtError
('virDomainCreateWithFlags() failed', dom=self)
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] libvirtError: internal error: early end of file from
monitor: possible problem:
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.577395Z qemu-kvm: -device
vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: vfio: Error: Failed to setup INTx
fd: Device or resource busy
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.578216Z qemu-kvm: -device
vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device initialization failed
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508] 2016-08-08T21:20:13.578231Z qemu-kvm: -device
vfio-pci,host=03:00.0,id=hostdev1,bus=pci.0,addr=0x6: Device 'vfio-pci' could not
be initialized
2016-08-08 21:20:14.214 20236 ERROR nova.compute.manager [instance:
74abfd25-aab6-4362-ab28-49c3bbd5e508]
2016-08-08 21:20:54.118 20236 WARNING nova.compute.manager
[req-d1bd757e-ab71-472b-8555-935bfbecbe89 - - - - -] Bandwidth usage not supported by
hypervisor.