Hello,

I want to passthrough an InfiniBand-Card to a guest and have some problems with that. My CPU (Intel Xeon E5-2650) is vt-d capable and vt-d is activated in the BIOS. My CentOS 6.4 (2.6.32-358.11.1.el6.x86_64) is booting with the command line "... intel_iommu=on iommu=pt".

The guest has the same OS and kernel as the host, also the driver used for the InfiniBand-Card is the same (used the official Mellanox OFED driver pack).

This is what I've done:

0.) Verify that InfiniBand working on host:

root@host# ibv_devinfo
hca_id:   mthca0
   transport:           InfiniBand (0)
   fw_ver:               1.0.800
   node_guid:            ...
   sys_image_guid:            ...
   vendor_id:           0x08f1
   vendor_part_id:           25204
   hw_ver:               0xA0
   board_id:            ...
   phys_port_cnt:           1
       port:   1
           state:           PORT_ACTIVE (4)
           max_mtu:       2048 (4)
           active_mtu:       2048 (4)
           sm_lid:           3
           port_lid:       5
           port_lmc:       0x00
           link_layer:       InfiniBand

1.) DMAR and IOMMU messages at host:

root@host# dmesg | grep -e DMAR -e IOMMU
ACPI: DMAR 000000007e27ea30 00160 (v01 A M I OEMDMAR 00000001 INTL 00000001)
Intel-IOMMU: enabled
dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0462 ecap f020fe
dmar: IOMMU 1: reg_base_addr dfffc000 ver 1:0 cap d2078c106f0462 ecap f020fe
IOMMU 0xfbffe000: using Queued invalidation
IOMMU 0xdfffc000: using Queued invalidation
IOMMU: hardware identity mapping for device 0000:00:00.0

[... (a lot of mapping messages) ...]

IOMMU: hardware identity mapping for device 0000:81:00.0 (this is the IB card)
IOMMU: Setting RMRR:
IOMMU: Prepare 0-16MiB unity mapping for LPC
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000
DMAR:[fault reason 06] PTE Read access is not set
dmar: DMAR:[DMA Read] Request device [81:00.0] fault addr 107294f000
DMAR:[fault reason 06] PTE Read access is not set

2.) Output of lspci at host

81:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev a0)
        Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
        Flags: fast devsel, IRQ 114
        Memory at f8a00000 (64-bit, non-prefetchable) [size=1M]
        Memory at 3c1e00000000 (64-bit, prefetchable) [size=8M]
        Capabilities: [40] Power Management version 2
        Capabilities: [48] Vital Product Data
        Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
        Capabilities: [84] MSI-X: Enable- Count=32 Masked-
        Capabilities: [60] Express Endpoint, MSI 00

Kernel driver in use: ib_mthca
Kernel modules: ib_mthca

3.) Adding the device with virsh edit:
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>

The last <address .../>-Tag with "slot=0x07" was added automatically by virsh.

4.) Removing device with nodedev-dettach:

root@host# virsh nodedev-dettach pci_0000_81_00_0
Device pci_0000_81_00_0 detached

lspci on the host now shows the same as in (1) but the "Kernel driver in use" changed:

Kernel driver in use: pci-stub

5.) InfiniBand now stops working on host (as expected):

root@host# ibv_devinfo
No IB devices found

6.) starting guest, then lspci -v:

00:07.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev a0)
   Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
   Physical Slot: 7
   Flags: fast devsel, IRQ 10
   Memory at f2100000 (32-bit, non-prefetchable) [size=1M]
   Memory at f2800000 (32-bit, prefetchable) [size=8M]
   Capabilities: [48] Vital Product Data
   Capabilities: [60] Express Endpoint, MSI 00
   Capabilities: [40] Power Management version 2
   Capabilities: [84] MSI-X: Enable- Count=32 Masked-
   Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit-
   Kernel modules: ib_mthca

I noticed that there is a difference: The memory on guest is 32-bit, on the host it says 64-bit

7.) IB not working on guest:

root@guest# ibv_devinfo
No IB devices found

root@guest# ibhosts

src/query_smp.c:228; can't open UMAD port ((null):0)
/usr/sbin/ibnetdiscover: iberror: failed: discover failed

Do you have any clue where I must tweak my settings in order to get InfiniBand working on my virtual machine? If this is no libvirt problem, I am sorry :)

Regards,

Sebastian