I enabled iommu in the grub-config:
grep iommu /boot/grub2/grub.cfg
linux16 /vmlinuz-3.10.0-123.8.1.el7.x86_64 root=UUID=9cfb58ce-b401-47d1-8dc0-babbd1f21ec3
ro rootflags=subvol=root crashkernel=auto vconsole.keymap=de
vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on
Here is the relevant output of /var/log/messages during boot:
Oct 7 09:18:01 artemis2 kernel: dmar: Host address width 39
Oct 7 09:18:01 artemis2 kernel: dmar: DRHD base: 0x000000fed90000 flags: 0x1
Oct 7 09:18:01 artemis2 kernel: dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap
d2008c20660462 ecap f010da
Oct 7 09:18:01 artemis2 kernel: dmar: RMRR base: 0x000000ceefa000 end: 0x000000cef06fff
Oct 7 09:18:01 artemis2 kernel: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0
Later on I get the following error:
Oct 7 09:18:01 artemis2 kernel: DMAR: No ATSR found
Oct 7 09:18:01 artemis2 kernel: IOMMU 0 0xfed90000: using Queued invalidation
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting RMRR:
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1d.0
[0xceefa000 - 0xcef06fff]
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1a.0
[0xceefa000 - 0xcef06fff]
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:14.0
[0xceefa000 - 0xcef06fff]
Oct 7 09:18:01 artemis2 kernel: IOMMU: Prepare 0-16MiB unity mapping for LPC
Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1f.0 [0x0
- 0xffffff]
Oct 7 09:18:01 artemis2 kernel: PCI-DMA: Intel(R) Virtualization Technology for Directed
I/O
Oct 7 09:18:01 artemis2 kernel: ------------[ cut here ]------------
Oct 7 09:18:01 artemis2 kernel: WARNING: at drivers/pci/search.c:46
pci_find_upstream_pcie_bridge+0x87/0x90()
Oct 7 09:18:01 artemis2 kernel: Modules linked in:
Oct 7 09:18:01 artemis2 kernel: CPU: 0 PID: 1 Comm: swapper/0 Not tainted
3.10.0-123.8.1.el7.x86_64 #1
Oct 7 09:18:01 artemis2 kernel: Hardware name: Hewlett-Packard HP Z230 Tower
Workstation/1905, BIOS L51 v01.13 10/18/2013
Oct 7 09:18:01 artemis2 kernel: 0000000000000000 00000000eccfd72f ffff8802220e3cf0
ffffffff815e237b
Oct 7 09:18:01 artemis2 kernel: ffff8802220e3d28 ffffffff8105dee1 ffff880221f95000
ffff880221f9f000
Oct 7 09:18:01 artemis2 kernel: ffff880221f9f098 0000000000000000 0000000000000000
ffff8802220e3d38
Oct 7 09:18:01 artemis2 kernel: Call Trace:
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815e237b>] dump_stack+0x19/0x1b
Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105dee1>]
warn_slowpath_common+0x61/0x80
Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105e00a>] warn_slowpath_null+0x1a/0x20
Oct 7 09:18:01 artemis2 kernel: [<ffffffff812ec607>]
pci_find_upstream_pcie_bridge+0x87/0x90
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814aa04d>]
intel_iommu_add_device+0x4d/0x230
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10f0>] ? bus_set_iommu+0x50/0x50
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a111a>] add_iommu_group+0x2a/0x50
Oct 7 09:18:01 artemis2 kernel: [<ffffffff813b4443>] bus_for_each_dev+0x73/0xc0
Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10e8>] bus_set_iommu+0x48/0x50
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a5837d>] intel_iommu_init+0x3ea/0x413
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11c93>] ?
memblock_find_dma_reserve+0x147/0x147
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11ca5>] pci_iommu_init+0x12/0x3c
Oct 7 09:18:01 artemis2 kernel: [<ffffffff810020e2>] do_one_initcall+0xe2/0x190
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a09153>]
kernel_init_freeable+0x18b/0x22a
Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a0892b>] ? do_early_param+0x88/0x88
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c2e>] kernel_init+0xe/0x180
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815f29ac>] ret_from_fork+0x7c/0xb0
Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80
Oct 7 09:18:01 artemis2 kernel: ---[ end trace 56f25a222bb09abf ]---
Best regards
Michael
-----Ursprüngliche Nachricht-----
Von: libvirt-users-bounces(a)redhat.com [mailto:libvirt-users-bounces@redhat.com] Im Auftrag
von Pierre Schweitzer
Gesendet: Montag, 13. Oktober 2014 13:19
An: libvirt-users(a)redhat.com
Betreff: Re: [libvirt-users] passthrough of PCI-device
And what about IO MMU ?
On 10/13/2014 12:02 PM, Weis, Michael (DWIE) wrote:
Hi Pierre,
thanks for your reply.
I am using kernel 3.10.0-123.8.1.el7.x86_64.
The kernel modul used after nodedev-detach is vfio-pci
This is the output of lspci -vv after I did a virsh nodedev-detatch
pci_0000_02_00_0
02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e
Subsystem: PLX Technology, Inc. Device 235e
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping-
SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort-
<MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 16
Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled] [size=256]
Region 1: I/O ports at e000 [disabled] [size=256]
Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=256M]
Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M]
Expansion ROM at f0800000 [disabled] [size=64K]
Capabilities: [40] Power Management version 1
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [48] CompactPCI hot-swap <?>
Capabilities: [4c] Vital Product Data
Unknown large resource type 35, will not decode more.
Kernel driver in use: vfio-pci
Best regards
Michael
-----Ursprüngliche Nachricht-----
Von: libvirt-users-bounces(a)redhat.com
[mailto:libvirt-users-bounces@redhat.com] Im Auftrag von Pierre
Schweitzer
Gesendet: Montag, 13. Oktober 2014 10:04
An: libvirt-users(a)redhat.com
Betreff: Re: [libvirt-users] passthrough of PCI-device
Dear Michael,
Did you activate the Intel IO MMU (or its equivalent for AMD)?
Also, did you load the pci_stub module for Linux? It is mandatory (it will replace
current driver for your passed through hardware).
Cheers,
Pierre
On 10/13/2014 07:54 AM, Weis, Michael (DWIE) wrote:
> Good morning,
>
> there is a typo in my description;
> the line
> <address domain='0x0' bus='0x1' slot='0x00'
function='0x0'/>
> should be
> <address domain='0x0' bus='0x2' slot='0x00'
function='0x0'/>
> That was correct in my xml-file.
>
> Isn't there anybody how can help me with that?
>
>
> Regards
>
> Michael Weis
>
>
>
> Von: Weis, Michael (DWIE)
> Gesendet: Dienstag, 7. Oktober 2014 11:45
> An: 'libvirt-users(a)redhat.com'
> Betreff: passthrough of PCI-device
>
> Hello,
>
> I try to passthrough a PCI-card to a VM named testvm I want to do
> that with an xml-file named hga.xml including the following content:
>
> <hostdev mode='subsystem' type='pci' managed='yes'>
> <source>
> <address domain='0x0' bus='0x1' slot='0x00'
function='0x0'/>
> </source>
> </hostdev>
>
> When I execute
> virsh attach-device testvm hga.xml
> I get the following error:
> Fehler: Fehler beim Anhängen des Geräts von hga.xml
> (error: error while attaching device from hga.xml)
> Fehler: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen:
Device initialization failed.
> (error: internal error: cannot execute 'device_add':
>
> In the syslog of the host I get the following errors:
> Oct 7 11:32:34 artemis2 journal: Interner Fehler: Kann Qemu-Befehl
'device_add' nicht ausführen: Device initialization failed.
> Oct 7 11:32:53 artemis2 kernel: ffff880220fadd10 ffffffff810f924a
> ffff880220fadd28 0000000000000246 Oct 7 11:32:53 artemis2 kernel:
> ffff8800bd07a400 000000007265756d ffffffffa0866ee0 0000000000000000 Oct 7 11:32:53
artemis2 kernel: Call Trace:
> Oct 7 11:32:53 artemis2 kernel: [<ffffffff815e237b>]
> dump_stack+0x19/0x1b Oct 7 11:32:53 artemis2 kernel:
> [<ffffffff810f924a>] __setup_irq+0x54a/0x570 Oct 7 11:32:53 artemis2
> kernel: [<ffffffffa0866ee0>] ? virqfd_wakeup+0xc0/0xc0 [vfio_pci] Oct
> 7 11:32:53 artemis2 kernel: [<ffffffff810f93cc>]
> request_threaded_irq+0xcc/0x170 Oct 7 11:32:53 artemis2 kernel:
> [<ffffffffa0866b8a>] vfio_intx_set_signal+0x10a/0x1f0 [vfio_pci] Oct
> 7 11:32:53 artemis2 kernel: [<ffffffffa08670e8>]
> vfio_pci_set_intx_trigger+0xc8/0x1b0 [vfio_pci] Oct 7 11:32:53
> artemis2 kernel: [<ffffffffa086799b>]
> vfio_pci_set_irqs_ioctl+0x7b/0x100 [vfio_pci] Oct 7 11:32:53
> artemis2
> kernel: [<ffffffff8115d352>] ? memdup_user+0x42/0x70 Oct 7 11:32:53
> artemis2 kernel: [<ffffffffa0865e74>] vfio_pci_ioctl+0x314/0xa00
> [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffff81194b6a>] ?
> kmem_cache_alloc+0x1ba/0x1d0 Oct 7 11:32:53 artemis2 kernel:
> [<ffffffff811b156c>] ? get_empty_filp+0x5c/0x190 Oct 7 11:32:53
> artemis2 kernel: [<ffffffff811f1891>] ? fsnotify+0x251/0x350 Oct 7
> 11:32:53 artemis2 kernel: [<ffffffff81251806>] ?
> security_file_alloc+0x16/0x20 Oct 7 11:32:53 artemis2 kernel:
> [<ffffffff811b15a0>] ? get_empty_filp+0x90/0x190 Oct 7 11:32:53 artemis2
kernel: [<ffffffffa084f163>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio] Oct 7
11:32:53 artemis2 kernel: [<ffffffff811c2af5>] do_vfs_ioctl+0x2e5/0x4c0 Oct 7
11:32:53 artemis2 kernel: [<ffffffff811af780>] ? vfs_write+0x160/0x1e0 Oct 7
11:32:53 artemis2 kernel: [<ffffffff811c2d71>] SyS_ioctl+0xa1/0xc0 Oct 7 11:32:53
artemis2 kernel: [<ffffffff815f2a59>] system_call_fastpath+0x16/0x1b Oct 7 11:32:53
artemis2 journal: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen:
Device initialization failed.
>
> My version-information:
> OS: CentOS Linux release 7.0.1406 (Core)
> Kernel: Linux artemis2 3.10.0-123.8.1.el7.x86_64
> libvirt-daemon-driver-secret-1.2.8-1.el7.centos.x86_64
> libvirt-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-kvm-1.2.8-1.el7.centos.x86_64
> libvirt-client-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-driver-qemu-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-config-network-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-driver-interface-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-driver-nodedev-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-driver-storage-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-config-nwfilter-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-driver-network-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-driver-lxc-1.2.8-1.el7.centos.x86_64
> libvirt-daemon-driver-nwfilter-1.2.8-1.el7.centos.x86_64
>
> The PCI-Card I want to passthrough:
> 02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e
> Subsystem: PLX Technology, Inc. Device 235e
> Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping-
SERR- FastB2B- DisINTx-
> Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
> Interrupt: pin A routed to IRQ 16
> Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled]
[size=256]
> Region 1: I/O ports at e000 [disabled] [size=256]
> Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled]
[size=256M]
> Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M]
> Expansion ROM at f0800000 [disabled] [size=64K]
> Capabilities: [40] Power Management version 1
> Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
> Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
> Capabilities: [48] CompactPCI hot-swap <?>
> Capabilities: [4c] Vital Product Data
> Unknown large resource type 35, will not decode more.
> Kernel driver in use: vfio-pci
>
> Please help.
>
>
> Best regards
>
> Michael Weis
>
>
>
> PROPRIETARY: This e-mail contains proprietary information some or all of which may be
legally privileged. It is intended for the recipient only. If an addressing or
transmission error has misdirected this e-mail, please notify the authority by replying to
this e-mail. If you are not the intended recipient you must not use, disclose,
distribute, copy, print, or rely on this e-mail.
>
>
>
> _______________________________________________
> libvirt-users mailing list
> libvirt-users(a)redhat.com
>
https://www.redhat.com/mailman/listinfo/libvirt-users
>
--
Pierre Schweitzer <pierre(a)reactos.org> System & Network Administrator
Senior Kernel Developer ReactOS Deutschland e.V.
______________________________________________________________________
CAUTION: This message was sent via the Public Internet and its authenticity cannot be
guaranteed.
PROPRIETARY: This e-mail contains proprietary information some or all of which may be
legally privileged. It is intended for the recipient only. If an addressing or
transmission error has misdirected this e-mail, please notify the authority by replying to
this e-mail. If you are not the intended recipient you must not use, disclose,
distribute, copy, print, or rely on this e-mail.
_______________________________________________
libvirt-users mailing list
libvirt-users(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users
--
Pierre Schweitzer <pierre(a)reactos.org>
System & Network Administrator
Senior Kernel Developer
ReactOS Deutschland e.V.
______________________________________________________________________
CAUTION: This message was sent via the Public Internet and its authenticity cannot be
guaranteed.
PROPRIETARY: This e-mail contains proprietary information some or all of which may be
legally privileged. It is intended for the recipient only. If an addressing or
transmission error has misdirected this e-mail, please notify the authority by replying to
this e-mail. If you are not the intended recipient you must not use, disclose,
distribute, copy, print, or rely on this e-mail.