I enabled iommu in the grub-config: grep iommu /boot/grub2/grub.cfg linux16 /vmlinuz-3.10.0-123.8.1.el7.x86_64 root=UUID=9cfb58ce-b401-47d1-8dc0-babbd1f21ec3 ro rootflags=subvol=root crashkernel=auto vconsole.keymap=de vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=de_DE.UTF-8 intel_iommu=on Here is the relevant output of /var/log/messages during boot: Oct 7 09:18:01 artemis2 kernel: dmar: Host address width 39 Oct 7 09:18:01 artemis2 kernel: dmar: DRHD base: 0x000000fed90000 flags: 0x1 Oct 7 09:18:01 artemis2 kernel: dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da Oct 7 09:18:01 artemis2 kernel: dmar: RMRR base: 0x000000ceefa000 end: 0x000000cef06fff Oct 7 09:18:01 artemis2 kernel: IOAPIC id 2 under DRHD base 0xfed90000 IOMMU 0 Later on I get the following error: Oct 7 09:18:01 artemis2 kernel: DMAR: No ATSR found Oct 7 09:18:01 artemis2 kernel: IOMMU 0 0xfed90000: using Queued invalidation Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting RMRR: Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1d.0 [0xceefa000 - 0xcef06fff] Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1a.0 [0xceefa000 - 0xcef06fff] Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:14.0 [0xceefa000 - 0xcef06fff] Oct 7 09:18:01 artemis2 kernel: IOMMU: Prepare 0-16MiB unity mapping for LPC Oct 7 09:18:01 artemis2 kernel: IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff] Oct 7 09:18:01 artemis2 kernel: PCI-DMA: Intel(R) Virtualization Technology for Directed I/O Oct 7 09:18:01 artemis2 kernel: ------------[ cut here ]------------ Oct 7 09:18:01 artemis2 kernel: WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x87/0x90() Oct 7 09:18:01 artemis2 kernel: Modules linked in: Oct 7 09:18:01 artemis2 kernel: CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.10.0-123.8.1.el7.x86_64 #1 Oct 7 09:18:01 artemis2 kernel: Hardware name: Hewlett-Packard HP Z230 Tower Workstation/1905, BIOS L51 v01.13 10/18/2013 Oct 7 09:18:01 artemis2 kernel: 0000000000000000 00000000eccfd72f ffff8802220e3cf0 ffffffff815e237b Oct 7 09:18:01 artemis2 kernel: ffff8802220e3d28 ffffffff8105dee1 ffff880221f95000 ffff880221f9f000 Oct 7 09:18:01 artemis2 kernel: ffff880221f9f098 0000000000000000 0000000000000000 ffff8802220e3d38 Oct 7 09:18:01 artemis2 kernel: Call Trace: Oct 7 09:18:01 artemis2 kernel: [<ffffffff815e237b>] dump_stack+0x19/0x1b Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105dee1>] warn_slowpath_common+0x61/0x80 Oct 7 09:18:01 artemis2 kernel: [<ffffffff8105e00a>] warn_slowpath_null+0x1a/0x20 Oct 7 09:18:01 artemis2 kernel: [<ffffffff812ec607>] pci_find_upstream_pcie_bridge+0x87/0x90 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814aa04d>] intel_iommu_add_device+0x4d/0x230 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10f0>] ? bus_set_iommu+0x50/0x50 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a111a>] add_iommu_group+0x2a/0x50 Oct 7 09:18:01 artemis2 kernel: [<ffffffff813b4443>] bus_for_each_dev+0x73/0xc0 Oct 7 09:18:01 artemis2 kernel: [<ffffffff814a10e8>] bus_set_iommu+0x48/0x50 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a5837d>] intel_iommu_init+0x3ea/0x413 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11c93>] ? memblock_find_dma_reserve+0x147/0x147 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a11ca5>] pci_iommu_init+0x12/0x3c Oct 7 09:18:01 artemis2 kernel: [<ffffffff810020e2>] do_one_initcall+0xe2/0x190 Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a09153>] kernel_init_freeable+0x18b/0x22a Oct 7 09:18:01 artemis2 kernel: [<ffffffff81a0892b>] ? do_early_param+0x88/0x88 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c2e>] kernel_init+0xe/0x180 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815f29ac>] ret_from_fork+0x7c/0xb0 Oct 7 09:18:01 artemis2 kernel: [<ffffffff815c3c20>] ? rest_init+0x80/0x80 Oct 7 09:18:01 artemis2 kernel: ---[ end trace 56f25a222bb09abf ]--- Best regards Michael -----Ursprüngliche Nachricht----- Von: libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@redhat.com] Im Auftrag von Pierre Schweitzer Gesendet: Montag, 13. Oktober 2014 13:19 An: libvirt-users@redhat.com Betreff: Re: [libvirt-users] passthrough of PCI-device And what about IO MMU ? On 10/13/2014 12:02 PM, Weis, Michael (DWIE) wrote:
Hi Pierre,
thanks for your reply.
I am using kernel 3.10.0-123.8.1.el7.x86_64. The kernel modul used after nodedev-detach is vfio-pci
This is the output of lspci -vv after I did a virsh nodedev-detatch pci_0000_02_00_0
02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e Subsystem: PLX Technology, Inc. Device 235e Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled] [size=256] Region 1: I/O ports at e000 [disabled] [size=256] Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=256M] Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M] Expansion ROM at f0800000 [disabled] [size=64K] Capabilities: [40] Power Management version 1 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] CompactPCI hot-swap <?> Capabilities: [4c] Vital Product Data Unknown large resource type 35, will not decode more. Kernel driver in use: vfio-pci
Best regards
Michael
-----Ursprüngliche Nachricht----- Von: libvirt-users-bounces@redhat.com [mailto:libvirt-users-bounces@redhat.com] Im Auftrag von Pierre Schweitzer Gesendet: Montag, 13. Oktober 2014 10:04 An: libvirt-users@redhat.com Betreff: Re: [libvirt-users] passthrough of PCI-device
Dear Michael,
Did you activate the Intel IO MMU (or its equivalent for AMD)? Also, did you load the pci_stub module for Linux? It is mandatory (it will replace current driver for your passed through hardware).
Cheers, Pierre
On 10/13/2014 07:54 AM, Weis, Michael (DWIE) wrote:
Good morning,
there is a typo in my description; the line <address domain='0x0' bus='0x1' slot='0x00' function='0x0'/> should be <address domain='0x0' bus='0x2' slot='0x00' function='0x0'/> That was correct in my xml-file.
Isn't there anybody how can help me with that?
Regards
Michael Weis
Von: Weis, Michael (DWIE) Gesendet: Dienstag, 7. Oktober 2014 11:45 An: 'libvirt-users@redhat.com' Betreff: passthrough of PCI-device
Hello,
I try to passthrough a PCI-card to a VM named testvm I want to do that with an xml-file named hga.xml including the following content:
<hostdev mode='subsystem' type='pci' managed='yes'> <source> <address domain='0x0' bus='0x1' slot='0x00' function='0x0'/> </source> </hostdev>
When I execute virsh attach-device testvm hga.xml I get the following error: Fehler: Fehler beim Anhängen des Geräts von hga.xml (error: error while attaching device from hga.xml) Fehler: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed. (error: internal error: cannot execute 'device_add':
In the syslog of the host I get the following errors: Oct 7 11:32:34 artemis2 journal: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed. Oct 7 11:32:53 artemis2 kernel: ffff880220fadd10 ffffffff810f924a ffff880220fadd28 0000000000000246 Oct 7 11:32:53 artemis2 kernel: ffff8800bd07a400 000000007265756d ffffffffa0866ee0 0000000000000000 Oct 7 11:32:53 artemis2 kernel: Call Trace: Oct 7 11:32:53 artemis2 kernel: [<ffffffff815e237b>] dump_stack+0x19/0x1b Oct 7 11:32:53 artemis2 kernel: [<ffffffff810f924a>] __setup_irq+0x54a/0x570 Oct 7 11:32:53 artemis2 kernel: [<ffffffffa0866ee0>] ? virqfd_wakeup+0xc0/0xc0 [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffff810f93cc>] request_threaded_irq+0xcc/0x170 Oct 7 11:32:53 artemis2 kernel: [<ffffffffa0866b8a>] vfio_intx_set_signal+0x10a/0x1f0 [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffffa08670e8>] vfio_pci_set_intx_trigger+0xc8/0x1b0 [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffffa086799b>] vfio_pci_set_irqs_ioctl+0x7b/0x100 [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffff8115d352>] ? memdup_user+0x42/0x70 Oct 7 11:32:53 artemis2 kernel: [<ffffffffa0865e74>] vfio_pci_ioctl+0x314/0xa00 [vfio_pci] Oct 7 11:32:53 artemis2 kernel: [<ffffffff81194b6a>] ? kmem_cache_alloc+0x1ba/0x1d0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811b156c>] ? get_empty_filp+0x5c/0x190 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811f1891>] ? fsnotify+0x251/0x350 Oct 7 11:32:53 artemis2 kernel: [<ffffffff81251806>] ? security_file_alloc+0x16/0x20 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811b15a0>] ? get_empty_filp+0x90/0x190 Oct 7 11:32:53 artemis2 kernel: [<ffffffffa084f163>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio] Oct 7 11:32:53 artemis2 kernel: [<ffffffff811c2af5>] do_vfs_ioctl+0x2e5/0x4c0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811af780>] ? vfs_write+0x160/0x1e0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff811c2d71>] SyS_ioctl+0xa1/0xc0 Oct 7 11:32:53 artemis2 kernel: [<ffffffff815f2a59>] system_call_fastpath+0x16/0x1b Oct 7 11:32:53 artemis2 journal: Interner Fehler: Kann Qemu-Befehl 'device_add' nicht ausführen: Device initialization failed.
My version-information: OS: CentOS Linux release 7.0.1406 (Core) Kernel: Linux artemis2 3.10.0-123.8.1.el7.x86_64 libvirt-daemon-driver-secret-1.2.8-1.el7.centos.x86_64 libvirt-1.2.8-1.el7.centos.x86_64 libvirt-daemon-kvm-1.2.8-1.el7.centos.x86_64 libvirt-client-1.2.8-1.el7.centos.x86_64 libvirt-daemon-driver-qemu-1.2.8-1.el7.centos.x86_64 libvirt-daemon-config-network-1.2.8-1.el7.centos.x86_64 libvirt-daemon-driver-interface-1.2.8-1.el7.centos.x86_64 libvirt-daemon-1.2.8-1.el7.centos.x86_64 libvirt-daemon-driver-nodedev-1.2.8-1.el7.centos.x86_64 libvirt-daemon-driver-storage-1.2.8-1.el7.centos.x86_64 libvirt-daemon-config-nwfilter-1.2.8-1.el7.centos.x86_64 libvirt-daemon-driver-network-1.2.8-1.el7.centos.x86_64 libvirt-daemon-driver-lxc-1.2.8-1.el7.centos.x86_64 libvirt-daemon-driver-nwfilter-1.2.8-1.el7.centos.x86_64
The PCI-Card I want to passthrough: 02:00.0 Ethernet controller: PLX Technology, Inc. Device 235e Subsystem: PLX Technology, Inc. Device 235e Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 16 Region 0: Memory at f0810000 (32-bit, non-prefetchable) [disabled] [size=256] Region 1: I/O ports at e000 [disabled] [size=256] Region 2: Memory at e0000000 (32-bit, non-prefetchable) [disabled] [size=256M] Region 3: Memory at f0000000 (32-bit, non-prefetchable) [disabled] [size=8M] Expansion ROM at f0800000 [disabled] [size=64K] Capabilities: [40] Power Management version 1 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] CompactPCI hot-swap <?> Capabilities: [4c] Vital Product Data Unknown large resource type 35, will not decode more. Kernel driver in use: vfio-pci
Please help.
Best regards
Michael Weis
PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail.
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
-- Pierre Schweitzer <pierre@reactos.org> System & Network Administrator Senior Kernel Developer ReactOS Deutschland e.V.
______________________________________________________________________ CAUTION: This message was sent via the Public Internet and its authenticity cannot be guaranteed.
PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail.
_______________________________________________ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
-- Pierre Schweitzer <pierre@reactos.org> System & Network Administrator Senior Kernel Developer ReactOS Deutschland e.V. ______________________________________________________________________ CAUTION: This message was sent via the Public Internet and its authenticity cannot be guaranteed. PROPRIETARY: This e-mail contains proprietary information some or all of which may be legally privileged. It is intended for the recipient only. If an addressing or transmission error has misdirected this e-mail, please notify the authority by replying to this e-mail. If you are not the intended recipient you must not use, disclose, distribute, copy, print, or rely on this e-mail.