Re: [libvirt-users] PCI Passthrough of 2 identical devices

hi laine, thx for the fast answer. i tried VFIO as it seems supported and got at least one step further. starting the guest with one of the DVB cards is possible now, even when the other card is in the hosts pci config (not removed soft or hard). adding the second card/device thru ... > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> > </hostdev> > <hostdev mode='subsystem' type='pci' managed='yes'> > <driver name='vfio'/> > <source> > <address domain='0x0000' bus='0x03' slot='0x01' function='0x0'/> > </source> > <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> > </hostdev> will render the guest unbootable: --><-- /var/log/libvirt/qemu/tvBackend.log 2014-03-12 16:48:48.018+0000: starting up LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin QEMU_AUDIO_DRV=none /usr/bin/kvm-spice -name tvBackend -S -machine pc-i440fx-1.5,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid f647bb10-7f9a-f94c-33b9-3d99e8e753e0 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/tvBackend.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/kvm/tvBackend.img,if=none,id=drive-virtio-disk0,format=raw -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev tap,fd=29,id=hostnet0,vhost=on,vhostfd=30 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:10:26:ec,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:2 -vga std -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6 -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 W: kvm binary is deprecated, please use qemu-system-x86_64 instead char device redirected to /dev/pts/0 (label charserial0) qemu-system-x86_64: -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x6: Warning, device 0000:03:00.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Warning, device 0000:03:01.0 does not support reset qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: vfio: Error: Failed to setup INTx fd: Device or resource busy qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=03:01.0,id=hostdev1,bus=pci.0,addr=0x7: Device 'vfio-pci' could not be initialized 2014-03-12 16:48:48.639+0000: shutting down --><-- /var/log/syslog Mar 12 18:07:45 father kernel: [34317.290958] genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:03:01.0)) vs. 00000080 (ehci_hcd:usb1) --><-- as i have only 2 pci slots on the board and no other board with serious VT-d/IOMMU support i have no way to swap slots. as the second card shared IRQ 16 with the ehci controller i soft-removed the ehci controller for testing purposes and voila the guest started with the 2 cards successfully inside, even mythtv works ... for now. i hope to find a way to somehow rearrange the irqs as i sometimes need USB on the machine :D wbr,tja... ----- Ursprüngliche Mail ----- Von: "Laine Stump" <laine(a)laine.org&gt; An: libvirt-users(a)redhat.com CC: "Thomas Jagoditsch" <tja(a)tjasoft.com&gt; Gesendet: Mittwoch, 12. März 2014 16:13:15 Betreff: Re: [libvirt-users] PCI Passthrough of 2 identical devices On 03/12/2014 06:29 AM, Thomas Jagoditsch wrote: > hi, > > i have a small trouble with pci-passthrough. > > i have a working configuration passing a tv card into the guest, all > is fine and dandy. > as soon as i add my second identical card into the host i cant start > the guest anymore, whether i add the second card to the guest or not. > error message is identical in both cases. > > message of virt-manager|virsh|libvirtd.log: >> libvirtError: internal error: Unable to reset PCI device >> 0000:03:00.0: internal error: Active 0000:03:01.0 devices on bus with >> 0000:03:00.0, not doing bus reset > the host is a plain kvm server, no services or apps whatsoever > accessing the two devices. > the other guests (firewall and fileserver) do not use the cards > either. The problem is that for kvm device assignment to work properly, the device needs to be reset by libvirt after detaching it from the host driver and before passing it to kvm, and these devices you are trying to reset support neither "function level reset" nor "power management reset", so libvirt must fallback to resetting the entire bus where the device is plugged in. But of course it can't do that if the bus contains other devices that are in use by the host or by other guests. If your host OS is new enough to support vfio device assignment, I would suggest using that instead (if you update to the latest upstrem libvirt it will automatically use VFIO if it is available, otherwise you can add "<driver name='vfio'/>" to the device definition to assure that it will either use vfio or fail. You will also probably need to run "modprobe vfio" before starting the guest). The reason I suggest this is that VFIO will automatically handle resetting the assigned devices whenever necessary, compared to the old KVM device assignment, where libvirt must always reset the device because it has no better information about whether or not it is really necessary. Another thing you can try is plugging one of the cards into a different slot - if you can find a slot that is on a different bus, then libvirt will be able to reset the bus containing the card that will be assigned (since there won't be any other active devices on that same bus). > > if i (soft) remove the 2nd card via >> echo -n 1 > /sys/bus/pci/devices/0000\:03\:01.0/remove > i can start the guest with the 1st card assigned. > > thx for anyone looking into this. libvirt attempts to reset the > > wbr,tja.. > > > PS: > host: >> root@father:~# lspci >> 00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM >> Controller (rev 06) >> 00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 >> v3/4th Gen Core Processor Integrated Graphics Controller (rev 06) >> 00:14.0 USB controller: Intel Corporation 8 Series/C220 Series >> Chipset Family USB xHCI (rev 05) >> 00:16.0 Communication controller: Intel Corporation 8 Series/C220 >> Series Chipset Family MEI Controller #1 (rev 04) >> 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection >> I217-V (rev 05) >> 00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series >> Chipset Family USB EHCI #2 (rev 05) >> 00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset >> Family PCI Express Root Port #1 (rev d5) >> 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d5) >> 00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series >> Chipset Family USB EHCI #1 (rev 05) >> 00:1f.0 ISA bridge: Intel Corporation H87 Express LPC Controller (rev >> 05) >> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series >> Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) >> 00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family >> SMBus Controller (rev 05) >> 02:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI >> Bridge (rev 03) >> 03:00.0 Multimedia controller: Philips Semiconductors SAA7146 (rev >> 01) >> 03:01.0 Multimedia controller: Philips Semiconductors SAA7146 (rev >> 01) >> root@father:~# cat /etc/lsb-release >> DISTRIB_ID=Ubuntu >> DISTRIB_RELEASE=13.10 >> DISTRIB_CODENAME=saucy >> DISTRIB_DESCRIPTION="Ubuntu 13.10" >> root@father:~# uname -a >> Linux father 3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC >> 2014 x86_64 x86_64 x86_64 GNU/Linux >> root@father:~# libvirtd --version >> libvirtd (libvirt) 1.1.1 > guest: >> root@father:~# cat /etc/libvirt/qemu/tvBackend.xml >> <!-- >> WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO >> BE >> OVERWRITTEN AND LOST. Changes to this xml configuration should be >> made using: >> virsh edit tvBackend >> or other application using the libvirt API. >> --> >> >> <domain type='kvm'> >> <name>tvBackend</name> >> <uuid>f647bb10-7f9a-f94c-33b9-3d99e8e753e0</uuid> >> <memory unit='KiB'>1048576</memory> >> <currentMemory unit='KiB'>1048576</currentMemory> >> <vcpu placement='static'>2</vcpu> >> <os> >> <type arch='x86_64' machine='pc-i440fx-1.5'>hvm</type> >> <boot dev='hd'/> >> </os> >> <features> >> <acpi/> >> <apic/> >> <pae/> >> </features> >> <clock offset='utc'/> >> <on_poweroff>destroy</on_poweroff> >> <on_reboot>restart</on_reboot> >> <on_crash>restart</on_crash> >> <devices> >> <emulator>/usr/bin/kvm-spice</emulator> >> <disk type='file' device='disk'> >> <driver name='qemu' type='raw'/> >> <source file='/kvm/tvBackend.img'/> >> <target dev='vda' bus='virtio'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' >> function='0x0'/> >> </disk> >> <disk type='block' device='cdrom'> >> <driver name='qemu' type='raw'/> >> <target dev='hdc' bus='ide'/> >> <readonly/> >> <address type='drive' controller='0' bus='1' target='0' >> unit='0'/> >> </disk> >> <controller type='usb' index='0'> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x2'/> >> </controller> >> <controller type='pci' index='0' model='pci-root'/> >> <controller type='ide' index='0'> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' >> function='0x1'/> >> </controller> >> <interface type='bridge'> >> <mac address='52:54:00:10:26:ec'/> >> <source bridge='brlan'/> >> <model type='virtio'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' >> function='0x0'/> >> </interface> >> <serial type='pty'> >> <target port='0'/> >> </serial> >> <console type='pty'> >> <target type='serial' port='0'/> >> </console> >> <input type='mouse' bus='ps2'/> >> <graphics type='vnc' port='-1' autoport='yes'/> >> <video> >> <model type='vga' vram='9216' heads='1'/> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' >> function='0x0'/> >> </video> >> <hostdev mode='subsystem' type='pci' managed='yes'> >> <source> >> <address domain='0x0000' bus='0x03' slot='0x00' >> function='0x0'/> >> </source> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' >> function='0x0'/> >> </hostdev> >> <memballoon model='virtio'> >> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' >> function='0x0'/> >> </memballoon> >> </devices> >> </domain>