PCI/GPU Passthrough with xen
Hi All, it is possible to use pci/gpu passthrough with xen and libvirt? If yes, how is the syntax? Can I get an example? -- ------ Greetz
On 1/30/20 1:24 AM, Christoph wrote:
Hi All,
it is possible to use pci/gpu passthrough with xen and libvirt? If yes, how is the syntax? Can I get an example?
Yes, host PCI devices can be passed to xen guests using libvirt. Below is example config of a "managed" PCI host device. For more details see https://libvirt.org/formatdomain.html#elementsHostDevSubsys <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x0a' slot='0x10' function='0x1'/> </source> </hostdev> Regards, Jim
this config does not work... why? <domain type='xen'> <name>marax.chao5.int</name> <uuid>72f8f7cf-d538-41cd-828a-9945b9157719</uuid> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <vcpu placement='static'>16</vcpu> <os> <type arch='x86_64' machine='xenfv'>hvm</type> <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/xen/bin/qemu-system-i386</emulator> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/mapper/marax_c'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/vg_lilith/lv_marax_d'/> <target dev='hdc' bus='ide'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='00:16:3e:8c:97:27'/> <source bridge='xenbr5'/> <model type='e1000'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5900' autoport='no' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='xen' vram='16384' heads='1' primary='yes'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </source> </hostdev> <memballoon model='xen'/> </devices> </domain> --- ------ Greetz Am 31.01.2020 23:07, schrieb Jim Fehlig:
On 1/30/20 1:24 AM, Christoph wrote:
Hi All,
it is possible to use pci/gpu passthrough with xen and libvirt? If yes, how is the syntax? Can I get an example?
Yes, host PCI devices can be passed to xen guests using libvirt. Below is example config of a "managed" PCI host device. For more details see https://libvirt.org/formatdomain.html#elementsHostDevSubsys
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x0a' slot='0x10' function='0x1'/> </source> </hostdev>
Regards, Jim
On 2/4/20 1:04 AM, Christoph wrote:
this config does not work... why?
Without more details, I don't know why :-). In what way does it "not work"? Does 'virsh create' fail with this config? If so, what is the error? Also check /var/log/libvirt/libxl/libxl-driver.log for any errors emitted during the create process.
<domain type='xen'> <name>marax.chao5.int</name> <uuid>72f8f7cf-d538-41cd-828a-9945b9157719</uuid> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <vcpu placement='static'>16</vcpu> <os> <type arch='x86_64' machine='xenfv'>hvm</type> <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/xen/bin/qemu-system-i386</emulator> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/mapper/marax_c'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/vg_lilith/lv_marax_d'/> <target dev='hdc' bus='ide'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='00:16:3e:8c:97:27'/> <source bridge='xenbr5'/> <model type='e1000'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5900' autoport='no' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='xen' vram='16384' heads='1' primary='yes'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> </hostdev>
For debugging purposes try to passthrough only one device. Once that is working you can add the other devices. Another good debugging step would be to remove libvirt entirely and see if you can passthrough the device(s) using the xen tools https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough Regards, Jim
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </source> </hostdev> <memballoon model='xen'/> </devices> </domain>
--- ------ Greetz
Am 31.01.2020 23:07, schrieb Jim Fehlig:
On 1/30/20 1:24 AM, Christoph wrote:
Hi All,
it is possible to use pci/gpu passthrough with xen and libvirt? If yes, how is the syntax? Can I get an example?
Yes, host PCI devices can be passed to xen guests using libvirt. Below is example config of a "managed" PCI host device. For more details see https://libvirt.org/formatdomain.html#elementsHostDevSubsys
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x0a' slot='0x10' function='0x1'/> </source> </hostdev>
Regards, Jim
I know these are mostly gamers but they have a lot of experience doing PCI pass though: https://discord.gg/du9ecG I have found them extremely helpful in the past doing libvirt PCI passthough. *Paul O'Rorke* On 2020-02-05 10:13 a.m., Jim Fehlig wrote:
On 2/4/20 1:04 AM, Christoph wrote:
this config does not work... why?
Without more details, I don't know why :-). In what way does it "not work"? Does 'virsh create' fail with this config? If so, what is the error? Also check /var/log/libvirt/libxl/libxl-driver.log for any errors emitted during the create process.
<domain type='xen'> <name>marax.chao5.int</name> <uuid>72f8f7cf-d538-41cd-828a-9945b9157719</uuid> <memory unit='GiB'>32</memory> <currentMemory unit='GiB'>32</currentMemory> <vcpu placement='static'>16</vcpu> <os> <type arch='x86_64' machine='xenfv'>hvm</type> <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> </features> <clock offset='localtime'/> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <devices> <emulator>/usr/lib/xen/bin/qemu-system-i386</emulator> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/mapper/marax_c'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver type='raw'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/vg_lilith/lv_marax_d'/> <target dev='hdc' bus='ide'/> <address type='drive' controller='0' bus='1' target='0' unit='0'/> </disk> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='00:16:3e:8c:97:27'/> <source bridge='xenbr5'/> <model type='e1000'/> </interface> <serial type='pty'> <target port='0'/> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <input type='tablet' bus='usb'/> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='5900' autoport='no' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <video> <model type='xen' vram='16384' heads='1' primary='yes'/> </video> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> </hostdev>
For debugging purposes try to passthrough only one device. Once that is working you can add the other devices.
Another good debugging step would be to remove libvirt entirely and see if you can passthrough the device(s) using the xen tools
https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough
Regards, Jim
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </source> </hostdev> <memballoon model='xen'/> </devices> </domain>
--- ------ Greetz
Am 31.01.2020 23:07, schrieb Jim Fehlig:
On 1/30/20 1:24 AM, Christoph wrote:
Hi All,
it is possible to use pci/gpu passthrough with xen and libvirt? If yes, how is the syntax? Can I get an example?
Yes, host PCI devices can be passed to xen guests using libvirt. Below is example config of a "managed" PCI host device. For more details see https://libvirt.org/formatdomain.html#elementsHostDevSubsys
<hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x0a' slot='0x10' function='0x1'/> </source> </hostdev>
Regards, Jim
Hi the machine starts now but the config between libvirt (xml) and xen seems to be different, because with xen cfg it starts as it should and with libvirt conf it hangs at the boot process of win10... does someone see a difference what is responsible for the problem? Thats the xen config (working): name = "marax.chao5.int" uuid = "e0de3cb7-3937-417d-8d63-b0993b377b6a" maxmem = 32768 memory = 32768 kernel = '/usr/lib64/xen/boot/hvmloader' vcpus = 16 rtc_timeoffset = 0 localtime = 1 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vif = [ "mac=00:16:3e:05:01:10,bridge=xenbr5,script=vif-bridge,model=e1000" ] parallel = "none" serial = "none" type = "hvm" loader = "/usr/lib64/xen/boot/hvmloader" disk = [ "phy:/dev/mapper/marax_c,hda,rw", "phy:/dev/vg_lilith/lv_marax_d,hdb,rw" ] max_grant_frames = "128" pci = [ "01:00.0", "01:00.1", "01:00.2", "01:00.3" ] pci_permissive = 1 keymap = "de" vnclisten="0.0.0.0" pci_power_mgmt=1 xen_platform_pci=1 pci_msitranslate=1 viridian=1 hpet=1 acpi=1 apic=1 pae=1 thats the libvirt config (starting but win10 'hangs'): <domain type='xen'> <name>marax.chao5.int</name> <uuid>e0de3cb7-3937-417d-8d63-b0993b377b6a</uuid> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <vcpu placement='static'>16</vcpu> <os> <type arch='x86_64' machine='xenfv'>hvm</type> <loader type='rom'>/usr/lib64/xen/boot/hvmloader</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <viridian/> </features> <clock offset='variable' adjustment='0' basis='localtime'> <timer name='hpet' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/mapper/marax_c'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/vg_lilith/lv_marax_d'/> <target dev='hdb' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='xenbus' index='0'/> <controller type='ide' index='0'/> <interface type='bridge'> <mac address='00:16:3e:05:01:10'/> <source bridge='xenbr5'/> <script path='vif-bridge'/> <model type='e1000'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='6000' autoport='no' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </source> </hostdev> <memballoon model='xen'/> </devices> </domain> --- ------ Greetz
On 2/14/20 3:37 AM, Christoph wrote:
Hi
the machine starts now but the config between libvirt (xml) and xen seems to be different, because with xen cfg it starts as it should and with libvirt conf it hangs at the boot process of win10...
does someone see a difference what is responsible for the problem?
Thats the xen config (working):
name = "marax.chao5.int" uuid = "e0de3cb7-3937-417d-8d63-b0993b377b6a" maxmem = 32768 memory = 32768 kernel = '/usr/lib64/xen/boot/hvmloader' vcpus = 16 rtc_timeoffset = 0 localtime = 1 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vif = [ "mac=00:16:3e:05:01:10,bridge=xenbr5,script=vif-bridge,model=e1000" ] parallel = "none" serial = "none" type = "hvm" loader = "/usr/lib64/xen/boot/hvmloader" disk = [ "phy:/dev/mapper/marax_c,hda,rw", "phy:/dev/vg_lilith/lv_marax_d,hdb,rw" ] max_grant_frames = "128"
You increase max_grant_frames in the xen config...
pci = [ "01:00.0", "01:00.1", "01:00.2", "01:00.3" ] pci_permissive = 1 keymap = "de" vnclisten="0.0.0.0" pci_power_mgmt=1 xen_platform_pci=1 pci_msitranslate=1 viridian=1 hpet=1 acpi=1 apic=1 pae=1
thats the libvirt config (starting but win10 'hangs'):
<domain type='xen'> <name>marax.chao5.int</name> <uuid>e0de3cb7-3937-417d-8d63-b0993b377b6a</uuid> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <vcpu placement='static'>16</vcpu> <os> <type arch='x86_64' machine='xenfv'>hvm</type> <loader type='rom'>/usr/lib64/xen/boot/hvmloader</loader> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <pae/> <viridian/> </features> <clock offset='variable' adjustment='0' basis='localtime'> <timer name='hpet' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/mapper/marax_c'/> <target dev='hda' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='block' device='disk'> <driver name='phy' type='raw'/> <source dev='/dev/vg_lilith/lv_marax_d'/> <target dev='hdb' bus='ide'/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='xenbus' index='0'/>
but not in the libvirt config. It is an attribute of the xenbus controller and can be defined like this <controller type='xenbus' maxGrantFrames='128'/> For more info search for 'maxGrantFrames' in the libvirt domXML documentation https://libvirt.org/formatdomain.html Regards, Jim
<controller type='ide' index='0'/> <interface type='bridge'> <mac address='00:16:3e:05:01:10'/> <source bridge='xenbr5'/> <script path='vif-bridge'/> <model type='e1000'/> </interface> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <graphics type='vnc' port='6000' autoport='no' listen='0.0.0.0' keymap='de'> <listen type='address' address='0.0.0.0'/> </graphics> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='xen'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </source> </hostdev> <memballoon model='xen'/> </devices> </domain>
--- ------ Greetz
participants (3)
-
Christoph -
Jim Fehlig -
Paul O'Rorke