2:24 a.m.
Hi All,
it is possible to use pci/gpu passthrough with xen and libvirt?
If yes, how is the syntax? Can I get an example?
--
------
Greetz
4:07 p.m.
On 1/30/20 1:24 AM, Christoph wrote:
Hi All,
it is possible to use pci/gpu passthrough with xen and libvirt?
If yes, how is the syntax? Can I get an example?
Yes, host PCI devices can be passed to xen guests using libvirt. Below is
example config of a "managed" PCI host device. For more details see
https://libvirt.org/formatdomain.html#elementsHostDevSubsys
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x0a' slot='0x10'
function='0x1'/>
</source>
</hostdev>
Regards,
Jim
2:04 a.m.
this config does not work... why?
<domain type='xen'>
<name>marax.chao5.int</name>
<uuid>72f8f7cf-d538-41cd-828a-9945b9157719</uuid>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>16</vcpu>
<os>
<type arch='x86_64' machine='xenfv'>hvm</type>
<loader type='rom'>/usr/lib/xen/boot/hvmloader</loader>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/lib/xen/bin/qemu-system-i386</emulator>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/mapper/marax_c'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver type='raw'/>
<target dev='hdb' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0'
target='0' unit='1'/>
</disk>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/vg_lilith/lv_marax_d'/>
<target dev='hdc' bus='ide'/>
<address type='drive' controller='0' bus='1'
target='0' unit='0'/>
</disk>
<controller type='ide' index='0'/>
<interface type='bridge'>
<mac address='00:16:3e:8c:97:27'/>
<source bridge='xenbr5'/>
<model type='e1000'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='no'
listen='0.0.0.0'
keymap='de'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='xen' vram='16384' heads='1'
primary='yes'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x1'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x2'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x3'/>
</source>
</hostdev>
<memballoon model='xen'/>
</devices>
</domain>
---
------
Greetz
Am 31.01.2020 23:07, schrieb Jim Fehlig:
On 1/30/20 1:24 AM, Christoph wrote:
> Hi All,
>
> it is possible to use pci/gpu passthrough with xen and libvirt?
> If yes, how is the syntax? Can I get an example?
Yes, host PCI devices can be passed to xen guests using libvirt. Below
is example config of a "managed" PCI host device. For more details see
https://libvirt.org/formatdomain.html#elementsHostDevSubsys
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x0a' slot='0x10'
function='0x1'/>
</source>
</hostdev>
Regards,
Jim
12:13 p.m.
On 2/4/20 1:04 AM, Christoph wrote:
this config does not work... why?
Without more details, I don't know why :-). In what way does it "not work"?
Does
'virsh create' fail with this config? If so, what is the error? Also check
/var/log/libvirt/libxl/libxl-driver.log for any errors emitted during the create
process.
<domain type='xen'>
<name>marax.chao5.int</name>
<uuid>72f8f7cf-d538-41cd-828a-9945b9157719</uuid>
<memory unit='GiB'>32</memory>
<currentMemory unit='GiB'>32</currentMemory>
<vcpu placement='static'>16</vcpu>
<os>
<type arch='x86_64' machine='xenfv'>hvm</type>
<loader type='rom'>/usr/lib/xen/boot/hvmloader</loader>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/lib/xen/bin/qemu-system-i386</emulator>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/mapper/marax_c'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver type='raw'/>
<target dev='hdb' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='0'
target='0' unit='1'/>
</disk>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/vg_lilith/lv_marax_d'/>
<target dev='hdc' bus='ide'/>
<address type='drive' controller='0' bus='1'
target='0' unit='0'/>
</disk>
<controller type='ide' index='0'/>
<interface type='bridge'>
<mac address='00:16:3e:8c:97:27'/>
<source bridge='xenbr5'/>
<model type='e1000'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='5900' autoport='no'
listen='0.0.0.0'
keymap='de'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='xen' vram='16384' heads='1'
primary='yes'/>
</video>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</source>
</hostdev>
For debugging purposes try to passthrough only one device. Once that is working
you can add the other devices.
Another good debugging step would be to remove libvirt entirely and see if you
can passthrough the device(s) using the xen tools
https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough
Regards,
Jim
<hostdev mode='subsystem' type='pci'
managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x1'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x2'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x3'/>
</source>
</hostdev>
<memballoon model='xen'/>
</devices>
</domain>
---
------
Greetz
Am 31.01.2020 23:07, schrieb Jim Fehlig:
> On 1/30/20 1:24 AM, Christoph wrote:
>> Hi All,
>>
>> it is possible to use pci/gpu passthrough with xen and libvirt?
>> If yes, how is the syntax? Can I get an example?
>
> Yes, host PCI devices can be passed to xen guests using libvirt. Below
> is example config of a "managed" PCI host device. For more details see
> https://libvirt.org/formatdomain.html#elementsHostDevSubsys
>
> <hostdev mode='subsystem' type='pci'
managed='yes'>
> <driver name='xen'/>
> <source>
> <address domain='0x0000' bus='0x0a' slot='0x10'
> function='0x1'/>
> </source>
> </hostdev>
>
> Regards,
> Jim
4:20 p.m.
I know these are mostly gamers but they have a lot of experience doing
PCI pass though: https://discord.gg/du9ecG
I have found them extremely helpful in the past doing libvirt PCI
passthough.
*Paul O'Rorke*
On 2020-02-05 10:13 a.m., Jim Fehlig wrote:
On 2/4/20 1:04 AM, Christoph wrote:
> this config does not work... why?
Without more details, I don't know why :-). In what way does it "not
work"? Does 'virsh create' fail with this config? If so, what is the
error? Also check /var/log/libvirt/libxl/libxl-driver.log for any
errors emitted during the create process.
> <domain type='xen'>
> <name>marax.chao5.int</name>
> <uuid>72f8f7cf-d538-41cd-828a-9945b9157719</uuid>
> <memory unit='GiB'>32</memory>
> <currentMemory unit='GiB'>32</currentMemory>
> <vcpu placement='static'>16</vcpu>
> <os>
> <type arch='x86_64' machine='xenfv'>hvm</type>
> <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader>
> <boot dev='hd'/>
> </os>
> <features>
> <acpi/>
> <apic/>
> <pae/>
> </features>
> <clock offset='localtime'/>
> <on_poweroff>destroy</on_poweroff>
> <on_reboot>restart</on_reboot>
> <on_crash>destroy</on_crash>
> <devices>
> <emulator>/usr/lib/xen/bin/qemu-system-i386</emulator>
> <disk type='block' device='disk'>
> <driver name='phy' type='raw'/>
> <source dev='/dev/mapper/marax_c'/>
> <target dev='hda' bus='ide'/>
> <address type='drive' controller='0' bus='0'
target='0'
> unit='0'/>
> </disk>
> <disk type='file' device='cdrom'>
> <driver type='raw'/>
> <target dev='hdb' bus='ide'/>
> <readonly/>
> <address type='drive' controller='0' bus='0'
target='0'
> unit='1'/>
> </disk>
> <disk type='block' device='disk'>
> <driver name='phy' type='raw'/>
> <source dev='/dev/vg_lilith/lv_marax_d'/>
> <target dev='hdc' bus='ide'/>
> <address type='drive' controller='0' bus='1'
target='0'
> unit='0'/>
> </disk>
> <controller type='ide' index='0'/>
> <interface type='bridge'>
> <mac address='00:16:3e:8c:97:27'/>
> <source bridge='xenbr5'/>
> <model type='e1000'/>
> </interface>
> <serial type='pty'>
> <target port='0'/>
> </serial>
> <console type='pty'>
> <target type='serial' port='0'/>
> </console>
> <input type='tablet' bus='usb'/>
> <input type='mouse' bus='ps2'/>
> <input type='keyboard' bus='ps2'/>
> <graphics type='vnc' port='5900' autoport='no'
listen='0.0.0.0'
> keymap='de'>
> <listen type='address' address='0.0.0.0'/>
> </graphics>
> <video>
> <model type='xen' vram='16384' heads='1'
primary='yes'/>
> </video>
> <hostdev mode='subsystem' type='pci'
managed='yes'>
> <driver name='xen'/>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00'
> function='0x0'/>
> </source>
> </hostdev>
For debugging purposes try to passthrough only one device. Once that
is working you can add the other devices.
Another good debugging step would be to remove libvirt entirely and
see if you can passthrough the device(s) using the xen tools
https://wiki.xenproject.org/wiki/Xen_PCI_Passthrough
Regards,
Jim
> <hostdev mode='subsystem' type='pci'
managed='yes'>
> <driver name='xen'/>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00'
> function='0x1'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='pci'
managed='yes'>
> <driver name='xen'/>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00'
> function='0x2'/>
> </source>
> </hostdev>
> <hostdev mode='subsystem' type='pci'
managed='yes'>
> <driver name='xen'/>
> <source>
> <address domain='0x0000' bus='0x01' slot='0x00'
> function='0x3'/>
> </source>
> </hostdev>
> <memballoon model='xen'/>
> </devices>
> </domain>
>
>
> ---
> ------
> Greetz
>
> Am 31.01.2020 23:07, schrieb Jim Fehlig:
>> On 1/30/20 1:24 AM, Christoph wrote:
>>> Hi All,
>>>
>>> it is possible to use pci/gpu passthrough with xen and libvirt?
>>> If yes, how is the syntax? Can I get an example?
>>
>> Yes, host PCI devices can be passed to xen guests using libvirt. Below
>> is example config of a "managed" PCI host device. For more details see
>> https://libvirt.org/formatdomain.html#elementsHostDevSubsys
>>
>> <hostdev mode='subsystem' type='pci'
managed='yes'>
>> <driver name='xen'/>
>> <source>
>> <address domain='0x0000' bus='0x0a'
slot='0x10'
>> function='0x1'/>
>> </source>
>> </hostdev>
>>
>> Regards,
>> Jim
4:37 a.m.
Hi
the machine starts now but the config between libvirt (xml) and xen
seems to be different, because with xen cfg it starts as it should and
with libvirt conf it hangs at the boot process of win10...
does someone see a difference what is responsible for the problem?
Thats the xen config (working):
name = "marax.chao5.int"
uuid = "e0de3cb7-3937-417d-8d63-b0993b377b6a"
maxmem = 32768
memory = 32768
kernel = '/usr/lib64/xen/boot/hvmloader'
vcpus = 16
rtc_timeoffset = 0
localtime = 1
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vif = [
"mac=00:16:3e:05:01:10,bridge=xenbr5,script=vif-bridge,model=e1000" ]
parallel = "none"
serial = "none"
type = "hvm"
loader = "/usr/lib64/xen/boot/hvmloader"
disk = [ "phy:/dev/mapper/marax_c,hda,rw",
"phy:/dev/vg_lilith/lv_marax_d,hdb,rw" ]
max_grant_frames = "128"
pci = [ "01:00.0", "01:00.1", "01:00.2", "01:00.3"
]
pci_permissive = 1
keymap = "de"
vnclisten="0.0.0.0"
pci_power_mgmt=1
xen_platform_pci=1
pci_msitranslate=1
viridian=1
hpet=1
acpi=1
apic=1
pae=1
thats the libvirt config (starting but win10 'hangs'):
<domain type='xen'>
<name>marax.chao5.int</name>
<uuid>e0de3cb7-3937-417d-8d63-b0993b377b6a</uuid>
<memory unit='KiB'>33554432</memory>
<currentMemory unit='KiB'>33554432</currentMemory>
<vcpu placement='static'>16</vcpu>
<os>
<type arch='x86_64' machine='xenfv'>hvm</type>
<loader type='rom'>/usr/lib64/xen/boot/hvmloader</loader>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<viridian/>
</features>
<clock offset='variable' adjustment='0'
basis='localtime'>
<timer name='hpet' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/mapper/marax_c'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/vg_lilith/lv_marax_d'/>
<target dev='hdb' bus='ide'/>
<address type='drive' controller='0' bus='0'
target='0' unit='1'/>
</disk>
<controller type='xenbus' index='0'/>
<controller type='ide' index='0'/>
<interface type='bridge'>
<mac address='00:16:3e:05:01:10'/>
<source bridge='xenbr5'/>
<script path='vif-bridge'/>
<model type='e1000'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='6000' autoport='no'
listen='0.0.0.0'
keymap='de'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x1'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x2'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x3'/>
</source>
</hostdev>
<memballoon model='xen'/>
</devices>
</domain>
---
------
Greetz
3:30 p.m.
On 2/14/20 3:37 AM, Christoph wrote:
Hi
the machine starts now but the config between libvirt (xml) and xen
seems to be different, because with xen cfg it starts as it should and
with libvirt conf it hangs at the boot process of win10...
does someone see a difference what is responsible for the problem?
Thats the xen config (working):
name = "marax.chao5.int"
uuid = "e0de3cb7-3937-417d-8d63-b0993b377b6a"
maxmem = 32768
memory = 32768
kernel = '/usr/lib64/xen/boot/hvmloader'
vcpus = 16
rtc_timeoffset = 0
localtime = 1
on_poweroff = "destroy"
on_reboot = "restart"
on_crash = "restart"
vif = [
"mac=00:16:3e:05:01:10,bridge=xenbr5,script=vif-bridge,model=e1000" ]
parallel = "none"
serial = "none"
type = "hvm"
loader = "/usr/lib64/xen/boot/hvmloader"
disk = [ "phy:/dev/mapper/marax_c,hda,rw",
"phy:/dev/vg_lilith/lv_marax_d,hdb,rw" ]
max_grant_frames = "128"
You increase max_grant_frames in the xen config...
pci = [ "01:00.0", "01:00.1",
"01:00.2", "01:00.3" ]
pci_permissive = 1
keymap = "de"
vnclisten="0.0.0.0"
pci_power_mgmt=1
xen_platform_pci=1
pci_msitranslate=1
viridian=1
hpet=1
acpi=1
apic=1
pae=1
thats the libvirt config (starting but win10 'hangs'):
<domain type='xen'>
<name>marax.chao5.int</name>
<uuid>e0de3cb7-3937-417d-8d63-b0993b377b6a</uuid>
<memory unit='KiB'>33554432</memory>
<currentMemory unit='KiB'>33554432</currentMemory>
<vcpu placement='static'>16</vcpu>
<os>
<type arch='x86_64' machine='xenfv'>hvm</type>
<loader type='rom'>/usr/lib64/xen/boot/hvmloader</loader>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
<viridian/>
</features>
<clock offset='variable' adjustment='0'
basis='localtime'>
<timer name='hpet' present='yes'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/mapper/marax_c'/>
<target dev='hda' bus='ide'/>
<address type='drive' controller='0' bus='0'
target='0' unit='0'/>
</disk>
<disk type='block' device='disk'>
<driver name='phy' type='raw'/>
<source dev='/dev/vg_lilith/lv_marax_d'/>
<target dev='hdb' bus='ide'/>
<address type='drive' controller='0' bus='0'
target='0' unit='1'/>
</disk>
<controller type='xenbus' index='0'/>
but not in the libvirt config. It is an attribute of the xenbus controller and
can be defined like this
<controller type='xenbus' maxGrantFrames='128'/>
For more info search for 'maxGrantFrames' in the libvirt domXML documentation
https://libvirt.org/formatdomain.html
Regards,
Jim
<controller type='ide' index='0'/>
<interface type='bridge'>
<mac address='00:16:3e:05:01:10'/>
<source bridge='xenbr5'/>
<script path='vif-bridge'/>
<model type='e1000'/>
</interface>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='6000' autoport='no'
listen='0.0.0.0'
keymap='de'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x0'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x1'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x2'/>
</source>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='xen'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00'
function='0x3'/>
</source>
</hostdev>
<memballoon model='xen'/>
</devices>
</domain>
---
------
Greetz
1624
days inactive
1639
days old
6 comments
3 participants
participants (3)
-
Christoph -
Jim Fehlig -
Paul O'Rorke