[libvirt-users] Using hostdev to plug a PCI-E host device into Q35 pcie-root port
Hello, I'm trying to migrate a working qemu command line configuration to libvirt. The part I'm currently failing on is: $ qemu-system-x86_64 -M Q35 ... -device vfio-pci,host=05:00.0,bus=pcie.0 The right way to translate this into libvirt XML seems to be using <hostdev>, but I seem to be unable to plug it into the pcie-root port This is how the interesting part looks like when I let "virsh edit" generate an <address> <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> [...] <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </hostdev> [...] To my understanding, this will plug the host device into the pci-bridge controller. The guest OS doesn't boot with this and resets right after bios. Manually setting <address type='pci' domain='0x0000' bus='0x00' slot='0x1E' function='0x0'/> cause XML validation failure. Is there any way in libvirt XML to plug a host's PCI-E device directly into the pcie-root port, like it works on qemu command line? I'm aware I could use something like <qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/> </qemu:commandline> but I insist on running the VM as non-root, and if I got that right I need to configure at least one vfio device (or memory locking) in order for libvirt to set a proper RLIMIT_MEMLOCK value. Any help would be be appreciated. Regards, Thomas
On 11/15/2013 03:35 PM, Thomas Kuther wrote:
Hello,
I'm trying to migrate a working qemu command line configuration to libvirt. The part I'm currently failing on is:
$ qemu-system-x86_64 -M Q35 ... -device vfio-pci,host=05:00.0,bus=pcie.0
The right way to translate this into libvirt XML seems to be using <hostdev>, but I seem to be unable to plug it into the pcie-root port
This is how the interesting part looks like when I let "virsh edit" generate an <address>
<controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> [...] <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </hostdev> [...]
To my understanding, this will plug the host device into the pci-bridge controller. The guest OS doesn't boot with this and resets right after bios.
Ugh. That's very unfortunate. This is the first report I've heard of something failing in such a bad way due to being plugged into a pci-bridge slot; up until now I'd only heard that there is some extra PCIe functionality that would be missing if a device was plugged into a PCI slot vs. PCIe. Can I ask what type of device this is?
Manually setting <address type='pci' domain='0x0000' bus='0x00' slot='0x1E' function='0x0'/> cause XML validation failure.
Is there any way in libvirt XML to plug a host's PCI-E device directly into the pcie-root port, like it works on qemu command line?
I'm sorry to say, no. With very few (and specific) exceptions, libvirt insists that all guest devices be plugged into a hot-pluggable PCI slot - this eliminates both the PCIe "root complex" (a.k.a. pcie.0) as well as the dmi-to-pci-controller that is plugged into pcie.0 (because pci-to-dmi controllers' slots don't support hot-plug). This is done because, for now, almost all devices that qemu knows about are PCI (no PCI-e) devices, and if we allowed plugging them into pcie.0 now, then on the day in the future when qemu begins enforcing the difference between PCI and PCIe (currently it doesn't), the world would be full of libvirt configs that would no longer work. There was some discussion about this a month or two ago either on libvir-list or maybe it was the qemu-devel list. We decided that qemu needs to provide some sort of introspection of the devices' connection types so that libvirt can determine what device can plug into which slots; at that time we'll be able to allow exactly what's proper in each case. In the meantime we're stuck with being overly cautious in order to prevent future catastrophe.
I'm aware I could use something like
<qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/> </qemu:commandline>
but I insist on running the VM as non-root, and if I got that right I need to configure at least one vfio device (or memory locking) in order for libvirt to set a proper RLIMIT_MEMLOCK value.
Any help would be be appreciated.
For now at least, you'll need to let it plug into the pci-bridge device pci.2 (which, as you've found, libvirt will automatically find when you don't specify any address). Unfortunately that doesn't do you much good, since that particular device you're assigning actually requires that it be plugged into the PCIe bus. I'm wondering as I type if possibly we could relax the enforcement of the "PCI only" rule such that we allow explicitly placing any device on any type of bus, but only auto-assign to a plain PCI slot. That may be a reasonable compromise until qemu has the required new device/controller introspection info available.
Am 19.11.2013 11:36, schrieb Laine Stump:
On 11/15/2013 03:35 PM, Thomas Kuther wrote:
Hello,
I'm trying to migrate a working qemu command line configuration to libvirt. The part I'm currently failing on is:
$ qemu-system-x86_64 -M Q35 ... -device vfio-pci,host=05:00.0,bus=pcie.0
The right way to translate this into libvirt XML seems to be using <hostdev>, but I seem to be unable to plug it into the pcie-root port
This is how the interesting part looks like when I let "virsh edit" generate an <address>
<controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> [...] <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </hostdev> [...]
To my understanding, this will plug the host device into the pci-bridge controller. The guest OS doesn't boot with this and resets right after bios.
Ugh. That's very unfortunate. This is the first report I've heard of something failing in such a bad way due to being plugged into a pci-bridge slot; up until now I'd only heard that there is some extra PCIe functionality that would be missing if a device was plugged into a PCI slot vs. PCIe.
Can I ask what type of device this is?
It's a Marvell 88SE9172 SATA controller, here is the lspci -vvv 03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11) (prog-if 01 [AHCI 1.0]) Subsystem: Gigabyte Technology Co., Ltd Device b000 Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 47 Region 0: I/O ports at d040 [disabled] [size=8] Region 1: I/O ports at d030 [disabled] [size=4] Region 2: I/O ports at d020 [disabled] [size=8] Region 3: I/O ports at d010 [disabled] [size=4] Region 4: I/O ports at d000 [disabled] [size=16] Region 5: Memory at f7610000 (32-bit, non-prefetchable) [disabled] [size=512] Expansion ROM at f7600000 [disabled by cmd] [size=64K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D3 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit- Address: 00000000 Data: 0000 Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <512ns, L1 <64us ClockPM- Surprise- LLActRep- BwNot- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [100 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn- Kernel driver in use: vfio-pci The second one I'm trying to pass through is a Renesas uPD720201 USB 3.0 Host Controller, but first I wanted to get the SATA controller working in libvirt. I will try to leave out the SATA controller and see what happens with only the USB3 controller.
Manually setting <address type='pci' domain='0x0000' bus='0x00' slot='0x1E' function='0x0'/> cause XML validation failure.
Is there any way in libvirt XML to plug a host's PCI-E device directly into the pcie-root port, like it works on qemu command line?
I'm sorry to say, no. With very few (and specific) exceptions, libvirt insists that all guest devices be plugged into a hot-pluggable PCI slot - this eliminates both the PCIe "root complex" (a.k.a. pcie.0) as well as the dmi-to-pci-controller that is plugged into pcie.0 (because pci-to-dmi controllers' slots don't support hot-plug).
This is done because, for now, almost all devices that qemu knows about are PCI (no PCI-e) devices, and if we allowed plugging them into pcie.0 now, then on the day in the future when qemu begins enforcing the difference between PCI and PCIe (currently it doesn't), the world would be full of libvirt configs that would no longer work.
There was some discussion about this a month or two ago either on libvir-list or maybe it was the qemu-devel list. We decided that qemu needs to provide some sort of introspection of the devices' connection types so that libvirt can determine what device can plug into which slots; at that time we'll be able to allow exactly what's proper in each case. In the meantime we're stuck with being overly cautious in order to prevent future catastrophe.
Understood, thanks for the explanation.
I'm aware I could use something like
<qemu:commandline> <qemu:arg value='-device'/> <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/> </qemu:commandline>
but I insist on running the VM as non-root, and if I got that right I need to configure at least one vfio device (or memory locking) in order for libvirt to set a proper RLIMIT_MEMLOCK value.
Any help would be be appreciated.
For now at least, you'll need to let it plug into the pci-bridge device pci.2 (which, as you've found, libvirt will automatically find when you don't specify any address). Unfortunately that doesn't do you much good, since that particular device you're assigning actually requires that it be plugged into the PCIe bus.
I'm wondering as I type if possibly we could relax the enforcement of the "PCI only" rule such that we allow explicitly placing any device on any type of bus, but only auto-assign to a plain PCI slot. That may be a reasonable compromise until qemu has the required new device/controller introspection info available.
I like the idea. Regards, Thomas
Am 19.11.2013 14:51, schrieb Thomas Kuther:
Am 19.11.2013 11:36, schrieb Laine Stump:
On 11/15/2013 03:35 PM, Thomas Kuther wrote:
Hello,
I'm trying to migrate a working qemu command line configuration to libvirt. The part I'm currently failing on is:
$ qemu-system-x86_64 -M Q35 ... -device vfio-pci,host=05:00.0,bus=pcie.0
The right way to translate this into libvirt XML seems to be using <hostdev>, but I seem to be unable to plug it into the pcie-root port
This is how the interesting part looks like when I let "virsh edit" generate an <address>
<controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='dmi-to-pci-bridge'> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </controller> <controller type='pci' index='2' model='pci-bridge'> <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/> </controller> [...] <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/> </hostdev> [...]
To my understanding, this will plug the host device into the pci-bridge controller. The guest OS doesn't boot with this and resets right after bios.
Ugh. That's very unfortunate. This is the first report I've heard of something failing in such a bad way due to being plugged into a pci-bridge slot; up until now I'd only heard that there is some extra PCIe functionality that would be missing if a device was plugged into a PCI slot vs. PCIe.
Can I ask what type of device this is?
It's a Marvell 88SE9172 SATA controller, here is the lspci -vvv
[...]
The second one I'm trying to pass through is a Renesas uPD720201 USB 3.0 Host Controller, but first I wanted to get the SATA controller working in libvirt. I will try to leave out the SATA controller and see what happens with only the USB3 controller.
I tried to pass through only the USB 3 controller. The guest boots, but the controller isn't really working (means: no mouse and keyboard) Also tried again with above SATA controller just to verify, same result: the guest resets right after bios, causing a bios-reset-bios-reset loop. If it matters, the hosts runs vanilla 3.12 + libvirt-1.1.4 + qemu-1.6.1
I'm wondering as I type if possibly we could relax the enforcement of the "PCI only" rule such that we allow explicitly placing any device on any type of bus, but only auto-assign to a plain PCI slot. That may be a reasonable compromise until qemu has the required new device/controller introspection info available.
I like the idea.
I _really_ like the idea meanwhile :) ~Thomas
participants (2)
-
Laine Stump -
Thomas Kuther