Hi, all   I have add two pcie-to-pci-bridge controllers, how hotplug devices onto this two controllers?  and libvirt knows to plug onto pci-bridge when there are no pcie-ports. defaultly libvirt plugs devices onto pcie-port.

Hi, Laine Stump and Oleg Vasilev: 1.  Laine, what is 'conventional PCI device'? in my case they are all virtio device.

2. Followed your guide,succeed in hotplug.  but there are lots of problems after hotplug. I think problems are more related to qemu.   2.1) disk devices(qcow2,virio) hotpluged onto pci-bridge can not alloc bar resource.

  2.2) disk devices(qcow2,virio) are staticly pluged to pci-bridge can not be found by guest os.          those devices are on pci bus, but can not be recognized by guest os, no block device are probed.

qemu qemu-6.2.0 -----hotplug error message-------- [ 1067.577384] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000 [ 1067.578312] pci 0000:06:02.0: reg 0x10: [io 0x0000-0x007f] [ 1067.578794] pci 0000:06:02.0: reg 0x14: [mem 0x00000000-0x00000fff] [ 1067.579419] pci 0000:06:02.0: reg 0x20: [mem 0x00000000-0x00003fff 64bit pref] [ 1067.581592] pci 0000:06:02.0: BAR 4: no space for [mem size 0x00004000 64bit pref] [ 1067.581942] pci 0000:06:02.0: BAR 4: failed to assign [mem size 0x00004000 64bit pref] [ 1067.582264] pci 0000:06:02.0: BAR 1: no space for [mem size 0x00001000] [ 1067.582625] pci 0000:06:02.0: BAR 1: failed to assign [mem size 0x00001000] [ 1067.582942] pci 0000:06:02.0: BAR 0: no space for [io size 0x0080] [ 1067.583253] pci 0000:06:02.0: BAR 0: failed to assign [io size 0x0080] [ 1067.584019] PCI Interrupt Link [GSIE] enabled at IRQ 20 [ 1067.584749] virtio-pci 0000:06:02.0: virtio_pci: leaving for legacy driver [ 1067.585296] virtio-pci: probe of 0000:06:02.0 failed with error -12 -------staticly plug------------ [ 0.680299] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000 [ 0.683035] pci 0000:06:02.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.688036] pci 0000:06:02.0: reg 0x14: [mem 0xfca01000-0xfca01fff] [ 0.700040] pci 0000:06:02.0: reg 0x20: [mem 0xfb404000-0xfb407fff 64bit pref] -------------------------------------- At 2022-11-04 05:28:47, "Laine Stump" <laine(a)laine.org&gt; wrote: >On 11/3/22 6:35 AM, longguang.yue wrote: >> Hi, all >>   I have add two pcie-to-pci-bridge controllers, how hotplug devices >> onto this two controllers?  and libvirt knows to plug onto pci-bridge >> when there are no pcie-ports. >> defaultly libvirt plugs devices onto pcie-port. > >Do you mean you want hotplugged devices to be placed on the >pcie-to-pci-bridge even though there are unused pci-root-ports? > >You can force a new device to be hotplugged to a particular slot of a >particular controller by including the <address> element in the XML of >the device you attach - whatever controller/slot/function you specify >there is what libvirt will use (practically speaking the function must >be 0). > >My recollection is that, *if the device is a conventional PCI device >(and not a PCIe device) then libvirt will anyway attempt to find a an >open conventional PCI slot, which would be found only on the >pcie-to-pci-bridge. However, very few devices are actually conventional >PCI, so this will very rarely happen automatically (I haven't looked at >that code in a long time, and retain no specific memory, but I think the >only emulated device that is conventional PCI is a watchdog device or >something) > >So, if libvirt is selecting a pcie-root-port to plug in a device, that's >because the device is a PCIe device. If you want to force it to do >otherwise, you'll need to manually specify the address (just set "bus" >of the address to the "index" of the controller, set "slot" to an >unoccupied slot (1-31), and set domain and function to 0).

>But in general it's correct that this isn't anything libvirt can easily fix. > >Rather than trying to make hotplug to a pcie-to-pci-bridge work, the >*proper* fix is to just allow the virtio devices to be hotplugged to >pcie-root-ports, which is what libvirt would do by default. Then they >will be recognized as virtio-1.0 devices and use a mode that doesn't >require any memory to be mapped for them. > Do I have to add as many pcie-root-ports as posibble in advance?

At 2022-11-08 02:53:55, "Laine Stump" <laine(a)laine.org&gt; wrote: >On 11/6/22 3:28 AM, longguang.yue wrote: >> Hi, Laine Stump and Oleg Vasilev: >> 1.  Laine, what is 'conventional PCI device'? in my case they are all >> virtio device. > >A "conventional PCI" device is one that is designed to work with the >original PCI spec, i.e. before PCI Express. Among other differences, >they lack a PCIe capabilities section in their config data. > >All virtio devices are PCI Express devices or conventional PCI, >depending on the type of slot they are plugged into (that's my >understanding of it anyway). Most other devices are always just one or >the other. > >On real hardware, a PCIe device could never be plugged into a >conventional PCI slot, nor could a conventional PCI device be plugged >into a PCIe slot. QEMU allows this though, and just kind of glosses over >the rough spots. It can lead to problems though, e.g. the PCIe extended >capabilities space is not visible and so things like bus width/speed are >not communicated properly to the guest OS (and can't be changed). > >> 2. Followed your guide,succeed in hotplug.  but there are lots of >> problems after hotplug. I think problems are more related to qemu. >>   2.1) disk devices(qcow2,virio) hotpluged onto pci-bridge can not >> alloc bar resource. > >When you plug a virtio device into a conventional PCI slot, it uses the >older (pre-virtio-1.0) method of host<->device communication, which >memory-maps a region. However, if a pci-bridge has no devices plugged >into it when the guest is started, then the BIOS won't reserve space for >this memory-mapped region (or something like that - I never retain >details). Because it's using the older conventional PCI virtio that >requires this region to be mapped for proper operation, it fails. > >>   2.2) disk devices(qcow2,virio) are staticly pluged to pci-bridge can >> not be found by guest os. >>          those devices are on pci bus, but can not be recognized by >> guest os, no block device are probed. > >No idea about that. > >But in general it's correct that this isn't anything libvirt can easily fix. > >Rather than trying to make hotplug to a pcie-to-pci-bridge work, the >*proper* fix is to just allow the virtio devices to be hotplugged to >pcie-root-ports, which is what libvirt would do by default. Then they >will be recognized as virtio-1.0 devices and use a mode that doesn't >require any memory to be mapped for them. > > >> qemu qemu-6.2.0 >> -----hotplug error message-------- >> [ 1067.577384] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000 >> [ 1067.578312] pci 0000:06:02.0: reg 0x10: [io 0x0000-0x007f] >> [ 1067.578794] pci 0000:06:02.0: reg 0x14: [mem 0x00000000-0x00000fff] >> [ 1067.579419] pci 0000:06:02.0: reg 0x20: [mem 0x00000000-0x00003fff >> 64bit pref] >> [ 1067.581592] pci 0000:06:02.0: BAR 4: no space for [mem size >> 0x00004000 64bit pref] >> [ 1067.581942] pci 0000:06:02.0: BAR 4: failed to assign [mem size >> 0x00004000 64bit pref] >> [ 1067.582264] pci 0000:06:02.0: BAR 1: no space for [mem size 0x00001000] >> [ 1067.582625] pci 0000:06:02.0: BAR 1: failed to assign [mem size >> 0x00001000] >> [ 1067.582942] pci 0000:06:02.0: BAR 0: no space for [io size 0x0080] >> [ 1067.583253] pci 0000:06:02.0: BAR 0: failed to assign [io size 0x0080] >> [ 1067.584019] PCI Interrupt Link [GSIE] enabled at IRQ 20 >> [ 1067.584749] virtio-pci 0000:06:02.0: virtio_pci: leaving for legacy >> driver >> [ 1067.585296] virtio-pci: probe of 0000:06:02.0 failed with error -12 >> >> -------staticly plug------------ >> [ 0.680299] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000 >> [ 0.683035] pci 0000:06:02.0: reg 0x10: [io 0xc080-0xc0ff] >> [ 0.688036] pci 0000:06:02.0: reg 0x14: [mem 0xfca01000-0xfca01fff] >> [ 0.700040] pci 0000:06:02.0: reg 0x20: [mem 0xfb404000-0xfb407fff 64bit >> pref] >> >> -------------------------------------- >> >> >> >> At 2022-11-04 05:28:47, "Laine Stump" <laine(a)laine.org&gt; wrote: >>>On 11/3/22 6:35 AM, longguang.yue wrote: >>>> Hi, all >>>>   I have add two pcie-to-pci-bridge controllers, how hotplug devices >>>> onto this two controllers?  and libvirt knows to plug onto pci-bridge >>>> when there are no pcie-ports. >>>> defaultly libvirt plugs devices onto pcie-port. >>> >>>Do you mean you want hotplugged devices to be placed on the >>>pcie-to-pci-bridge even though there are unused pci-root-ports? >>> >>>You can force a new device to be hotplugged to a particular slot of a >>>particular controller by including the <address> element in the XML of >>>the device you attach - whatever controller/slot/function you specify >>>there is what libvirt will use (practically speaking the function must >>>be 0). >>> >>>My recollection is that, *if the device is a conventional PCI device >>>(and not a PCIe device) then libvirt will anyway attempt to find a an >>>open conventional PCI slot, which would be found only on the >>>pcie-to-pci-bridge. However, very few devices are actually conventional >>>PCI, so this will very rarely happen automatically (I haven't looked at >>>that code in a long time, and retain no specific memory, but I think the >>>only emulated device that is conventional PCI is a watchdog device or >>>something) >>> >>>So, if libvirt is selecting a pcie-root-port to plug in a device, that's >>>because the device is a PCIe device. If you want to force it to do >>>otherwise, you'll need to manually specify the address (just set "bus" >>>of the address to the "index" of the controller, set "slot" to an >>>unoccupied slot (1-31), and set domain and function to 0).