>But in general it's correct that this isn't anything libvirt can easily fix.
>
>Rather than trying to make hotplug to a pcie-to-pci-bridge work, the
>*proper* fix is to just allow the virtio devices to be hotplugged to
>pcie-root-ports, which is what libvirt would do by default. Then they
>will be recognized as virtio-1.0 devices and use a mode that doesn't
>require any memory to be mapped for them.
>
Do I have to add as many pcie-root-ports as posibble in advance?
At 2022-11-08 02:53:55, "Laine Stump" <laine@laine.org> wrote:
>On 11/6/22 3:28 AM, longguang.yue wrote:
>> Hi, Laine Stump and Oleg Vasilev:
>> 1. Laine, what is 'conventional PCI device'? in my case they are all
>> virtio device.
>
>A "conventional PCI" device is one that is designed to work with the
>original PCI spec, i.e. before PCI Express. Among other differences,
>they lack a PCIe capabilities section in their config data.
>
>All virtio devices are PCI Express devices or conventional PCI,
>depending on the type of slot they are plugged into (that's my
>understanding of it anyway). Most other devices are always just one or
>the other.
>
>On real hardware, a PCIe device could never be plugged into a
>conventional PCI slot, nor could a conventional PCI device be plugged
>into a PCIe slot. QEMU allows this though, and just kind of glosses over
>the rough spots. It can lead to problems though, e.g. the PCIe extended
>capabilities space is not visible and so things like bus width/speed are
>not communicated properly to the guest OS (and can't be changed).
>
>> 2. Followed your guide,succeed in hotplug. but there are lots of
>> problems after hotplug. I think problems are more related to qemu.
>> 2.1) disk devices(qcow2,virio) hotpluged onto pci-bridge can not
>> alloc bar resource.
>
>When you plug a virtio device into a conventional PCI slot, it uses the
>older (pre-virtio-1.0) method of host<->device communication, which
>memory-maps a region. However, if a pci-bridge has no devices plugged
>into it when the guest is started, then the BIOS won't reserve space for
>this memory-mapped region (or something like that - I never retain
>details). Because it's using the older conventional PCI virtio that
>requires this region to be mapped for proper operation, it fails.
>
>> 2.2) disk devices(qcow2,virio) are staticly pluged to pci-bridge can
>> not be found by guest os.
>> those devices are on pci bus, but can not be recognized by
>> guest os, no block device are probed.
>
>No idea about that.
>
>But in general it's correct that this isn't anything libvirt can easily fix.
>
>Rather than trying to make hotplug to a pcie-to-pci-bridge work, the
>*proper* fix is to just allow the virtio devices to be hotplugged to
>pcie-root-ports, which is what libvirt would do by default. Then they
>will be recognized as virtio-1.0 devices and use a mode that doesn't
>require any memory to be mapped for them.
>
>
>> qemu qemu-6.2.0
>> -----hotplug error message--------
>> [ 1067.577384] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000
>> [ 1067.578312] pci 0000:06:02.0: reg 0x10: [io 0x0000-0x007f]
>> [ 1067.578794] pci 0000:06:02.0: reg 0x14: [mem 0x00000000-0x00000fff]
>> [ 1067.579419] pci 0000:06:02.0: reg 0x20: [mem 0x00000000-0x00003fff
>> 64bit pref]
>> [ 1067.581592] pci 0000:06:02.0: BAR 4: no space for [mem size
>> 0x00004000 64bit pref]
>> [ 1067.581942] pci 0000:06:02.0: BAR 4: failed to assign [mem size
>> 0x00004000 64bit pref]
>> [ 1067.582264] pci 0000:06:02.0: BAR 1: no space for [mem size 0x00001000]
>> [ 1067.582625] pci 0000:06:02.0: BAR 1: failed to assign [mem size
>> 0x00001000]
>> [ 1067.582942] pci 0000:06:02.0: BAR 0: no space for [io size 0x0080]
>> [ 1067.583253] pci 0000:06:02.0: BAR 0: failed to assign [io size 0x0080]
>> [ 1067.584019] PCI Interrupt Link [GSIE] enabled at IRQ 20
>> [ 1067.584749] virtio-pci 0000:06:02.0: virtio_pci: leaving for legacy
>> driver
>> [ 1067.585296] virtio-pci: probe of 0000:06:02.0 failed with error -12
>>
>> -------staticly plug------------
>> [ 0.680299] pci 0000:06:02.0: [1af4:1001] type 00 class 0x010000
>> [ 0.683035] pci 0000:06:02.0: reg 0x10: [io 0xc080-0xc0ff]
>> [ 0.688036] pci 0000:06:02.0: reg 0x14: [mem 0xfca01000-0xfca01fff]
>> [ 0.700040] pci 0000:06:02.0: reg 0x20: [mem 0xfb404000-0xfb407fff 64bit
>> pref]
>>
>> --------------------------------------
>>
>>
>>
>> At 2022-11-04 05:28:47, "Laine Stump" <laine@laine.org> wrote:
>>>On 11/3/22 6:35 AM, longguang.yue wrote:
>>>> Hi, all
>>>> I have add two pcie-to-pci-bridge controllers, how hotplug devices
>>>> onto this two controllers? and libvirt knows to plug onto pci-bridge
>>>> when there are no pcie-ports.
>>>> defaultly libvirt plugs devices onto pcie-port.
>>>
>>>Do you mean you want hotplugged devices to be placed on the
>>>pcie-to-pci-bridge even though there are unused pci-root-ports?
>>>
>>>You can force a new device to be hotplugged to a particular slot of a
>>>particular controller by including the <address> element in the XML of
>>>the device you attach - whatever controller/slot/function you specify
>>>there is what libvirt will use (practically speaking the function must
>>>be 0).
>>>
>>>My recollection is that, *if the device is a conventional PCI device
>>>(and not a PCIe device) then libvirt will anyway attempt to find a an
>>>open conventional PCI slot, which would be found only on the
>>>pcie-to-pci-bridge. However, very few devices are actually conventional
>>>PCI, so this will very rarely happen automatically (I haven't looked at
>>>that code in a long time, and retain no specific memory, but I think the
>>>only emulated device that is conventional PCI is a watchdog device or
>>>something)
>>>
>>>So, if libvirt is selecting a pcie-root-port to plug in a device, that's
>>>because the device is a PCIe device. If you want to force it to do
>>>otherwise, you'll need to manually specify the address (just set "bus"
>>>of the address to the "index" of the controller, set "slot" to an
>>>unoccupied slot (1-31), and set domain and function to 0).