On 08/11/2015 10:13 PM, Alex Williamson wrote:
On Tue, 2015-08-11 at 19:26 -0400, Laine Stump wrote:
(Alex - I cc'ed you because I addressed a question or two your way down towards the bottom).
Hello!
The original patches to support pcie-root severely restricted what could plug into what because in real hardware you can't plug a PCI device into a PCIe slot (physically it doesn't work) But how do you know whether the device is PCI or PCIe ? I don't see anything like this in the code, i see that for example "all network cards are PCI", which is, BTW, not true in the real world. Two years ago when I first added support for q35-based machinetypes and
On 08/11/2015 02:52 AM, Pavel Fedin wrote: the first pcie controllers, I had less information than I do now. When I looked in the ouput of "qemu-kvm -device ?" I saw that each device listed the type of bus it connected to (PCI or ISA), and assumed that even though at the time qemu didn't differentiate between PCI and PCIe there, since the two things *are* different in the real world eventually they likely would. I wanted the libvirt code to be prepared for that eventuality. Of course every example device (except the PCIe controllers themselves) ends up with the flag saying that it can connect to a PCI bus, not PCIe).
Later I was told that, unlike the real world where, if nothing else, the physical slots themselves limit you, any normal PCI device in qemu could be plugged into a PCI or PCIe slot. There still are several restrictions though, which showed themselves as more complicated than just the naive PCI vs PCIe that I originally imagined - just look at the restrictions on the different PCIe controllers:
("pcie-sw-up-port" == "pcie-switch-upstream-port", "pcie-sw-dn-port" == "pcie-switch-downstream-port")
name upstream downstream ----------------- ----------------- ------------------- pcie-root none any endpoint pcie-root-port dmi-to-pci-bridge pci-bridge 31 ports NO hotplug
dmi-to-pci-bridge pcie-root any endpoint device pcie-root-port pcie-sw-up-port pcie-sw-dn-port NO hotplug 32 ports NO hotplug Hmm, pcie-sw-up-port on the downstream is a stretch here. pci-bridge should be allowed downstream though.
You're right, I messed up the chart. pcie-sw-up-port can only plug into pcie-root-port or pcie-sw-dn-port. And I forgot to add in pci-bridge. Of course my main objective was to graphically point out that you can't just plug "anything" into "anything" :-)
pcie-root-port pcie-root-only any endpoint NO hotplug dmi-to-pci-bridge pcie-sw-up-port 1 port hotpluggable
pcie-sw-up-port pcie-root-port pcie-sw-dn-port pcie-sw-dn-port 32 ports "kind of" hotpluggable "kind of" hotpluggable
pcie-sw-dn-port pcie-sw-up-port any endpoint "kind of" hotplug pcie-sw-up-port 1 port hotpluggable
pci-bridge pci-root any endpoint pcie-root pci-bridge dmi-to-pci-bridge 32 ports hotpluggable pcie-root-port pcie-sw-dn-port NO hotplug (now)
So the original restrictions I placed on what could plugin where were *too* restrictive for endpoint devices, but other restrictions were useful, and the framework came in handy as I learned the restrictions of each new pci controller model. System software ends up being pretty amiable as well since PCIe is software compatible with conventional PCI. If we have a guest-based IOMMU though, things could start to get interesting because the difference isn't so transparent. The kernel starts to care about whether a device is express and expects certain compatible upstream devices as it walks the topology. Thankfully though real hardware gets plenty wrong too, so we only have to be not substantially worse than real hardware ;)