Re: [libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: ARGHH!! Just when I think I'm starting to understand *something* about these devices... (later edit: after some coaching on IRC, I *think* I've got a bit better handle on it.) There is a problem here - within a given controller type, we will now have the possibility of multiple controllers with the same index - the differentiating attribute will be in the <address> subelement, which could create some awkwardness. Maybe instead this should be handled with a different model of pci controller, and we can add a "domain" attribute at the toplevel rather than specifying an <address>? (The descriptions of different models below tell what each of these other devices does; in short, they're all just some sort of electronic Lego to help connect PCI and PCIe devices into a tree). Okay, I'll make yet another attempt at understanding these devices, and suggesting how they can all be described in the XML. I'm thinking that *all* of the express hubs, switch ports, bridges, etc can be described in xml in the manner above, i.e.: <controller type='pci' index='n'> <model type='xxx'/> </controller> and that the method for connecting a device to any of them would be by specifying: <address type='pci' domain='n' bus='n' slot='n' function='n'/> Any limitations about which devices/controllers can connect to which controllers, and how many devices can connect to any particular controller will be derived from the <model type='xxx'/>. (And, as we've said before, although qemu doesn't assign each of these controllers a numeric bus id, and although we can make no guarantee that the bus id we use for a particular controller is what will be used by the guest BIOS/OS, it's still a convenient notation and works well with other hypervisors as well as qemu. I'll also note that when I run lspci on an X58-based machine I have here, *all* of the relationships between all the devices listed below are described with simple bus:slot.function numbers.) Here is a list of the pci controller model types and their restrictions (thanks to mst and aw for repeating these over and over to me; I'm sure I still have made mistakes, but at least it's getting closer). <controller type='pci-root'> ============================ Upstream: nothing Downstream: only a single pci-root-bus (implied) qemu commandline: nothing (it's implied in the q35 machinetype) Explanation: Each machine will have a different controller called "pci-root" as outlined above by Daniel. Two types of pci-root will be supported: i440FX and q35. If a pci-root is not spelled out in the config, one will be auto-added (depending on machinetype). An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and any bridge that has an address of slot='0' on its own bus is, by definition, connected to a pci-root controller - the two are matched by setting "domain" in the address of the pci-bridge to "index" of the pci-root). This bridge can only have PCI devices added. A q35 pci-root also implies a different kind of pci-bridge device - one that can only have PCIe devices/controllers attached, but is otherwise identical to the pci-bridge added for i440FX. This bus will be called "root-bus" (Note that there are generally followed conventions for what can be connected to which slot on this bus, and we will probably follow those conventions when building a machine, *but* we will not hardcode this convention into libvirt; each q35 machine will be an empty slate) <controller type='pci'> ======================= This will be used for *all* of the following controller devices supported by qemu: <model type='pcie-root-bus'/> (implicit/integrated) ---------------------------- Upstream: connect to pci-root controller *only* Downstream: 32 slots, PCIe devices only, no hotplug. qemu commandline: nothing (implicit in the q35-* machinetype) This controller is the bus described above that connects to a q35's pci-root, and provides places for PCIe devices to connect. Examples are root-ports, dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet devices (do any of those that can be connected to the pcie-root-bus exist yet?). There is only one of these controllers, and it will *always* be index='0', and will always have the following address: <address type='pci' domain='0' bus='0' slot='0' function='0'/> <model type='root-port'/> (ioh3420) ------------------------- Upstream: PCIe, connect to pcie-root-bus *only* (?) Downstream: 1 slot, PCIe devices only (?) qemu commandline: -device ioh3420,... These can only connect to the "pcie-root-bus" of of a q35 (implying that this bus will need to have a different model name than the simple "pci-bridge" <model type='dmi-to-pci-bridge'/> (i82801b11-bridge) --------------------------------- (btw, what does "dmi" mean?) Upstream: pcie-root-bus *only* Downstream: 32 slots, any PCI device, no hotplug (?) qemu commandline: -device i82801b11-bridge,... <model type='upstream-switch-port'/> (x3130-upstream) ------------------------------------ Upstream: PCIe, connect to pcie-root-bus, root-port, or downstream-switch-port (?) Downstream: 32 slots, connect *only* to downstream-switch-port qemu-commandline: -device x3130-upstream This is the upper side of a switch that can multiplex multiple devices onto a single port. It's only useful when one or more downstream switch ports are connected to it. <model type='downstream-switch-port'/> (xio3130-downstream) -------------------------------------- Upstream: connect *only* to upstream-switch-port Downstream: 1 slot, any PCIe device qemu commandline: -device xio3130-downstream You can connect one or more of these to an upstream-switch-port in order to effectively plug multiple devices into a single PCIe port. <model type='pci-bridge'/> (pci-bridge) -------------------------- Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3) another pci-bridge Downstream: any PCI device, 32 slots qemu commandline: -device pci-bridge,... This differs from dmi-to-pci-bridge in that its upstream connection is PCI rather than PCIe (so it will work on an i440FX system, which has no root PCIe bus) and that hotplug is supported. In general, if a guest will have any PCI devices, one of these controllers should be added, and =============================================================== Comment: I'm not quite convinced that we really need the separate "pci-root" device. Since 1) every pci-root will *always* have either a pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will only ever be connected to the pci-root, and 3) the pci-bridge that connects to it will need special handling within the pci-bridge case anyway, why not: 1) eliminate the separate pci-root controller type 2) within <controller type='pci'>, a new <model type='pci-root-bus'/> will be added. 3) a pcie-root-bus will automatically be added for q35 machinetypes, and pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*") 4) model type='pci-root-bus' will behave like pci-bridge, except that it will be an implicit device (nothing on qemu commandline) and it won't need an <address> element (neither will pcie-root-bus). 5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller.