Re: [libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

On Mon, 2013-04-08 at 12:37 -0400, Laine Stump wrote: > On 04/05/2013 03:26 PM, Alex Williamson wrote: > >> Sure. I just mean "pci bus 0" (hmm, but actually this does point out a > >> problem with my logic - the same namespace (well, "numbering space") is > >> used for both pcie and pci buses, so on a q35 system, bus=0 is already > >> taken by pcie.0; that means that the first pci bus would need to use a > >> different bus number anyway, so it wouldn't be so easy to switch an > >> existing domain from pc to q35 - every PCI device would need to have its > >> bus number modified. I suppose that's reasonable to expect, though. > > I would think you'd want to differentiate PCI from PCIe anyway. PCI is > > a bus and you have 32 slots per bus to fill. PCIe is a point-to-point > > link and you really only have slot 0 available. Perhaps that puts them > > in different number spaces already. > > Are you saying that it's okay to have a bus=0 for pci and a different > bus=0 for pcie? In bus=<identifier> the identifier needs to be unique, but it's not a bus #, it's just an identifier.

> I was hoping that what is used in libvirt's config could mirror as > closely as possible the numbering that you see in the output of lspci on > the guest, but it sounds like that numbering is something done at the > whim of the guest, with no basis in (standard) reality, is that right? Correct, the the BIOS determines the initial bus numbers and it can do it however it wants. Most guests won't renumber buses, but they can if they want. It's a lost cause to expect any correlation between the libvirt bus identifier and the actual bus number.

> I think we're starting to get closer to the concrete problem that's > bothering me. As I understand it (and again - "what I understand" has > repeatedly been shown to be incorrect in this thread :-): > > * Ihere are multiple different types of devices that provide a bus with > 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the > e1000 network device, etc) can be plugged into. > > * In the config for those devices, there is a required (auto-generated > if not explicitly provided) <address> element that indicates what > controller that device is plugged into e.g.: > > <interface type='direct'> > ... > <address type='pci' domain='0' bus='0' slot='3' function='0'/> > ... > </interface> The above is not really how QEMU works. QEMU PCI devices take an addr= parameter that specifies the "slot.function". The bus= option is not numeric. That's the identifier value. So if you create a bus with: -device i82801b11-bridge,id=dmi-to-pci-bridge,addr=1e.0 Then to put a device on that bus you'd do: -device e1000,id=e1000-net-0,bus=dmi-to-pci-bridge,addr=0.0

We don't have a way to generate new domains yet, but I imagine it would require a PCI host bridge device and be a parameter to that. For instance: - device pci-host-bridge,domain=1,id=domain1-pcie.0 Which would create a new "pci-root". You would then specify a device in the domain using the same bus= notation. An e1000 on bus=pcie.0 is in domain0, but a nic on bus=domain1-pcie.0 is in domain1. This is all just speculation though. > * domain is always hardcoded to 0, and in the past bus was also always > hardcoded to 0 because until now there has only been a single place > where PCI devices could be connected - the builtin pci.0 bus, which is a > part of the basic "pc" (and some others) virtual machine and provides 32 > slots. Again, bus= is an identifier and I would guess that it will implicitly specify the domain when we get that far. Libvirt specifying a numerical domain:bus:slot.fn and expecting the device to appear there to the guest is a flawed concept.