Re: [libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

On 04/08/2013 12:48 PM, Daniel P. Berrange wrote: > On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: >> I think we're starting to get closer to the concrete problem that's >> bothering me. As I understand it (and again - "what I understand" has >> repeatedly been shown to be incorrect in this thread :-): >> >> * Ihere are multiple different types of devices that provide a bus with >> 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the >> e1000 network device, etc) can be plugged into. >> >> * In the config for those devices, there is a required (auto-generated >> if not explicitly provided) <address> element that indicates what >> controller that device is plugged into e.g.: >> >> <interface type='direct'> >> ... >> <address type='pci' domain='0' bus='0' slot='3' function='0'/> >> ... >> </interface> >> >> * domain is always hardcoded to 0, and in the past bus was also always >> hardcoded to 0 because until now there has only been a single place >> where PCI devices could be connected - the builtin pci.0 bus, which is a >> part of the basic "pc" (and some others) virtual machine and provides 32 >> slots. >> >> * Now we are adding the ability to define new PCI buses, for now just a >> single kind - a pci-bridge controller, which itself must connect to an >> existing PCI slot, and provides 32 new PCI slots. But in the future >> there will be more different types of controllers that provide one or >> more PCI slots where PCI devices/controllers can be plugged in. >> >> * In these patches adding support for pci-bridge, we are making the >> assumption that there is a 1:1 correspondence between the "index='n'" >> attribute of the pci-bridge controller and the "bus='n'" attribute of >> the <address> element in devices that will be plugged into that >> controller. So for example if we have: >> >> >> <controller type='pci-bridge' index='1'> >> <address type='pci' domain='0' bus='0' slot='10' function='0'/> >> </controller> >> >> and then change the <interface> definition above to say "bus='1'", that >> interface device will plug into this new bus at slot 3. >> >> * So let's assume that we add a new controller called "dmi-to-pci-bridge": >> >> <controller type='dmi-to-pci-bridge' index='0'/> >> >> Ignoring for now the question of what address we give in the definition >> of *this* device (which is itself problematic - do we need a new "pcie" >> address type?), if some device is then defined with >> >> >> <address type='pci bus='0' .../> >> >> How do we differentiate between that meaning "the pci-ptp controller >> that is index='0'" and "the pci-bridge controller that is index='0'"? Do >> we need to expand our <address> element further? If, as I think you >> suggest, we have multiple different kinds of controllers that provide >> PCI slots, each with its own namespace, the current pci address element >> is inadequate to unambiguously describe where a pci device should be >> plugged in. > Hmm yes, you're right - as long as we only have <adress type='pci'> > then all <controller> elements should use type='pci' too, and we should > just distinguish based on the model name of the controller. So ignore > my previous suggestion to have 'pci-bridge' and 'pci-root' types, we > can only use type='pci' on <controller> elements. Okay, so that means we preserve the correlation between <controller type='pci' index='1'> and <address type='pci' bus='1' ..../> Should the <controller> device use, e.g. <model type='pci-bridge'/> for the model, as is done for <interface> devices? One notable difference is that in the case of <interface> (with the exception of "<model type='virtio'/>"), the model isn't used for anything except passing directly through to qemu (and very recently validating against a list of known interface models), while in the case of controllers with type='pci', different models will have different rules about what they can connect to and what can connect to them, and they will affect what is valid in other devices. An example on a "pc" machinetype that has the builtin PCI bus, one extra pci-pci bridge, and an interface device plugged into slot 3 of the pci-bridge: <controller type='pci' index='0'> <model type='pci-root'/> <!-- builtin pci bus --> </controller> <controller type='pci' index='1'> <model type='pci-bridge'/> </controller> <interface type='direct'> ... <address type='pci' bus='1' slot='3'/> </controller> And for a q35 machinetype that has the root pcie, an i82801b11-bridge connected to slot 1e of that, a pci bridge connected to slot 1 of the i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge: <controller type='pci' index='0'> <model type='pcie-root'/> </controller> <controller type='pci' index='1'> <model type='i82801b11-bridge'/> <!-- [*] --> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci' index='2'> <model type='pci-bridge'/> <address type='pci' bus='1' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' bus='2' slot='3'/> </controller> (note that controllers with model='(pci|pcie)-root' will not have any <address> element, because they exist in the basic machine so we don't need to connect them to anywhere.)

(also note that it might happen that the bus number in libvirt's config will correspond to the bus numbering that shows up in the guest OS, but that will just be a happy coincidence) Does this make sense?