Re: [libvirt] new libvirt "pci" controller type and pcie/q35 (was Re: [PATCH 4/7] add pci-bridge controller type)

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: > On 04/08/2013 12:48 PM, Daniel P. Berrange wrote: >> On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: >>> I think we're starting to get closer to the concrete problem that's >>> bothering me. As I understand it (and again - "what I understand" has >>> repeatedly been shown to be incorrect in this thread :-): >>> >>> * Ihere are multiple different types of devices that provide a bus with >>> 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the >>> e1000 network device, etc) can be plugged into. >>> >>> * In the config for those devices, there is a required (auto-generated >>> if not explicitly provided) <address> element that indicates what >>> controller that device is plugged into e.g.: >>> >>> <interface type='direct'> >>> ... >>> <address type='pci' domain='0' bus='0' slot='3' function='0'/> >>> ... >>> </interface> >>> >>> * domain is always hardcoded to 0, and in the past bus was also always >>> hardcoded to 0 because until now there has only been a single place >>> where PCI devices could be connected - the builtin pci.0 bus, which is a >>> part of the basic "pc" (and some others) virtual machine and provides 32 >>> slots. >>> >>> * Now we are adding the ability to define new PCI buses, for now just a >>> single kind - a pci-bridge controller, which itself must connect to an >>> existing PCI slot, and provides 32 new PCI slots. But in the future >>> there will be more different types of controllers that provide one or >>> more PCI slots where PCI devices/controllers can be plugged in. >>> >>> * In these patches adding support for pci-bridge, we are making the >>> assumption that there is a 1:1 correspondence between the "index='n'" >>> attribute of the pci-bridge controller and the "bus='n'" attribute of >>> the <address> element in devices that will be plugged into that >>> controller. So for example if we have: >>> >>> >>> <controller type='pci-bridge' index='1'> >>> <address type='pci' domain='0' bus='0' slot='10' function='0'/> >>> </controller> >>> >>> and then change the <interface> definition above to say "bus='1'", that >>> interface device will plug into this new bus at slot 3. >>> >>> * So let's assume that we add a new controller called "dmi-to-pci-bridge": >>> >>> <controller type='dmi-to-pci-bridge' index='0'/> >>> >>> Ignoring for now the question of what address we give in the definition >>> of *this* device (which is itself problematic - do we need a new "pcie" >>> address type?), if some device is then defined with >>> >>> >>> <address type='pci bus='0' .../> >>> >>> How do we differentiate between that meaning "the pci-ptp controller >>> that is index='0'" and "the pci-bridge controller that is index='0'"? Do >>> we need to expand our <address> element further? If, as I think you >>> suggest, we have multiple different kinds of controllers that provide >>> PCI slots, each with its own namespace, the current pci address element >>> is inadequate to unambiguously describe where a pci device should be >>> plugged in. >> Hmm yes, you're right - as long as we only have <adress type='pci'> >> then all <controller> elements should use type='pci' too, and we should >> just distinguish based on the model name of the controller. So ignore >> my previous suggestion to have 'pci-bridge' and 'pci-root' types, we >> can only use type='pci' on <controller> elements. > Okay, so that means we preserve the correlation between > > <controller type='pci' index='1'> > > and > > <address type='pci' bus='1' ..../> > > > Should the <controller> device use, e.g. <model type='pci-bridge'/> for > the model, as is done for <interface> devices? One notable difference is > that in the case of <interface> (with the exception of "<model > type='virtio'/>"), the model isn't used for anything except passing > directly through to qemu (and very recently validating against a list of > known interface models), while in the case of controllers with > type='pci', different models will have different rules about what they > can connect to and what can connect to them, and they will affect what > is valid in other devices. > > An example on a "pc" machinetype that has the builtin PCI bus, one extra > pci-pci bridge, and an interface device plugged into slot 3 of the > pci-bridge: > > <controller type='pci' index='0'> > <model type='pci-root'/> <!-- builtin pci bus --> > </controller> > <controller type='pci' index='1'> > <model type='pci-bridge'/> > </controller> > <interface type='direct'> > ... > <address type='pci' bus='1' slot='3'/> > </controller> > > And for a q35 machinetype that has the root pcie, an i82801b11-bridge > connected to slot 1e of that, a pci bridge connected to slot 1 of the > i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge: > > <controller type='pci' index='0'> > <model type='pcie-root'/> > </controller> > <controller type='pci' index='1'> > <model type='i82801b11-bridge'/> <!-- [*] --> > <address type='pci' bus='0' slot='0x1e'/> > </controller> > <controller type='pci' index='2'> > <model type='pci-bridge'/> > <address type='pci' bus='1' slot='1'/> > </controller> > <interface type='direct'> > ... > <address type='pci' bus='2' slot='3'/> > </controller> > > (note that controllers with model='(pci|pcie)-root' will not have any > <address> element, because they exist in the basic machine so we don't > need to connect them to anywhere.) > > (also note that it might happen that the bus number in libvirt's config > will correspond to the bus numbering that shows up in the guest OS, but > that will just be a happy coincidence) > > Does this make sense? Confused. So why are you using bus numbers at all?