On 04/03/2013 11:50 AM, Ján Tomko wrote: > Change QEMU_PCI_ADDRESS_LAST_SLOT to the number of slots in the bus, > not the maximum slot value, to match QEMU_PCI_ADDRESS_LAST_FUNCTION. > --- If you want to be *really* consistent, you should rename these to VIR_QEMU_PCI_ADDRESS_SLOT_LAST and VIR_QEMU_PCI_ADDRESS_FUNCTION_LAST :-) (i.e. start with "VIR_" and put "LAST" at the end)

On Fri, Apr 05, 2013 at 12:32:04PM -0400, Laine Stump wrote: > On 04/03/2013 11:50 AM, Ján Tomko wrote: >> From: liguang <lig.fnst(a)cn.fujitsu.com&gt; >> >> add a new controller type, then one can >> define a pci-bridge controller like this: >> <controller type='pci-bridge' index='0'/> > In the next patch we're prohibiting exactly this config (index='0') > because the pre-existing pci bus on the "pc-*" machinetypes is already > named pci.0. If we don't allow it, we shouldn't include it as an example > in the commit log :-) NB, it isn't always named 'pci.0' - on many arches it is merely 'pci'.

> More on this - one of the things this points out is that there is no > representation in the config of the pci.0 bus, it's just assumed to > always be there. That is the case for pc-* machinetypes (and probably > several others with PCI buses), but for q35, there is no pci.0 bus in > the basic machine, only a pcie.0; if you want a pci.0 on q35 (which > *will* be necessary in order to attach any pci devices, so I imagine we > will always want one), you have to attach a pcie->pci bridge, which is > the device "i82801b11-bridge", to pcie.0. > The reason I bring this up here, is I'm wondering: > > 1) should we have some representation of the default pci.0 bus in the > config, even though it is just "always there" for the pc machinetypes > and there is no way to disable it, and nothing on the commandline that > specifies its existence? Yep, we should be aiming for the XML to fully describe the machine hardware. So since we're adding the concept of PCI controllers/bridges etc to the XML, we should be auto-adding the default bus to the XML. > 2) For the q35 machinetype, should we just always add an > i82801b11-bridge device and name it pci.0? Or should that need to be > present in the xml? We've been burnt before auto-adding stuff that ought to have been optional. So I'd tend towards only having the minimal config that is required. If the users want this, let them explicitly ask for the bridge Also from the apps POV the QEMU device name is irrelevant. The XML config works off the PCI addresses. So there's no need to force/specialcase a i82801b11-bridge to use the name 'pci.0'.

> 3) Most important - depending on the answers to (1) and (2), should we > maybe name this device "pci", and use a different backend depending on > index and machinetype? (or alternately explicitly specifiable with a > <driver> subelement). To be specific, we would have: > > <controller type='pci' index='0'/> > > which on pc machinetypes would just be a placeholder in the config (and > always inserted if it wasn't there, for machinetypes that have a pci > bus). On the q35 machinetype, that same line would equate to adding an > i82801b11-bridge device (with source defaulting to > bus=pcie.0,addr=1e.0). This would serve several purposes: > > a) on pc machinetypes, it would be a visual aid indicating that pci.0 > exists, and that index='0' isn't available for a new pci controller. > > b) it would make switching a domain config from pc to q35 simpler, since > pci.0 would always already be in place for attaching pci devices > (including pci.1, pci.2, etc) > > c) it would make the config a true complete description of the machine > being created. > > (I've suggested naming the controller "pci" rather than "pci-bridge" > because in the case of a "root" bus like pci.0 it seems to not be a > "bridge", but maybe the name "pci-bridge" is always appropriate, even > when it's a root bus. Maybe someone with better pci/pcie knowledge can > provide an opinion on this) I think "pci" is a little too generic - how about we call it 'pci-root'

On Fri, 2013-04-05 at 14:42 -0400, Laine Stump wrote: > On 04/05/2013 01:38 PM, Daniel P. Berrange wrote: >> On Fri, Apr 05, 2013 at 12:32:04PM -0400, Laine Stump wrote: >>> On 04/03/2013 11:50 AM, Ján Tomko wrote: >>>> From: liguang <lig.fnst(a)cn.fujitsu.com&gt; >>>> >>>> add a new controller type, then one can >>>> define a pci-bridge controller like this: >>>> <controller type='pci-bridge' index='0'/> >>> In the next patch we're prohibiting exactly this config (index='0') >>> because the pre-existing pci bus on the "pc-*" machinetypes is already >>> named pci.0. If we don't allow it, we shouldn't include it as an example >>> in the commit log :-) >> NB, it isn't always named 'pci.0' - on many arches it is merely 'pci'. > Yeah, I'm just using that as a convenient shorthand. The final decision > on whether to use pci.0 or pci happens down in the qemuBuildCommandline(). > >>> More on this - one of the things this points out is that there is no >>> representation in the config of the pci.0 bus, it's just assumed to >>> always be there. That is the case for pc-* machinetypes (and probably >>> several others with PCI buses), but for q35, there is no pci.0 bus in >>> the basic machine, only a pcie.0; if you want a pci.0 on q35 (which >>> *will* be necessary in order to attach any pci devices, so I imagine we >>> will always want one), you have to attach a pcie->pci bridge, which is >>> the device "i82801b11-bridge", to pcie.0. >>> The reason I bring this up here, is I'm wondering: >>> >>> 1) should we have some representation of the default pci.0 bus in the >>> config, even though it is just "always there" for the pc machinetypes >>> and there is no way to disable it, and nothing on the commandline that >>> specifies its existence? >> Yep, we should be aiming for the XML to fully describe the machine >> hardware. So since we're adding the concept of PCI controllers/bridges >> etc to the XML, we should be auto-adding the default bus to the XML. >> >>> 2) For the q35 machinetype, should we just always add an >>> i82801b11-bridge device and name it pci.0? Or should that need to be >>> present in the xml? >> We've been burnt before auto-adding stuff that ought to have >> been optional. So I'd tend towards only having the minimal >> config that is required. If the users want this, let them >> explicitly ask for the bridge

>> >> Also from the apps POV the QEMU device name is irrelevant. The >> XML config works off the PCI addresses. So there's no need >> to force/specialcase a i82801b11-bridge to use the name >> 'pci.0'. > > Sure. I just mean "pci bus 0" (hmm, but actually this does point out a > problem with my logic - the same namespace (well, "numbering space") is > used for both pcie and pci buses, so on a q35 system, bus=0 is already > taken by pcie.0; that means that the first pci bus would need to use a > different bus number anyway, so it wouldn't be so easy to switch an > existing domain from pc to q35 - every PCI device would need to have its > bus number modified. I suppose that's reasonable to expect, though. I would think you'd want to differentiate PCI from PCIe anyway. PCI is a bus and you have 32 slots per bus to fill. PCIe is a point-to-point link and you really only have slot 0 available. Perhaps that puts them in different number spaces already.

>>> 3) Most important - depending on the answers to (1) and (2), should we >>> maybe name this device "pci", and use a different backend depending on >>> index and machinetype? (or alternately explicitly specifiable with a >>> <driver> subelement). To be specific, we would have: >>> >>> <controller type='pci' index='0'/> >>> >>> which on pc machinetypes would just be a placeholder in the config (and >>> always inserted if it wasn't there, for machinetypes that have a pci >>> bus). On the q35 machinetype, that same line would equate to adding an >>> i82801b11-bridge device (with source defaulting to >>> bus=pcie.0,addr=1e.0). This would serve several purposes: >>> >>> a) on pc machinetypes, it would be a visual aid indicating that pci.0 >>> exists, and that index='0' isn't available for a new pci controller. >>> >>> b) it would make switching a domain config from pc to q35 simpler, since >>> pci.0 would always already be in place for attaching pci devices >>> (including pci.1, pci.2, etc) >>> >>> c) it would make the config a true complete description of the machine >>> being created. >>> >>> (I've suggested naming the controller "pci" rather than "pci-bridge" >>> because in the case of a "root" bus like pci.0 it seems to not be a >>> "bridge", but maybe the name "pci-bridge" is always appropriate, even >>> when it's a root bus. Maybe someone with better pci/pcie knowledge can >>> provide an opinion on this) >> I think "pci" is a little too generic - how about we call it 'pci-root' > Okay, so a separate "pci-root" device along with "pci-bridge"? What I > was really hoping was to have all PCI buses represented in a common way > in the config. How about a controller called "pci" with different types, > "root" and "bridge"? And since they use the same numbering space as pcie > buses, maybe the pcie controllers (including the root and the hubs and > ???) would be different types of PCI controllers. That would make it > easier (i.e. *possible*) to avoid collisions in use of bus numbers. > > Alex or mst, any advice/opinions on how to represent all the different > q35 devices that consume bus numbers in a succinct fashion? Note that none of these are really bus numbers, they're just bus identifiers. The BIOS and the guest running define the bus numbers. "root" also has special meaning in PCI, so for instance I wouldn't name a bus behind the i82801b11-bridge "pci-root". Somehow we also need to deal with what can be attached where. For instance a pci-bridge is a PCI device and can only go on a PCI bus. The equivalent structure on PCIe is an upstream switch port with some number of downstream switch ports. Each of those are specific to the bus type.

For PCIe, we create new buses for root ports (ioh3420), upstream switch ports (xio3130-upstream), downstream switch ports (xio3130-downstream), and the dmi-to-pci bridge (i82801b11-bridge). For PCI, PCI-to-PCI bridges create new buses (pci-bridge and dec-21154). One of my goals is to move us away from emulation of specific chips and create more devices like pci-bridge that adhere to the standard, but don't try to emulate a specific device. Then we might have "root-port", "pcie-upstream-switch-port", "pcie-downstream-switch-port", and "dmi-to-pci-bridge" (none of these names have been discussed).

On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: > I think we're starting to get closer to the concrete problem that's > bothering me. As I understand it (and again - "what I understand" has > repeatedly been shown to be incorrect in this thread :-): > > * Ihere are multiple different types of devices that provide a bus with > 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the > e1000 network device, etc) can be plugged into. > > * In the config for those devices, there is a required (auto-generated > if not explicitly provided) <address> element that indicates what > controller that device is plugged into e.g.: > > <interface type='direct'> > ... > <address type='pci' domain='0' bus='0' slot='3' function='0'/> > ... > </interface> > > * domain is always hardcoded to 0, and in the past bus was also always > hardcoded to 0 because until now there has only been a single place > where PCI devices could be connected - the builtin pci.0 bus, which is a > part of the basic "pc" (and some others) virtual machine and provides 32 > slots. > > * Now we are adding the ability to define new PCI buses, for now just a > single kind - a pci-bridge controller, which itself must connect to an > existing PCI slot, and provides 32 new PCI slots. But in the future > there will be more different types of controllers that provide one or > more PCI slots where PCI devices/controllers can be plugged in. > > * In these patches adding support for pci-bridge, we are making the > assumption that there is a 1:1 correspondence between the "index='n'" > attribute of the pci-bridge controller and the "bus='n'" attribute of > the <address> element in devices that will be plugged into that > controller. So for example if we have: > > > <controller type='pci-bridge' index='1'> > <address type='pci' domain='0' bus='0' slot='10' function='0'/> > </controller> > > and then change the <interface> definition above to say "bus='1'", that > interface device will plug into this new bus at slot 3. > > * So let's assume that we add a new controller called "dmi-to-pci-bridge": > > <controller type='dmi-to-pci-bridge' index='0'/> > > Ignoring for now the question of what address we give in the definition > of *this* device (which is itself problematic - do we need a new "pcie" > address type?), if some device is then defined with > > > <address type='pci bus='0' .../> > > How do we differentiate between that meaning "the pci-ptp controller > that is index='0'" and "the pci-bridge controller that is index='0'"? Do > we need to expand our <address> element further? If, as I think you > suggest, we have multiple different kinds of controllers that provide > PCI slots, each with its own namespace, the current pci address element > is inadequate to unambiguously describe where a pci device should be > plugged in. Hmm yes, you're right - as long as we only have <adress type='pci'> then all <controller> elements should use type='pci' too, and we should just distinguish based on the model name of the controller. So ignore my previous suggestion to have 'pci-bridge' and 'pci-root' types, we can only use type='pci' on <controller> elements.

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: > On 04/08/2013 12:48 PM, Daniel P. Berrange wrote: >> On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: >>> I think we're starting to get closer to the concrete problem that's >>> bothering me. As I understand it (and again - "what I understand" has >>> repeatedly been shown to be incorrect in this thread :-): >>> >>> * Ihere are multiple different types of devices that provide a bus with >>> 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the >>> e1000 network device, etc) can be plugged into. >>> >>> * In the config for those devices, there is a required (auto-generated >>> if not explicitly provided) <address> element that indicates what >>> controller that device is plugged into e.g.: >>> >>> <interface type='direct'> >>> ... >>> <address type='pci' domain='0' bus='0' slot='3' function='0'/> >>> ... >>> </interface> >>> >>> * domain is always hardcoded to 0, and in the past bus was also always >>> hardcoded to 0 because until now there has only been a single place >>> where PCI devices could be connected - the builtin pci.0 bus, which is a >>> part of the basic "pc" (and some others) virtual machine and provides 32 >>> slots. >>> >>> * Now we are adding the ability to define new PCI buses, for now just a >>> single kind - a pci-bridge controller, which itself must connect to an >>> existing PCI slot, and provides 32 new PCI slots. But in the future >>> there will be more different types of controllers that provide one or >>> more PCI slots where PCI devices/controllers can be plugged in. >>> >>> * In these patches adding support for pci-bridge, we are making the >>> assumption that there is a 1:1 correspondence between the "index='n'" >>> attribute of the pci-bridge controller and the "bus='n'" attribute of >>> the <address> element in devices that will be plugged into that >>> controller. So for example if we have: >>> >>> >>> <controller type='pci-bridge' index='1'> >>> <address type='pci' domain='0' bus='0' slot='10' function='0'/> >>> </controller> >>> >>> and then change the <interface> definition above to say "bus='1'", that >>> interface device will plug into this new bus at slot 3. >>> >>> * So let's assume that we add a new controller called "dmi-to-pci-bridge": >>> >>> <controller type='dmi-to-pci-bridge' index='0'/> >>> >>> Ignoring for now the question of what address we give in the definition >>> of *this* device (which is itself problematic - do we need a new "pcie" >>> address type?), if some device is then defined with >>> >>> >>> <address type='pci bus='0' .../> >>> >>> How do we differentiate between that meaning "the pci-ptp controller >>> that is index='0'" and "the pci-bridge controller that is index='0'"? Do >>> we need to expand our <address> element further? If, as I think you >>> suggest, we have multiple different kinds of controllers that provide >>> PCI slots, each with its own namespace, the current pci address element >>> is inadequate to unambiguously describe where a pci device should be >>> plugged in. >> Hmm yes, you're right - as long as we only have <adress type='pci'> >> then all <controller> elements should use type='pci' too, and we should >> just distinguish based on the model name of the controller. So ignore >> my previous suggestion to have 'pci-bridge' and 'pci-root' types, we >> can only use type='pci' on <controller> elements. > Okay, so that means we preserve the correlation between > > <controller type='pci' index='1'> > > and > > <address type='pci' bus='1' ..../> > > > Should the <controller> device use, e.g. <model type='pci-bridge'/> for > the model, as is done for <interface> devices? One notable difference is > that in the case of <interface> (with the exception of "<model > type='virtio'/>"), the model isn't used for anything except passing > directly through to qemu (and very recently validating against a list of > known interface models), while in the case of controllers with > type='pci', different models will have different rules about what they > can connect to and what can connect to them, and they will affect what > is valid in other devices. > > An example on a "pc" machinetype that has the builtin PCI bus, one extra > pci-pci bridge, and an interface device plugged into slot 3 of the > pci-bridge: > > <controller type='pci' index='0'> > <model type='pci-root'/> <!-- builtin pci bus --> > </controller> > <controller type='pci' index='1'> > <model type='pci-bridge'/> > </controller> > <interface type='direct'> > ... > <address type='pci' bus='1' slot='3'/> > </controller> > > And for a q35 machinetype that has the root pcie, an i82801b11-bridge > connected to slot 1e of that, a pci bridge connected to slot 1 of the > i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge: > > <controller type='pci' index='0'> > <model type='pcie-root'/> > </controller> > <controller type='pci' index='1'> > <model type='i82801b11-bridge'/> <!-- [*] --> > <address type='pci' bus='0' slot='0x1e'/> > </controller> > <controller type='pci' index='2'> > <model type='pci-bridge'/> > <address type='pci' bus='1' slot='1'/> > </controller> > <interface type='direct'> > ... > <address type='pci' bus='2' slot='3'/> > </controller> > > (note that controllers with model='(pci|pcie)-root' will not have any > <address> element, because they exist in the basic machine so we don't > need to connect them to anywhere.) > > (also note that it might happen that the bus number in libvirt's config > will correspond to the bus numbering that shows up in the guest OS, but > that will just be a happy coincidence) > > Does this make sense? Confused. So why are you using bus numbers at all?

On Mon, Apr 08, 2013 at 10:37:45PM +0300, Michael S. Tsirkin wrote: > On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: > > On 04/08/2013 12:48 PM, Daniel P. Berrange wrote: > > > On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: > > >> I think we're starting to get closer to the concrete problem that's > > >> bothering me. As I understand it (and again - "what I understand" has > > >> repeatedly been shown to be incorrect in this thread :-): > > >> > > >> * Ihere are multiple different types of devices that provide a bus with > > >> 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the > > >> e1000 network device, etc) can be plugged into. > > >> > > >> * In the config for those devices, there is a required (auto-generated > > >> if not explicitly provided) <address> element that indicates what > > >> controller that device is plugged into e.g.: > > >> > > >> <interface type='direct'> > > >> ... > > >> <address type='pci' domain='0' bus='0' slot='3' function='0'/> > > >> ... > > >> </interface> > > >> > > >> * domain is always hardcoded to 0, and in the past bus was also always > > >> hardcoded to 0 because until now there has only been a single place > > >> where PCI devices could be connected - the builtin pci.0 bus, which is a > > >> part of the basic "pc" (and some others) virtual machine and provides 32 > > >> slots. > > >> > > >> * Now we are adding the ability to define new PCI buses, for now just a > > >> single kind - a pci-bridge controller, which itself must connect to an > > >> existing PCI slot, and provides 32 new PCI slots. But in the future > > >> there will be more different types of controllers that provide one or > > >> more PCI slots where PCI devices/controllers can be plugged in. > > >> > > >> * In these patches adding support for pci-bridge, we are making the > > >> assumption that there is a 1:1 correspondence between the "index='n'" > > >> attribute of the pci-bridge controller and the "bus='n'" attribute of > > >> the <address> element in devices that will be plugged into that > > >> controller. So for example if we have: > > >> > > >> > > >> <controller type='pci-bridge' index='1'> > > >> <address type='pci' domain='0' bus='0' slot='10' function='0'/> > > >> </controller> > > >> > > >> and then change the <interface> definition above to say "bus='1'", that > > >> interface device will plug into this new bus at slot 3. > > >> > > >> * So let's assume that we add a new controller called "dmi-to-pci-bridge": > > >> > > >> <controller type='dmi-to-pci-bridge' index='0'/> > > >> > > >> Ignoring for now the question of what address we give in the definition > > >> of *this* device (which is itself problematic - do we need a new "pcie" > > >> address type?), if some device is then defined with > > >> > > >> > > >> <address type='pci bus='0' .../> > > >> > > >> How do we differentiate between that meaning "the pci-ptp controller > > >> that is index='0'" and "the pci-bridge controller that is index='0'"? Do > > >> we need to expand our <address> element further? If, as I think you > > >> suggest, we have multiple different kinds of controllers that provide > > >> PCI slots, each with its own namespace, the current pci address element > > >> is inadequate to unambiguously describe where a pci device should be > > >> plugged in. > > > Hmm yes, you're right - as long as we only have <adress type='pci'> > > > then all <controller> elements should use type='pci' too, and we should > > > just distinguish based on the model name of the controller. So ignore > > > my previous suggestion to have 'pci-bridge' and 'pci-root' types, we > > > can only use type='pci' on <controller> elements. > > > > Okay, so that means we preserve the correlation between > > > > <controller type='pci' index='1'> > > > > and > > > > <address type='pci' bus='1' ..../> > > > > > > Should the <controller> device use, e.g. <model type='pci-bridge'/> for > > the model, as is done for <interface> devices? One notable difference is > > that in the case of <interface> (with the exception of "<model > > type='virtio'/>"), the model isn't used for anything except passing > > directly through to qemu (and very recently validating against a list of > > known interface models), while in the case of controllers with > > type='pci', different models will have different rules about what they > > can connect to and what can connect to them, and they will affect what > > is valid in other devices. > > > > An example on a "pc" machinetype that has the builtin PCI bus, one extra > > pci-pci bridge, and an interface device plugged into slot 3 of the > > pci-bridge: > > > > <controller type='pci' index='0'> > > <model type='pci-root'/> <!-- builtin pci bus --> > > </controller> > > <controller type='pci' index='1'> > > <model type='pci-bridge'/> > > </controller> > > <interface type='direct'> > > ... > > <address type='pci' bus='1' slot='3'/> > > </controller> > > > > And for a q35 machinetype that has the root pcie, an i82801b11-bridge > > connected to slot 1e of that, a pci bridge connected to slot 1 of the > > i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge: > > > > <controller type='pci' index='0'> > > <model type='pcie-root'/> > > </controller> > > <controller type='pci' index='1'> > > <model type='i82801b11-bridge'/> <!-- [*] --> > > <address type='pci' bus='0' slot='0x1e'/> > > </controller> > > <controller type='pci' index='2'> > > <model type='pci-bridge'/> > > <address type='pci' bus='1' slot='1'/> > > </controller> > > <interface type='direct'> > > ... > > <address type='pci' bus='2' slot='3'/> > > </controller> > > > > (note that controllers with model='(pci|pcie)-root' will not have any > > <address> element, because they exist in the basic machine so we don't > > need to connect them to anywhere.) > > > > (also note that it might happen that the bus number in libvirt's config > > will correspond to the bus numbering that shows up in the guest OS, but > > that will just be a happy coincidence) > > > > Does this make sense? > > Confused. So why are you using bus numbers at all? > It's just wrong. They are not wrong. We use to link the <address> element to the <controller> element. Daniel

-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: > On 04/08/2013 12:48 PM, Daniel P. Berrange wrote: >> On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: >>> I think we're starting to get closer to the concrete problem that's >>> bothering me. As I understand it (and again - "what I understand" has >>> repeatedly been shown to be incorrect in this thread :-): >>> >>> * Ihere are multiple different types of devices that provide a bus with >>> 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the >>> e1000 network device, etc) can be plugged into. >>> >>> * In the config for those devices, there is a required (auto-generated >>> if not explicitly provided) <address> element that indicates what >>> controller that device is plugged into e.g.: >>> >>> <interface type='direct'> >>> ... >>> <address type='pci' domain='0' bus='0' slot='3' function='0'/> >>> ... >>> </interface> >>> >>> * domain is always hardcoded to 0, and in the past bus was also always >>> hardcoded to 0 because until now there has only been a single place >>> where PCI devices could be connected - the builtin pci.0 bus, which is a >>> part of the basic "pc" (and some others) virtual machine and provides 32 >>> slots. >>> >>> * Now we are adding the ability to define new PCI buses, for now just a >>> single kind - a pci-bridge controller, which itself must connect to an >>> existing PCI slot, and provides 32 new PCI slots. But in the future >>> there will be more different types of controllers that provide one or >>> more PCI slots where PCI devices/controllers can be plugged in. >>> >>> * In these patches adding support for pci-bridge, we are making the >>> assumption that there is a 1:1 correspondence between the "index='n'" >>> attribute of the pci-bridge controller and the "bus='n'" attribute of >>> the <address> element in devices that will be plugged into that >>> controller. So for example if we have: >>> >>> >>> <controller type='pci-bridge' index='1'> >>> <address type='pci' domain='0' bus='0' slot='10' function='0'/> >>> </controller> >>> >>> and then change the <interface> definition above to say "bus='1'", that >>> interface device will plug into this new bus at slot 3. >>> >>> * So let's assume that we add a new controller called "dmi-to-pci-bridge": >>> >>> <controller type='dmi-to-pci-bridge' index='0'/> >>> >>> Ignoring for now the question of what address we give in the definition >>> of *this* device (which is itself problematic - do we need a new "pcie" >>> address type?), if some device is then defined with >>> >>> >>> <address type='pci bus='0' .../> >>> >>> How do we differentiate between that meaning "the pci-ptp controller >>> that is index='0'" and "the pci-bridge controller that is index='0'"? Do >>> we need to expand our <address> element further? If, as I think you >>> suggest, we have multiple different kinds of controllers that provide >>> PCI slots, each with its own namespace, the current pci address element >>> is inadequate to unambiguously describe where a pci device should be >>> plugged in. >> Hmm yes, you're right - as long as we only have <adress type='pci'> >> then all <controller> elements should use type='pci' too, and we should >> just distinguish based on the model name of the controller. So ignore >> my previous suggestion to have 'pci-bridge' and 'pci-root' types, we >> can only use type='pci' on <controller> elements. > Okay, so that means we preserve the correlation between > > <controller type='pci' index='1'> > > and > > <address type='pci' bus='1' ..../> > > > Should the <controller> device use, e.g. <model type='pci-bridge'/> for > the model, as is done for <interface> devices? One notable difference is > that in the case of <interface> (with the exception of "<model > type='virtio'/>"), the model isn't used for anything except passing > directly through to qemu (and very recently validating against a list of > known interface models), while in the case of controllers with > type='pci', different models will have different rules about what they > can connect to and what can connect to them, and they will affect what > is valid in other devices. > > An example on a "pc" machinetype that has the builtin PCI bus, one extra > pci-pci bridge, and an interface device plugged into slot 3 of the > pci-bridge: > > <controller type='pci' index='0'> > <model type='pci-root'/> <!-- builtin pci bus --> > </controller> > <controller type='pci' index='1'> > <model type='pci-bridge'/> > </controller> > <interface type='direct'> > ... > <address type='pci' bus='1' slot='3'/> > </controller> > > And for a q35 machinetype that has the root pcie, an i82801b11-bridge > connected to slot 1e of that, a pci bridge connected to slot 1 of the > i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge: > > <controller type='pci' index='0'> > <model type='pcie-root'/> > </controller> > <controller type='pci' index='1'> > <model type='i82801b11-bridge'/> <!-- [*] --> > <address type='pci' bus='0' slot='0x1e'/> > </controller> > <controller type='pci' index='2'> > <model type='pci-bridge'/> > <address type='pci' bus='1' slot='1'/> > </controller> > <interface type='direct'> > ... > <address type='pci' bus='2' slot='3'/> > </controller> > > (note that controllers with model='(pci|pcie)-root' will not have any > <address> element, because they exist in the basic machine so we don't > need to connect them to anywhere.) Actually I do wonder if we should reprent a PCI root as two <controller> elements, one representing the actual PCI root device, and the other representing the host bridge that is built-in. Also we should use the actual model names, not 'pci-root' or 'pcie-root' but rather i440FX for "pc" machine type, and whatever the q35 model name is. - One PCI root with built-in PCI bus (ie todays' setup) <controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0''/>

</controller> <interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3'/> </controller> - One PCI root with built-in PCI bus and extra PCI bridge <controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller> - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge (ie possible q35 setup)

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller>

<interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller> So if we later allowed for mutiple PCI roots, then we'd have something like <controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci-root" index="1"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge 1 --> <address type='pci' domain='0' bus='0' slot='0''/> </controller> <controller type="pci" index="0"> <!-- Host bridge 2 --> <address type='pci' domain='1' bus='0' slot='0''/> </controller> <interface type='direct'> <!-- NIC on host bridge 2 --> ... <address type='pci' domain='1' bus='0' slot='3'/> </controller> NB this means that 'index' values can be reused against the <controller>, provided they are setup on different pci-roots. > (also note that it might happen that the bus number in libvirt's config > will correspond to the bus numbering that shows up in the guest OS, but > that will just be a happy coincidence) > > Does this make sense? Yep, I think we're fairly close.

On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote: > On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: >> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: >> Actually I do wonder if we should reprent a PCI root as two >> <controller> elements, one representing the actual PCI root >> device, and the other representing the host bridge that is >> built-in. >> >> Also we should use the actual model names, not 'pci-root' or >> 'pcie-root' but rather i440FX for "pc" machine type, and whatever >> the q35 model name is. >> >> - One PCI root with built-in PCI bus (ie todays' setup) >> >> <controller type="pci-root" index="0"> >> <model name="i440FX"/> >> </controller> >> <controller type="pci" index="0"> <!-- Host bridge --> >> <address type='pci' domain='0' bus='0' slot='0''/> > Isn't this saying that the bridge connects to itself? (since bus 0 is > this bus) > > I understand (again, possibly wrongly) that the builtin PCI bus connects > to the chipset using its own slot 0 (that's why it's reserved), but > that's its address on itself. How is this bridge associated with the > pci-root? > > Ah, I *think* I see it - the domain attribute of the pci controller is > matched to the index of the pci-root controller, correct? But there's > still something strange about the <address> of the pci controller being > self-referential. Yes, the index of the pci-root matches the 'domain' of <address>

>> </controller> >> <interface type='direct'> >> ... >> <address type='pci' domain='0' bus='0' slot='3'/> >> </controller> >> >> - One PCI root with built-in PCI bus and extra PCI bridge >> >> <controller type="pci-root" index="0"> >> <model name="i440FX"/> >> </controller> >> <controller type="pci" index="0"> <!-- Host bridge --> >> <address type='pci' domain='0' bus='0' slot='0'/> >> </controller> >> <controller type="pci" index="1"> <!-- Additional bridge --> >> <address type='pci' domain='0' bus='0' slot='1'/> >> </controller> >> <interface type='direct'> >> ... >> <address type='pci' domain='0' bus='1' slot='3'/> >> </controller> >> >> - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge >> (ie possible q35 setup) > Why would a q35 machine have an i440FX pci-root? It shouldn't, that's a typo >> <controller type="pci-root" index="0"> >> <model name="i440FX"/> >> </controller> >> <controller type="pci" index="0"> <!-- Host bridge --> >> <address type='pci' domain='0' bus='0' slot='0'/> >> </controller> >> <controller type="pci" index="1"> <!-- Additional bridge --> >> <address type='pci' domain='0' bus='0' slot='1'/> >> </controller> >> <controller type="pci" index="1"> <!-- Additional bridge --> >> <address type='pci' domain='0' bus='0' slot='1'/> >> </controller> > I think you did a cut-paste here and intended to change something, but > didn't - those two bridges are identical. Yep, the slot should be 2 in the second one >> <interface type='direct'> >> ... >> <address type='pci' domain='0' bus='1' slot='3'/> >> </controller> >> >> So if we later allowed for mutiple PCI roots, then we'd have something >> like >> >> <controller type="pci-root" index="0"> >> <model name="i440FX"/> >> </controller> >> <controller type="pci-root" index="1"> >> <model name="i440FX"/> >> </controller> >> <controller type="pci" index="0"> <!-- Host bridge 1 --> >> <address type='pci' domain='0' bus='0' slot='0''/> >> </controller> >> <controller type="pci" index="0"> <!-- Host bridge 2 --> >> <address type='pci' domain='1' bus='0' slot='0''/> >> </controller> >> <interface type='direct'> <!-- NIC on host bridge 2 --> >> ... >> <address type='pci' domain='1' bus='0' slot='3'/> >> </controller> >> >> >> NB this means that 'index' values can be reused against the >> <controller>, provided they are setup on different pci-roots. >> >>> (also note that it might happen that the bus number in libvirt's config >>> will correspond to the bus numbering that shows up in the guest OS, but >>> that will just be a happy coincidence) >>> >>> Does this make sense? >> Yep, I think we're fairly close. > What about the other types of pci controllers that are used by PCIe? We > should make sure they fit in this model before we settle on it. What do they do ?

For PCIe, we create new buses for root ports (ioh3420), upstream switch ports (xio3130-upstream), downstream switch ports (xio3130-downstream), and the dmi-to-pci bridge (i82801b11-bridge). For PCI, PCI-to-PCI bridges create new buses (pci-bridge and dec-21154).

One of my goals is to move us away from emulation of specific chips and create more devices like pci-bridge that adhere to the standard, but don't try to emulate a specific device. Then we might have "root-port", "pcie-upstream-switch-port", "pcie-downstream-switch-port", and "dmi-to-pci-bridge" (none of these names have been discussed).

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote: > On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: >> On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote: >>> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: >>>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: >>>> Actually I do wonder if we should reprent a PCI root as two >>>> <controller> elements, one representing the actual PCI root >>>> device, and the other representing the host bridge that is >>>> built-in. >>>> >>>> Also we should use the actual model names, not 'pci-root' or >>>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever >>>> the q35 model name is. >>>> >>>> - One PCI root with built-in PCI bus (ie todays' setup) >>>> >>>> <controller type="pci-root" index="0"> >>>> <model name="i440FX"/> >>>> </controller> >>>> <controller type="pci" index="0"> <!-- Host bridge --> >>>> <address type='pci' domain='0' bus='0' slot='0''/> >>> Isn't this saying that the bridge connects to itself? (since bus 0 is >>> this bus) >>> >>> I understand (again, possibly wrongly) that the builtin PCI bus connects >>> to the chipset using its own slot 0 (that's why it's reserved), but >>> that's its address on itself. How is this bridge associated with the >>> pci-root? >>> >>> Ah, I *think* I see it - the domain attribute of the pci controller is >>> matched to the index of the pci-root controller, correct? But there's >>> still something strange about the <address> of the pci controller being >>> self-referential. >> Yes, the index of the pci-root matches the 'domain' of <address> > > Okay, then the way that libvirt differentiates between a pci bridge that > is connected to the root, and one that is connected to a slot of another > bridge is 1) the "bus" attribute of the bridge's <address> matches the > "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct? > > (The corollary of this is that if slot == 0 and bus != index, or bus == > index and slot != 0, it is a configuration error). > > I'm still unclear on the usefulness of the pci-root controller though - > all the necessary information is contained in the pci controller, except > for the type of root. But in the case of pcie root, I think you're not > allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge" > (i82801b11-bridge) Yes you can connect a pci bridge to pcie-root. It's represented as a root complex integrated device.

>>>> </controller> >>>> <interface type='direct'> >>>> ... >>>> <address type='pci' domain='0' bus='0' slot='3'/> >>>> </controller> >>>> >>>> - One PCI root with built-in PCI bus and extra PCI bridge >>>> >>>> <controller type="pci-root" index="0"> >>>> <model name="i440FX"/> >>>> </controller> >>>> <controller type="pci" index="0"> <!-- Host bridge --> >>>> <address type='pci' domain='0' bus='0' slot='0'/> >>>> </controller> >>>> <controller type="pci" index="1"> <!-- Additional bridge --> >>>> <address type='pci' domain='0' bus='0' slot='1'/> >>>> </controller> >>>> <interface type='direct'> >>>> ... >>>> <address type='pci' domain='0' bus='1' slot='3'/> >>>> </controller> >>>> >>>> - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge >>>> (ie possible q35 setup) >>> Why would a q35 machine have an i440FX pci-root? >> It shouldn't, that's a typo >> >>>> <controller type="pci-root" index="0"> >>>> <model name="i440FX"/> >>>> </controller> >>>> <controller type="pci" index="0"> <!-- Host bridge --> >>>> <address type='pci' domain='0' bus='0' slot='0'/> >>>> </controller> >>>> <controller type="pci" index="1"> <!-- Additional bridge --> >>>> <address type='pci' domain='0' bus='0' slot='1'/> >>>> </controller> >>>> <controller type="pci" index="1"> <!-- Additional bridge --> >>>> <address type='pci' domain='0' bus='0' slot='1'/> >>>> </controller> >>> I think you did a cut-paste here and intended to change something, but >>> didn't - those two bridges are identical. >> Yep, the slot should be 2 in the second one >> >>>> <interface type='direct'> >>>> ... >>>> <address type='pci' domain='0' bus='1' slot='3'/> >>>> </controller> >>>> >>>> So if we later allowed for mutiple PCI roots, then we'd have something >>>> like >>>> >>>> <controller type="pci-root" index="0"> >>>> <model name="i440FX"/> >>>> </controller> >>>> <controller type="pci-root" index="1"> >>>> <model name="i440FX"/> >>>> </controller> >>>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>>> <address type='pci' domain='0' bus='0' slot='0''/> >>>> </controller> >>>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>>> <address type='pci' domain='1' bus='0' slot='0''/> >>>> </controller>

>>>> <interface type='direct'> <!-- NIC on host bridge 2 --> >>>> ... >>>> <address type='pci' domain='1' bus='0' slot='3'/> >>>> </controller> >>>> >>>> >>>> NB this means that 'index' values can be reused against the >>>> <controller>, provided they are setup on different pci-roots. >>>> >>>>> (also note that it might happen that the bus number in libvirt's config >>>>> will correspond to the bus numbering that shows up in the guest OS, but >>>>> that will just be a happy coincidence) >>>>> >>>>> Does this make sense? >>>> Yep, I think we're fairly close. >>> What about the other types of pci controllers that are used by PCIe? We >>> should make sure they fit in this model before we settle on it. >> What do they do ?

On Fri, Apr 12, 2013 at 11:46:15AM -0400, Laine Stump wrote: > On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: >> On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote: >>> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: >>>>>> So if we later allowed for mutiple PCI roots, then we'd have something >>>>>> like >>>>>> >>>>>> <controller type="pci-root" index="0"> >>>>>> <model name="i440FX"/> >>>>>> </controller> >>>>>> <controller type="pci-root" index="1"> >>>>>> <model name="i440FX"/> >>>>>> </controller> >>>>>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>>>>> <address type='pci' domain='0' bus='0' slot='0''/> >>>>>> </controller> >>>>>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>>>>> <address type='pci' domain='1' bus='0' slot='0''/> >>>>>> </controller> > > There is a problem here - within a given controller type, we will now > have the possibility of multiple controllers with the same index - the > differentiating attribute will be in the <address> subelement, which > could create some awkwardness. Maybe instead this should be handled with > a different model of pci controller, and we can add a "domain" attribute > at the toplevel rather than specifying an <address>? IIUC there is a limit on the number of PCI buses you can create per domain, due to fixed size of PCI addresses. Google suggests to me the limit is 256. So for domain 1, we could just start index at 256, and domain 2 at 512, etc

> Comment: I'm not quite convinced that we really need the separate > "pci-root" device. Since 1) every pci-root will *always* have either a > pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will > only ever be connected to the pci-root, and 3) the pci-bridge that > connects to it will need special handling within the pci-bridge case > anyway, why not: > > 1) eliminate the separate pci-root controller type Ok, lets leave it out - we can always add it later if desired.

> 2) within <controller type='pci'>, a new <model type='pci-root-bus'/> > will be added. > > 3) a pcie-root-bus will automatically be added for q35 machinetypes, and > pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*") > > 4) model type='pci-root-bus' will behave like pci-bridge, except that it > will be an implicit device (nothing on qemu commandline) and it won't > need an <address> element (neither will pcie-root-bus). > > 5) to support multiple domains, we can simply add a "domain" attribute > to the toplevel of controller. Or use index numbers modulo 256 to identify domain numbers.

One note on q35 - we need to make sure whatever we do in terms of creating default <controller>s in the XML 'just works' for applications. eg if they define a guest using <type machine="q35">hvm</type>, and then add a <interface>, it should do the right thing wrt PCI addressing/connectivity. We must not require applications to manually add <controller> elements for q35 for things to work. Adding <controller>s must purely be an opt-in for apps which have the detailed knowledge rquired & need full control over bus layout.

On 04/15/2013 04:09 PM, Laine Stump wrote: > On 04/15/2013 06:29 AM, Daniel P. Berrange wrote: >> On Fri, Apr 12, 2013 at 11:46:15AM -0400, Laine Stump wrote: >>> On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: >>>> On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote: >>>>> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: >>>>>>>> So if we later allowed for mutiple PCI roots, then we'd have >>>>>>>> something >>>>>>>> like >>>>>>>> >>>>>>>> <controller type="pci-root" index="0"> >>>>>>>> <model name="i440FX"/> >>>>>>>> </controller> >>>>>>>> <controller type="pci-root" index="1"> >>>>>>>> <model name="i440FX"/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='0''/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>>>>>>> <address type='pci' domain='1' bus='0' slot='0''/> >>>>>>>> </controller> >>> >>> There is a problem here - within a given controller type, we will now >>> have the possibility of multiple controllers with the same index - the >>> differentiating attribute will be in the<address> subelement, which >>> could create some awkwardness. Maybe instead this should be handled >>> with >>> a different model of pci controller, and we can add a "domain" >>> attribute >>> at the toplevel rather than specifying an<address>? >> IIUC there is a limit on the number of PCI buses you can create per >> domain, due to fixed size of PCI addresses. Google suggests to me >> the limit is 256. So for domain 1, we could just start index at >> 256, and domain 2 at 512, etc > > Okay. Whether we choose that method, or a separate domain attribute, I'm > satisfied that we'll be able to find a way to solve it when the time > comes (and it hasn't yet), so we can ignore that problem for now. > > *PLEASE* don't create a new/competing naming/numbering scheme for differentiating PCI domains.... as much as I dislike the overuse of the term 'domain', it's what is used. No sane person is going to look to assign PCI bus numbers > 256 in order to get new/different domains. The name sucks, but that's what it's called in the code, and what customers are use to.

>> >> >>> Comment: I'm not quite convinced that we really need the separate >>> "pci-root" device. Since 1) every pci-root will *always* have either a >>> pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus >>> will >>> only ever be connected to the pci-root, and 3) the pci-bridge that >>> connects to it will need special handling within the pci-bridge case >>> anyway, why not: >>> >>> 1) eliminate the separate pci-root controller type >> Ok, lets leave it out - we can always add it later if desired. > > Okay. > Not so fast.... something that represents the PCI Root Complex might be handy -- error handling and embedded devices (like IOMMUs, intr-remapping table) come to mind... ACPI tables (if they get duped from real systems) may need unconventional naming schemes for qemu if an RC isn't modelled.

>> >>> 2) within<controller type='pci'>, a new<model type='pci-root-bus'/> >>> will be added. >>> >>> 3) a pcie-root-bus will automatically be added for q35 >>> machinetypes, and >>> pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*") >>> >>> 4) model type='pci-root-bus' will behave like pci-bridge, except >>> that it >>> will be an implicit device (nothing on qemu commandline) and it won't >>> need an<address> element (neither will pcie-root-bus). >>> >>> 5) to support multiple domains, we can simply add a "domain" attribute >>> to the toplevel of controller. >> Or use index numbers modulo 256 to identify domain numbers. > > > Right. One or the other. But we can defer that discussion. > Just say 'domain' .... again! ;-)

> > >> One note on q35 - we need to make sure whatever we do in terms of >> creating >> default<controller>s in the XML 'just works' for applications. eg if >> they >> define a guest using<type machine="q35">hvm</type>, and then add a >> <interface>, it should do the right thing wrt PCI >> addressing/connectivity. >> We must not require applications to manually add<controller> elements >> for q35 for things to work. Adding<controller>s must purely be an >> opt-in >> for apps which have the detailed knowledge rquired& need full control >> over bus layout. > > Yep. What I see happening is that the place where we currently add > default controllers will, in the future, automatically add this for > machinetype pc* and rhel-*: > > <controller type='pci'> <!-- implied index='0' --> > <model type='pci-root'/> > </controller> > > and for machinetype q35* it will add (something like): > > <controller type='pci'> <!-- index='0' --> > <model type='pcie-root'/> > </controller> > <controller type='pci'> <!-- index='1' --> > <model type='dmi-to-pci-bridge'/> > <address type='pci' bus='0' slot='0x1e'/> > </controller> > <controller type='pci'> <!-- index='2' --> > <model type='pci-bridge'> > <address type='pci' bus='1' slot='1'/> > </controller> > > The slot-auto-reserve code will look through all pci controllers and > only auto-reserve slots on controllers appropriate for the given device > - controller 0 is already inappropriate for PCI devices, and we can mark > the dmi-to-pci-bridge type as being inappropriate for auto-reserve > (since, if I recall correctly, I was told that you can't hotplug devices > on that bus). So, all new PCI devices in the config will get addresses > with bus='2'. > > Of course this means that it will not be possible to switch an existing > domain config from pc to q35 simply by changing the machinetype - the > bus number in the address of all devices will need to be changed from 0 > to 2. But this is another case of "opt in", and already requires editing > the domain config anyway. If someone creates a brand new q35 machine > though, all PCI devices will get added with bus='whatever is the bus > number of the first pci-root or pci-bridge controller' (in this case, > '2'). > > So, here are the proposed pci controller types cleaned up an > re-summarized, followed by an example. > > <controller type='pci'> > ======================= > > This will be used for *all* of the following PCI controller devices > supported by qemu: > > > <model type='pci-root'/> (implicit/integrated) > ------------------------ > > Upstream: implicit connection to the host > Downstream: 32 slots (slot 0 reserved), PCI devices only > qemu commandline: nothing (implicit in the pc-* etc. machinetypes) > > This controller represents a pc* (or rhel-*) machine's integrated PCI > bus (pci.0) and provides places for PCI devices to connect (including > the "pci-bridge" type of PCI controller). > > There is only one of these controllers, and it will *always* be > index='0', and will have no<address> element. > ok. > > <model type='pcie-root'/> (implicit/integrated) > ------------------------- > > Upstream: implicit connection to the host > Downstream: 32 slots (slot 0 reserved), PCIe devices only, no > hotplug. > qemu commandline: nothing (implicit in the q35-* machinetype) > > This controller represents a q35's PCI "root complex", and provides > places for PCIe devices to connect. Examples are root-ports, > dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet > devices (do any of those integrated devices that can be connected to > the pcie-root-bus exist yet?). > > There is only one of these controllers, and it will *always* be > index='0', and will have no<address> element. > ok. > > <model type='root-port'/> (ioh3420) > ------------------------- > > Upstream: PCIe, connect to pcie-root-bus *only* (?) > Downstream: 1 slot, PCIe devices only (?) > qemu commandline: -device ioh3420,... > > These can only connect to the "pcie-root" of a q35. Any PCIe > devices can connect to it, including an upstream-switch-port. > ioh on q35; ich9/10/xx for other intel chipsets > > <model type='upstream-switch-port'/> (x3130-upstream) > ------------------------------------ > > Upstream: PCIe, connect to pcie-root-bus, root-port, or > downstream-switch-port (?) > Downstream: 32 slots, connect *only* to downstream-switch-port > qemu-commandline: -device x3130-upstream > > > This is the upper side of a switch that can multiplex multiple devices > onto a single port. It's only useful when one or more downstream switch > ports are connected to it. > > > <model type='downstream-switch-port'/> (xio3130-downstream) > -------------------------------------- > > Upstream: connect *only* to upstream-switch-port > Downstream: 1 slot, any PCIe device > qemu commandline: -device xio3130-downstream > > You can connect one or more of these to an upstream-switch-port in order > to effectively plug multiple devices into a single PCIe port. > ugh! one cannot have 3130-downstream w/o 3130upstream; simplify: PCIe-PPB-up; PCIe-PPB-down -- then it can be anything (not TI, not IDT, not Intel, etc.).

> > <model type='dmi-to-pci-bridge'/> (i82801b11-bridge) > ---------------------------------

> > (btw, what does "dmi" mean?) > > Upstream: pcie-root *only* > Downstream: 32 slots, any PCI device (including "pci-bridge"), > no hotplug (?) > qemu commandline: -device i82801b11-bridge,... > > This is the gateway to the world of standard old PCI. > why needed?

> > <model type='pci-bridge'/> (pci-bridge) > --------------------------

> > Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge > 3) another pci-bridge > Downstream: any PCI device, 32 slots > qemu commandline: -device pci-bridge,... > > This differs from dmi-to-pci-bridge in that its upstream connection is > PCI rather than PCIe (so it will work on an i440FX system, which has a > pci-root rather than pcie-root) and that hotplug is supported. In > general, if a guest will have any PCI devices, one of these > controllers should be added, and the PCI devices connected to it > rather than to the dmi-to-pci-bridge. > > ************************************ > (For q35, we *may* decide to always auto-add a dmi-to-pci-bridge at > 00:1E.0, and a pci-bridge on slot 1 of the dmi-to-pci-bridge. This > will allow a continuation of the tradition of simply adding new > devices to the config without worrying about where they connect.) > > > ============================================================================ > > Just to make sure this config model will work, here is the XML to > replicate the layout (only the ones involved in the PCI tree, along with > 3 ethernet devices as examples) of the X58 hardware I have sitting under > my desk (I've attached lspci and virsh nodedev-list --tree output from > that machine): > > > <controller type='pci' index='0'> > <model type='pcie-root'/> > </controller> > > <controller type='pci' index='1'> > <model type='root-port'/> > <address type='pci' bus='0' slot='1'/> > </controller> > > ( there is a scsi controller connected to bus='1') > > > <controller type='pci' index='2'> > <model type='root-port'/> > <address type='pci' bus='0' slot='3'/> > </controller> > > (the VGA controller is connected to bus='2') > > <controller type='pci' index='3'> > <model type='root-port'/> > <address type='pci' bus='0' slot='7'/> > </controller> > > (PCIe SRIOV network card (in external PCIe slot) connected to bus='3') > > <controller type='pci' index='4'> > <model type='root-port'/> > <address type='pci' bus='0' slot='0x1c' function='0'/> > </controller> > > (unused PCIe slot available on bus='4') > > <!-- pcie-root (0:1c.4) -> root-port (5:0.0) -> onboard > ethernet -> > <controller type='pci' index='5'> > <model type='root-port'/> > <address type='pci' bus='0' slot='0x1c' function='4'/> > </controller> > <interface type='blah'> > ... > <mac address='00:27:13:53:db:76'/> > <address type='pci' bus='5' slot='0' function='0'/> > </interface> > > <!-- more complicated connection to 2nd systemboard ethernet --> > <!-- pcie-root ->(0:1c:5)root-port -> (6:0.0)upstream-switch-port > -> (7:3.0)downstream-switch-port -> (9:0.0)ethernet --> > <controller type='pci' index='6'> > <model type='root-port'/> > <address type='pci' bus='0' slot='0x1c' function='5'/> > </controller> > <controller type='pci' index='7'> > <model type='upstream-switch-port'/> > <address type='pci' bus='6' slot='0' function='0'/> > </controller> > <controller type='pci' index='8'> > <model type='downstream-switch-port'/> > <address type='pci' bus='7' slot='2' function='0'/> > </controller> > <controller type='pci' index='9'> > <model type='downstream-switch-port'/> > <address type='pci' bus='7' slot='3' function='0'/> > </controller> > <interface type='blah'> > ... > <mac address='00:27:13:53:db:77'/> > <address type='pci' bus='9' slot='0' function='0'/> > </interface> > > > <!-- old-fashioned PCI ethernet in an external PCI slot --> > <controller type='pci' index='0x0a'> > <model type='dmi-to-pci-bridge'/> > <address type='pci' bus='0x1e' slot='0' function='0'/> > </controller> > <interface type='blah'> > ... > <mac address='00:03:47:7b:63:e6'/> > <address type='pci' bus='0x0a' slot='0x0e' function='0'/> > </interface> > > So I think this will all work. Does anyone see any problems? > > If not, then we can draw it all back to the *current* patchset - support > for multiple PCI buses using the pci-bridge device. For *that*, we only > need to implement the follow bits of the above: > > 1) There will be a new<controller type='pci'> device, with a<model > type='xyz'/> subelement. Initially we will support types "pci-root" and > "pci-bridge" (all the other types discussed above can be added later). > pci-root will have *no<address>* element (and will generate nothing on > the qemu commandline, but will create a 32 slot "bus='0'" to plug PCI > devices into). pci-bridge will have an<address> element, will generate > a -device option on the qemu commandline, and will also create a 32 slot > "bus='n'" to plug PCI devices into. > > 2) for machinetypes that have a PCI bus, the config should have this > controller auto-added: > > <controller type='pci'> > <model type='pci-root'/> > </controller> > > This will make bus='0' available (but add nothing to the qemu > commandline). Any attempt to add a PCI device when there is no bus > available should be an error. > > 3) The way to add more buses will be to add a controller like this: > > <controller type='pci'> > <model type='pci-bridge'/> > </controller> > for legacy PCI, yes;

but for PCIe, one needs PCIe-PPB-up & at least one PCI-PPB-down One _cannot_ have just a single pci-bridge except as driving bridge from a root-complex port.

> 4) When<controller type='usb'> was added, resulting in auto-generated > devices, that caused problems when migrating from a host with newer > libvirt to one with older libvirt. We need to make sure we don't suffer > the same problem this time. See the following two BZes for details > (unless you have a better memory than me! :-): > > https://bugzilla.redhat.com/show_bug.cgi?id=815503 > https://bugzilla.redhat.com/show_bug.cgi?id=856864 > > (and note how danpb eerily prophesied the current pending situation :-) > > > I think everything else about Jan's/Liguang's pci-bridge patches can > remain. >

On 04/15/2013 06:14 PM, Don Dutile wrote: > On 04/15/2013 04:09 PM, Laine Stump wrote: >> On 04/15/2013 06:29 AM, Daniel P. Berrange wrote: >>> On Fri, Apr 12, 2013 at 11:46:15AM -0400, Laine Stump wrote: >>>> On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: >>>>> On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote: >>>>>> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: >>>>>>>>> So if we later allowed for mutiple PCI roots, then we'd have >>>>>>>>> something >>>>>>>>> like >>>>>>>>> >>>>>>>>> <controller type="pci-root" index="0"> >>>>>>>>> <model name="i440FX"/> >>>>>>>>> </controller> >>>>>>>>> <controller type="pci-root" index="1"> >>>>>>>>> <model name="i440FX"/> >>>>>>>>> </controller> >>>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>>>>>>>> <address type='pci' domain='0' bus='0' slot='0''/> >>>>>>>>> </controller> >>>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>>>>>>>> <address type='pci' domain='1' bus='0' slot='0''/> >>>>>>>>> </controller> >>>> >>>> There is a problem here - within a given controller type, we will now >>>> have the possibility of multiple controllers with the same index - the >>>> differentiating attribute will be in the<address> subelement, which >>>> could create some awkwardness. Maybe instead this should be handled >>>> with >>>> a different model of pci controller, and we can add a "domain" >>>> attribute >>>> at the toplevel rather than specifying an<address>? >>> IIUC there is a limit on the number of PCI buses you can create per >>> domain, due to fixed size of PCI addresses. Google suggests to me >>> the limit is 256. So for domain 1, we could just start index at >>> 256, and domain 2 at 512, etc >> >> Okay. Whether we choose that method, or a separate domain attribute, I'm >> satisfied that we'll be able to find a way to solve it when the time >> comes (and it hasn't yet), so we can ignore that problem for now. >> >> > *PLEASE* don't create a new/competing naming/numbering scheme for > differentiating > PCI domains.... as much as I dislike the overuse of the term 'domain', > it's what > is used. No sane person is going to look to assign PCI bus numbers > > 256 in order > to get new/different domains. > The name sucks, but that's what it's called in the code, and what > customers are use to. I infer from this that you're okay with: <controller type='pci' domain='n' index='n'> when defining a new controller (using "index" instead of "bus" is a bit bothersome, but it is following current convention; should we reconsider and just call it "bus" this time?), and: <address domain='n bus='n' slot='n' function='n'/> For specifying where to connect a PCI/PCIe device (including PCI[e] controllers).

>>> >>> >>>> Comment: I'm not quite convinced that we really need the separate >>>> "pci-root" device. Since 1) every pci-root will *always* have either a >>>> pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus >>>> will >>>> only ever be connected to the pci-root, and 3) the pci-bridge that >>>> connects to it will need special handling within the pci-bridge case >>>> anyway, why not: >>>> >>>> 1) eliminate the separate pci-root controller type >>> Ok, lets leave it out - we can always add it later if desired. >> >> Okay. >> > > Not so fast.... something that represents the PCI Root Complex might be > handy -- error handling and embedded devices (like IOMMUs, > intr-remapping table) > come to mind... ACPI tables (if they get duped from real systems) may > need unconventional naming schemes for qemu if an RC isn't modelled. If I understand it correctly, that's what I intend <model type='pcie-root'/> to be - it is what shows up as "bus 0" in the output of lspci. Everything, including embedded/integrated devices, is shown as connected to domain 0 bus 0, isn't it?

What Dan had originally suggested was a separate "root" that only specified "i440FX" or "q35", and that the devices I'm naming "pci-root" and "pcie-root" in this latest draft would just magically connect to that "root" by giving their own connection address as "slot='0'" of their own bus. That seemed redundant to I suggested removing it and just having a special kind of PCI controller that had no connection address.

> >>> >>>> 2) within<controller type='pci'>, a new<model type='pci-root-bus'/> >>>> will be added. >>>> >>>> 3) a pcie-root-bus will automatically be added for q35 >>>> machinetypes, and >>>> pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*") >>>> >>>> 4) model type='pci-root-bus' will behave like pci-bridge, except >>>> that it >>>> will be an implicit device (nothing on qemu commandline) and it won't >>>> need an<address> element (neither will pcie-root-bus). >>>> >>>> 5) to support multiple domains, we can simply add a "domain" attribute >>>> to the toplevel of controller. >>> Or use index numbers modulo 256 to identify domain numbers. >> >> >> Right. One or the other. But we can defer that discussion. >> > Just say 'domain' .... again! ;-) Okay, I'm on board with you (and we have quite awhile to convince anyone who isn't :-)

> >> >> >>> One note on q35 - we need to make sure whatever we do in terms of >>> creating >>> default<controller>s in the XML 'just works' for applications. eg if >>> they >>> define a guest using<type machine="q35">hvm</type>, and then add a >>> <interface>, it should do the right thing wrt PCI >>> addressing/connectivity. >>> We must not require applications to manually add<controller> elements >>> for q35 for things to work. Adding<controller>s must purely be an >>> opt-in >>> for apps which have the detailed knowledge rquired& need full control >>> over bus layout. >> >> Yep. What I see happening is that the place where we currently add >> default controllers will, in the future, automatically add this for >> machinetype pc* and rhel-*: >> >> <controller type='pci'> <!-- implied index='0' --> >> <model type='pci-root'/> >> </controller> >> >> and for machinetype q35* it will add (something like): >> >> <controller type='pci'> <!-- index='0' --> >> <model type='pcie-root'/> >> </controller> >> <controller type='pci'> <!-- index='1' --> >> <model type='dmi-to-pci-bridge'/> >> <address type='pci' bus='0' slot='0x1e'/> >> </controller> >> <controller type='pci'> <!-- index='2' --> >> <model type='pci-bridge'> >> <address type='pci' bus='1' slot='1'/> >> </controller> >> >> The slot-auto-reserve code will look through all pci controllers and >> only auto-reserve slots on controllers appropriate for the given device >> - controller 0 is already inappropriate for PCI devices, and we can mark >> the dmi-to-pci-bridge type as being inappropriate for auto-reserve >> (since, if I recall correctly, I was told that you can't hotplug devices >> on that bus). So, all new PCI devices in the config will get addresses >> with bus='2'. >> >> Of course this means that it will not be possible to switch an existing >> domain config from pc to q35 simply by changing the machinetype - the >> bus number in the address of all devices will need to be changed from 0 >> to 2. But this is another case of "opt in", and already requires editing >> the domain config anyway. If someone creates a brand new q35 machine >> though, all PCI devices will get added with bus='whatever is the bus >> number of the first pci-root or pci-bridge controller' (in this case, >> '2'). >> >> So, here are the proposed pci controller types cleaned up an >> re-summarized, followed by an example. >> >> <controller type='pci'> >> ======================= >> >> This will be used for *all* of the following PCI controller devices >> supported by qemu: >> >> >> <model type='pci-root'/> (implicit/integrated) >> ------------------------ >> >> Upstream: implicit connection to the host >> Downstream: 32 slots (slot 0 reserved), PCI devices only >> qemu commandline: nothing (implicit in the pc-* etc. machinetypes) >> >> This controller represents a pc* (or rhel-*) machine's integrated PCI >> bus (pci.0) and provides places for PCI devices to connect (including >> the "pci-bridge" type of PCI controller). >> >> There is only one of these controllers, and it will *always* be >> index='0', and will have no<address> element. >> > ok. > >> >> <model type='pcie-root'/> (implicit/integrated) >> ------------------------- >> >> Upstream: implicit connection to the host >> Downstream: 32 slots (slot 0 reserved), PCIe devices only, no >> hotplug. >> qemu commandline: nothing (implicit in the q35-* machinetype) >> >> This controller represents a q35's PCI "root complex", and provides >> places for PCIe devices to connect. Examples are root-ports, >> dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet >> devices (do any of those integrated devices that can be connected to >> the pcie-root-bus exist yet?). >> >> There is only one of these controllers, and it will *always* be >> index='0', and will have no<address> element. >> > ok. > >> >> <model type='root-port'/> (ioh3420) >> ------------------------- >> >> Upstream: PCIe, connect to pcie-root-bus *only* (?) >> Downstream: 1 slot, PCIe devices only (?) >> qemu commandline: -device ioh3420,... >> >> These can only connect to the "pcie-root" of a q35. Any PCIe >> devices can connect to it, including an upstream-switch-port. >> > ioh on q35; ich9/10/xx for other intel chipsets > >> >> <model type='upstream-switch-port'/> (x3130-upstream) >> ------------------------------------ >> >> Upstream: PCIe, connect to pcie-root-bus, root-port, or >> downstream-switch-port (?) >> Downstream: 32 slots, connect *only* to downstream-switch-port >> qemu-commandline: -device x3130-upstream >> >> >> This is the upper side of a switch that can multiplex multiple devices >> onto a single port. It's only useful when one or more downstream switch >> ports are connected to it. >> >> >> <model type='downstream-switch-port'/> (xio3130-downstream) >> -------------------------------------- >> >> Upstream: connect *only* to upstream-switch-port >> Downstream: 1 slot, any PCIe device >> qemu commandline: -device xio3130-downstream >> >> You can connect one or more of these to an upstream-switch-port in order >> to effectively plug multiple devices into a single PCIe port. >> > ugh! one cannot have 3130-downstream w/o 3130upstream; > simplify: PCIe-PPB-up; PCIe-PPB-down -- then it can be anything (not > TI, not IDT, not Intel, etc.). Are you just saying that the upstream and downstream ports must be a matching set? That was the intent with naming them (up|down)stream-switch-port - since I've been led to believe that all the different chipsets end up providing the same externally-visible functionality (as long as all the building blocks are used from the same set), I've just used functional names rather than specific chipset/device names, so that the appropriate choice can be made at domain start time, based on what's available in the qemu being run (or the machine type that is chosen). (the "qemu commandline" line merely tells what would be used to implement this controller *with existing qemu 1.4 devices*) (Hmmm - do you think that changing something like the type of upstream and downstream switch ports would lead to Windows requiring a license re-activation? If so, we may need to rethink this and hardcode the specific device that's used :-/) > >> >> <model type='dmi-to-pci-bridge'/> (i82801b11-bridge) >> --------------------------------- Based on Alex's feedback, do we maybe want to name this device "pcie-to-pci-bridge" instead? (or maybe just "pcie-to-pci"?) >> >> (btw, what does "dmi" mean?) >> >> Upstream: pcie-root *only* >> Downstream: 32 slots, any PCI device (including "pci-bridge"), >> no hotplug (?) >> qemu commandline: -device i82801b11-bridge,... >> >> This is the gateway to the world of standard old PCI. >> > why needed? My understanding is that this is the only type of bridge that can be directly connected to pcie-root (the "root complex") and provide plain old PCI slots. > >> >> <model type='pci-bridge'/> (pci-bridge) >> -------------------------- For consistency with the above "pcie-to-pci-bridge" type, should we name this "pci-to-pci-bridge" instead? (or maybe just "pci-to-pci"?) >> >> Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge >> 3) another pci-bridge >> Downstream: any PCI device, 32 slots >> qemu commandline: -device pci-bridge,... >> >> This differs from dmi-to-pci-bridge in that its upstream connection is >> PCI rather than PCIe (so it will work on an i440FX system, which has a >> pci-root rather than pcie-root) and that hotplug is supported. In >> general, if a guest will have any PCI devices, one of these >> controllers should be added, and the PCI devices connected to it >> rather than to the dmi-to-pci-bridge. >> >> ************************************ >> (For q35, we *may* decide to always auto-add a dmi-to-pci-bridge at >> 00:1E.0, and a pci-bridge on slot 1 of the dmi-to-pci-bridge. This >> will allow a continuation of the tradition of simply adding new >> devices to the config without worrying about where they connect.) >> >> >> ============================================================================ >> >> Just to make sure this config model will work, here is the XML to >> replicate the layout (only the ones involved in the PCI tree, along with >> 3 ethernet devices as examples) of the X58 hardware I have sitting under >> my desk (I've attached lspci and virsh nodedev-list --tree output from >> that machine): >> >> >> <controller type='pci' index='0'> >> <model type='pcie-root'/> >> </controller> >> >> <controller type='pci' index='1'> >> <model type='root-port'/> >> <address type='pci' bus='0' slot='1'/> >> </controller> >> >> ( there is a scsi controller connected to bus='1') >> >> >> <controller type='pci' index='2'> >> <model type='root-port'/> >> <address type='pci' bus='0' slot='3'/> >> </controller> >> >> (the VGA controller is connected to bus='2') >> >> <controller type='pci' index='3'> >> <model type='root-port'/> >> <address type='pci' bus='0' slot='7'/> >> </controller> >> >> (PCIe SRIOV network card (in external PCIe slot) connected to bus='3') >> >> <controller type='pci' index='4'> >> <model type='root-port'/> >> <address type='pci' bus='0' slot='0x1c' function='0'/> >> </controller> >> >> (unused PCIe slot available on bus='4') >> >> <!-- pcie-root (0:1c.4) -> root-port (5:0.0) -> onboard >> ethernet -> >> <controller type='pci' index='5'> >> <model type='root-port'/> >> <address type='pci' bus='0' slot='0x1c' function='4'/> >> </controller> >> <interface type='blah'> >> ... >> <mac address='00:27:13:53:db:76'/> >> <address type='pci' bus='5' slot='0' function='0'/> >> </interface> >> >> <!-- more complicated connection to 2nd systemboard ethernet --> >> <!-- pcie-root ->(0:1c:5)root-port -> (6:0.0)upstream-switch-port >> -> (7:3.0)downstream-switch-port -> (9:0.0)ethernet --> >> <controller type='pci' index='6'> >> <model type='root-port'/> >> <address type='pci' bus='0' slot='0x1c' function='5'/> >> </controller> >> <controller type='pci' index='7'> >> <model type='upstream-switch-port'/> >> <address type='pci' bus='6' slot='0' function='0'/> >> </controller> >> <controller type='pci' index='8'> >> <model type='downstream-switch-port'/> >> <address type='pci' bus='7' slot='2' function='0'/> >> </controller> >> <controller type='pci' index='9'> >> <model type='downstream-switch-port'/> >> <address type='pci' bus='7' slot='3' function='0'/> >> </controller> >> <interface type='blah'> >> ... >> <mac address='00:27:13:53:db:77'/> >> <address type='pci' bus='9' slot='0' function='0'/> >> </interface> >> >> >> <!-- old-fashioned PCI ethernet in an external PCI slot --> >> <controller type='pci' index='0x0a'> >> <model type='dmi-to-pci-bridge'/> >> <address type='pci' bus='0x1e' slot='0' function='0'/> >> </controller> >> <interface type='blah'> >> ... >> <mac address='00:03:47:7b:63:e6'/> >> <address type='pci' bus='0x0a' slot='0x0e' function='0'/> >> </interface> >> >> So I think this will all work. Does anyone see any problems? >> >> If not, then we can draw it all back to the *current* patchset - support >> for multiple PCI buses using the pci-bridge device. For *that*, we only >> need to implement the follow bits of the above: >> >> 1) There will be a new<controller type='pci'> device, with a<model >> type='xyz'/> subelement. Initially we will support types "pci-root" and >> "pci-bridge" (all the other types discussed above can be added later). >> pci-root will have *no<address>* element (and will generate nothing on >> the qemu commandline, but will create a 32 slot "bus='0'" to plug PCI >> devices into). pci-bridge will have an<address> element, will generate >> a -device option on the qemu commandline, and will also create a 32 slot >> "bus='n'" to plug PCI devices into. >> >> 2) for machinetypes that have a PCI bus, the config should have this >> controller auto-added: >> >> <controller type='pci'> >> <model type='pci-root'/> >> </controller> >> >> This will make bus='0' available (but add nothing to the qemu >> commandline). Any attempt to add a PCI device when there is no bus >> available should be an error. >> >> 3) The way to add more buses will be to add a controller like this: >> >> <controller type='pci'> >> <model type='pci-bridge'/> >> </controller> >> > for legacy PCI, yes; Right. When I said "current patchset", that's what I was referring to. I started this whole design discussion as a sub-thread of a patcheset that just adds support for the "pci-bridge" device to the existing pc-*/rhel-* machinetypes. I only got into the discussion of all the PCIe controller types because I wanted to make sure that the new controller type we added to support that would be logically expandable to support all of the controllers needed for q35/PCIe support (which is in the queue, but will happen later). > but for PCIe, one needs PCIe-PPB-up & at least one PCI-PPB-down > One _cannot_ have just a single pci-bridge except as driving bridge from > a root-complex port. I had thought that the possibilities for a bridge that provides a legacy PCI bus were: 1) create an i82801b11-bridge device connected to the root complex (or to any root-port or downstream-switch-port). This new bus will provide 32 legacy PCI slots (slot 0 reserved), but devices (currently) can't be hot-plugged. 2) create a pci-bridge device connected to any existing legacy PCI slot (e.g. one of the slots of an i82801b11-bridge). This new bus will provide 32 legacy PCI slots (slot 0 reserved) and hot-plug *is* supported. The (up|down)stream-switch-port based "buses" provide PCIe slots, not PCI. > >> 4) When<controller type='usb'> was added, resulting in auto-generated >> devices, that caused problems when migrating from a host with newer >> libvirt to one with older libvirt. We need to make sure we don't suffer >> the same problem this time. See the following two BZes for details >> (unless you have a better memory than me! :-): >> >> https://bugzilla.redhat.com/show_bug.cgi?id=815503 >> https://bugzilla.redhat.com/show_bug.cgi?id=856864 >> >> (and note how danpb eerily prophesied the current pending situation :-) >> >> >> I think everything else about Jan's/Liguang's pci-bridge patches can >> remain. >> > >

On Fri, 2013-04-12 at 11:46 -0400, Laine Stump wrote: > On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: > > On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote: > >> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: > >>> On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote: > >>>> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: > >>>>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: > >>>>> Actually I do wonder if we should reprent a PCI root as two > >>>>> <controller> elements, one representing the actual PCI root > >>>>> device, and the other representing the host bridge that is > >>>>> built-in. > >>>>> > >>>>> Also we should use the actual model names, not 'pci-root' or > >>>>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever > >>>>> the q35 model name is. > >>>>> > >>>>> - One PCI root with built-in PCI bus (ie todays' setup) > >>>>> > >>>>> <controller type="pci-root" index="0"> > >>>>> <model name="i440FX"/> > >>>>> </controller> > >>>>> <controller type="pci" index="0"> <!-- Host bridge --> > >>>>> <address type='pci' domain='0' bus='0' slot='0''/> > >>>> Isn't this saying that the bridge connects to itself? (since bus 0 is > >>>> this bus) > >>>> > >>>> I understand (again, possibly wrongly) that the builtin PCI bus connects > >>>> to the chipset using its own slot 0 (that's why it's reserved), but > >>>> that's its address on itself. How is this bridge associated with the > >>>> pci-root? > >>>> > >>>> Ah, I *think* I see it - the domain attribute of the pci controller is > >>>> matched to the index of the pci-root controller, correct? But there's > >>>> still something strange about the <address> of the pci controller being > >>>> self-referential. > >>> Yes, the index of the pci-root matches the 'domain' of <address> > >> > >> Okay, then the way that libvirt differentiates between a pci bridge that > >> is connected to the root, and one that is connected to a slot of another > >> bridge is 1) the "bus" attribute of the bridge's <address> matches the > >> "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct? > >> > >> (The corollary of this is that if slot == 0 and bus != index, or bus == > >> index and slot != 0, it is a configuration error). > >> > >> I'm still unclear on the usefulness of the pci-root controller though - > >> all the necessary information is contained in the pci controller, except > >> for the type of root. But in the case of pcie root, I think you're not > >> allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge" > >> (i82801b11-bridge) > > Yes you can connect a pci bridge to pcie-root. > > It's represented as a root complex integrated device. Is this accurate? Per the PCI express spec, any PCI express device needs to have a PCI express capability, which our pci-bridge does not. I think this is one of the main differences for our i82801b11-bridge, that it exposes itself as a root complex integrated endpoint, so we know it's effectively a PCIe-to-PCI bridge.

We'll be asking for trouble if/when we get guest IOMMU support if we are lax about using PCI-to-PCI bridges where we should have PCIe-to-PCI bridges.

There are plenty of examples to the contrary of root complex integrated endpoints without an express capability, but that doesn't make it correct to the spec.

> ARGHH!! Just when I think I'm starting to understand *something* about > these devices... > > (later edit: after some coaching on IRC, I *think* I've got a bit better > handle on it.) > > >>>>> </controller> > >>>>> <interface type='direct'> > >>>>> ... > >>>>> <address type='pci' domain='0' bus='0' slot='3'/> > >>>>> </controller> > >>>>> > >>>>> - One PCI root with built-in PCI bus and extra PCI bridge > >>>>> > >>>>> <controller type="pci-root" index="0"> > >>>>> <model name="i440FX"/> > >>>>> </controller> > >>>>> <controller type="pci" index="0"> <!-- Host bridge --> > >>>>> <address type='pci' domain='0' bus='0' slot='0'/> > >>>>> </controller> > >>>>> <controller type="pci" index="1"> <!-- Additional bridge --> > >>>>> <address type='pci' domain='0' bus='0' slot='1'/> > >>>>> </controller> > >>>>> <interface type='direct'> > >>>>> ... > >>>>> <address type='pci' domain='0' bus='1' slot='3'/> > >>>>> </controller> > >>>>> > >>>>> - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge > >>>>> (ie possible q35 setup) > >>>> Why would a q35 machine have an i440FX pci-root? > >>> It shouldn't, that's a typo > >>> > >>>>> <controller type="pci-root" index="0"> > >>>>> <model name="i440FX"/> > >>>>> </controller> > >>>>> <controller type="pci" index="0"> <!-- Host bridge --> > >>>>> <address type='pci' domain='0' bus='0' slot='0'/> > >>>>> </controller> > >>>>> <controller type="pci" index="1"> <!-- Additional bridge --> > >>>>> <address type='pci' domain='0' bus='0' slot='1'/> > >>>>> </controller> > >>>>> <controller type="pci" index="1"> <!-- Additional bridge --> > >>>>> <address type='pci' domain='0' bus='0' slot='1'/> > >>>>> </controller> > >>>> I think you did a cut-paste here and intended to change something, but > >>>> didn't - those two bridges are identical. > >>> Yep, the slot should be 2 in the second one > >>> > >>>>> <interface type='direct'> > >>>>> ... > >>>>> <address type='pci' domain='0' bus='1' slot='3'/> > >>>>> </controller> > >>>>> > >>>>> So if we later allowed for mutiple PCI roots, then we'd have something > >>>>> like > >>>>> > >>>>> <controller type="pci-root" index="0"> > >>>>> <model name="i440FX"/> > >>>>> </controller> > >>>>> <controller type="pci-root" index="1"> > >>>>> <model name="i440FX"/> > >>>>> </controller> > >>>>> <controller type="pci" index="0"> <!-- Host bridge 1 --> > >>>>> <address type='pci' domain='0' bus='0' slot='0''/> > >>>>> </controller> > >>>>> <controller type="pci" index="0"> <!-- Host bridge 2 --> > >>>>> <address type='pci' domain='1' bus='0' slot='0''/> > >>>>> </controller> > > > There is a problem here - within a given controller type, we will now > have the possibility of multiple controllers with the same index - the > differentiating attribute will be in the <address> subelement, which > could create some awkwardness. Maybe instead this should be handled with > a different model of pci controller, and we can add a "domain" attribute > at the toplevel rather than specifying an <address>? On real hardware, the platform can specify the _BBN (Base Bus Number = bus) and the _SEG (Segment = domain) of the host bridge. So perhaps you want something like: <controller type="pci-host-bridge"> <model name="i440FX"/> <address type="pci-host-bridge-addr" domain='1' bus='0'/> </controller>

"index" is confusing to me.

> >>>>> <interface type='direct'> <!-- NIC on host bridge 2 --> > >>>>> ... > >>>>> <address type='pci' domain='1' bus='0' slot='3'/> > >>>>> </controller> > >>>>> > >>>>> > >>>>> NB this means that 'index' values can be reused against the > >>>>> <controller>, provided they are setup on different pci-roots. > >>>>> > >>>>>> (also note that it might happen that the bus number in libvirt's config > >>>>>> will correspond to the bus numbering that shows up in the guest OS, but > >>>>>> that will just be a happy coincidence) > >>>>>> > >>>>>> Does this make sense? > >>>>> Yep, I think we're fairly close. > >>>> What about the other types of pci controllers that are used by PCIe? We > >>>> should make sure they fit in this model before we settle on it. > >>> What do they do ? > > (The descriptions of different models below tell what each of these > other devices does; in short, they're all just some sort of electronic > Lego to help connect PCI and PCIe devices into a tree). > > Okay, I'll make yet another attempt at understanding these devices, and > suggesting how they can all be described in the XML. I'm thinking that > *all* of the express hubs, switch ports, bridges, etc can be described > in xml in the manner above, i.e.: > > <controller type='pci' index='n'> > <model type='xxx'/> > </controller> > > and that the method for connecting a device to any of them would be by > specifying: > > <address type='pci' domain='n' bus='n' slot='n' function='n'/> > > Any limitations about which devices/controllers can connect to which > controllers, and how many devices can connect to any particular > controller will be derived from the <model type='xxx'/>. (And, as we've > said before, although qemu doesn't assign each of these controllers a > numeric bus id, and although we can make no guarantee that the bus id we > use for a particular controller is what will be used by the guest > BIOS/OS, it's still a convenient notation and works well with other > hypervisors as well as qemu. I'll also note that when I run lspci on an > X58-based machine I have here, *all* of the relationships between all > the devices listed below are described with simple bus:slot.function > numbers.) > > Here is a list of the pci controller model types and their restrictions > (thanks to mst and aw for repeating these over and over to me; I'm sure > I still have made mistakes, but at least it's getting closer). > > > <controller type='pci-root'> > ============================ > > Upstream: nothing > Downstream: only a single pci-root-bus (implied) > qemu commandline: nothing (it's implied in the q35 machinetype) > > Explanation: > > Each machine will have a different controller called "pci-root" as > outlined above by Daniel. Two types of pci-root will be supported: > i440FX and q35. If a pci-root is not spelled out in the config, one will > be auto-added (depending on machinetype). > > An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and > any bridge that has an address of slot='0' on its own bus is, by > definition, connected to a pci-root controller - the two are matched by > setting "domain" in the address of the pci-bridge to "index" of the > pci-root). This bridge can only have PCI devices added. > > A q35 pci-root also implies a different kind of pci-bridge device - one > that can only have PCIe devices/controllers attached, but is otherwise > identical to the pci-bridge added for i440FX. This bus will be called > "root-bus" (Note that there are generally followed conventions for what > can be connected to which slot on this bus, and we will probably follow > those conventions when building a machine, *but* we will not hardcode > this convention into libvirt; each q35 machine will be an empty slate) > > > <controller type='pci'> > ======================= > > This will be used for *all* of the following controller devices > supported by qemu: > > <model type='pcie-root-bus'/> (implicit/integrated) > ---------------------------- > > Upstream: connect to pci-root controller *only* > Downstream: 32 slots, PCIe devices only, no hotplug. > qemu commandline: nothing (implicit in the q35-* machinetype) > > This controller is the bus described above that connects to a q35's > pci-root, and provides places for PCIe devices to connect. Examples are > root-ports, dmi-to-pci-bridges sata controllers, integrated > sound/usb/ethernet devices (do any of those that can be connected to the > pcie-root-bus exist yet?). > > There is only one of these controllers, and it will *always* be > index='0', and will always have the following address: > > <address type='pci' domain='0' bus='0' slot='0' function='0'/> Implicit devices make me nervous, why wouldn't this just be a pcie-root (or pcie-host-bridge)? If we want to support multiple host bridges, there can certainly be more than one, so the index='0' assumption seems to fall apart. > <model type='root-port'/> (ioh3420) > ------------------------- > > Upstream: PCIe, connect to pcie-root-bus *only* (?) yes > Downstream: 1 slot, PCIe devices only (?) yes > qemu commandline: -device ioh3420,... > > These can only connect to the "pcie-root-bus" of of a q35 (implying that > this bus will need to have a different model name than the simple > "pci-bridge" > > > <model type='dmi-to-pci-bridge'/> (i82801b11-bridge) I'm worried this name is either too specific or too generic. What happens when we add a generic pcie-bridge and want to use that instead of the i82801b11-bridge? The guest really only sees this as a PCIe-to-PCI bridge, it just happens that on q35 this attaches at the DMI port of the MCH. > --------------------------------- > > (btw, what does "dmi" mean?) http://en.wikipedia.org/wiki/Direct_Media_Interface > Upstream: pcie-root-bus *only* And only to a specific q35 slot (1e.0) for the i82801b11-bridge. > Downstream: 32 slots, any PCI device, no hotplug (?) Yet, but I think this is where we want to implement ACPI based hotplug. > qemu commandline: -device i82801b11-bridge,... > > > <model type='upstream-switch-port'/> (x3130-upstream) > ------------------------------------ > > Upstream: PCIe, connect to pcie-root-bus, root-port, or > downstream-switch-port (?) yes > Downstream: 32 slots, connect *only* to downstream-switch-port I can't verify that there are 32 slots, mst? I've only setup downstream ports within slot 0. > qemu-commandline: -device x3130-upstream > > > This is the upper side of a switch that can multiplex multiple devices > onto a single port. It's only useful when one or more downstream switch > ports are connected to it. > > <model type='downstream-switch-port'/> (xio3130-downstream) > -------------------------------------- > > Upstream: connect *only* to upstream-switch-port > Downstream: 1 slot, any PCIe device > qemu commandline: -device xio3130-downstream > > You can connect one or more of these to an upstream-switch-port in order > to effectively plug multiple devices into a single PCIe port. > > <model type='pci-bridge'/> (pci-bridge) > -------------------------- > > Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3) > another pci-bridge > Downstream: any PCI device, 32 slots > qemu commandline: -device pci-bridge,... > > This differs from dmi-to-pci-bridge in that its upstream connection is > PCI rather than PCIe (so it will work on an i440FX system, which has no > root PCIe bus) and that hotplug is supported. In general, if a guest > will have any PCI devices, one of these controllers should be added, and > > =============================================================== > > > Comment: I'm not quite convinced that we really need the separate > "pci-root" device. Since 1) every pci-root will *always* have either a > pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will > only ever be connected to the pci-root, and 3) the pci-bridge that > connects to it will need special handling within the pci-bridge case > anyway, why not: > > 1) eliminate the separate pci-root controller type > > 2) within <controller type='pci'>, a new <model type='pci-root-bus'/> > will be added. > > 3) a pcie-root-bus will automatically be added for q35 machinetypes, and > pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*") > > 4) model type='pci-root-bus' will behave like pci-bridge, except that it > will be an implicit device (nothing on qemu commandline) and it won't > need an <address> element (neither will pcie-root-bus). I think they should both have a domain + bus address to make it possible to build multi-domain/multi-host bridge systems. They do not use any slots through. > 5) to support multiple domains, we can simply add a "domain" attribute > to the toplevel of controller. > Or this Wouldn't even be unnecessary if we supported a 'pci-root-addr' address type for the above with the default being domain=0, bus=0? I suppose it doesn't matter whether it's a separate attribute or new address type though. Thanks, Alex

On 04/15/2013 05:58 PM, Michael S. Tsirkin wrote: > On Mon, Apr 15, 2013 at 11:27:03AM -0600, Alex Williamson wrote: >> On Fri, 2013-04-12 at 11:46 -0400, Laine Stump wrote: >>> On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: > it's intended for multi-root support for legacy OSes. > >> "index" is confusing to me. index is being used just because that's been the convention for other controller types - when there are multiple controllers of the same type, each is given an index, and that's used in the "child" devices to indicate which of the parent controllers they connect to. > I'd prefer ID for bus not a number, I'm concerned users will > assume it's bus number and get confused by a mismatch. So you would rather that they were something like this? <controller type='pci' bus='pci.0'> <model type='pci-root'/> </controller> <interface type='blah'> ... <address type='pci' domain='0' bus='pci.0' slot='0' function='0'/> </interface> The problem is that the use of numeric bus IDs is fairly deeply ingrained in libvirt; every existing libvirt guest config has device addresses specifying "bus='0'" Switching to using an alphanumeric ID rather than a simple number would require extra care to maintain backward compatibility with all those existing configs and previous versions of libvirt that might end up being the recipient of xml generated by a newer libvirt. Because of this, at the very least the pci.0 bus must be referred to as bus='0'; once we've done that, we might as well refer to them *all* numerically (anyway, even if names were allowed, I'm sure everybody would just call them '1', '2', (or at the very most "pci.1", "pci.2") etc. anyway.

>>> 5) to support multiple domains, we can simply add a "domain" attribute >>> to the toplevel of controller. >>> >> Or this Wouldn't even be unnecessary if we supported a 'pci-root-addr' >> address type for the above with the default being domain=0, bus=0? I >> suppose it doesn't matter whether it's a separate attribute or new >> address type though. Thanks, I think you're mixing up the purpose of the <address> element vs the "index" attribute in the main <controller> element. To clarify, take this example: <controller type='pci' index='3'> <model type='pci-bridge'/> <address domain='0' bus='1' slot='9' function='0'/> </controller> This controller is connected to slot 9 of the already-existing bus 1. It provides a bus 3 for other devices to connect to. If we wanted to start up a domain 1, we would do something like this: <controller type='pci' domain='1' index='0'> <model type='pci-root'/> </controller> This would give us a PCI bus 0 in domain 1. You could then connect a pci-bridge to it like this: <controller type='pci' domain='1' index='1'> <model type='pci-bridge'/> <address type='pci' domain='1' bus='0' slot='1' function='0'/> </controller> The <address> tells us that this new bus connects to slot 1 of PCI bus 0 in domain 1. The <controller domain='1' index='1'> tells us that there is now a new bus other devices can connect to that is at domain='1' bus='1'.

On 04/15/2013 05:58 PM, Michael S. Tsirkin wrote: > On Mon, Apr 15, 2013 at 11:27:03AM -0600, Alex Williamson wrote: >> On Fri, 2013-04-12 at 11:46 -0400, Laine Stump wrote: >>> On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: >>>> On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote: >>>>> On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: >>>>>> On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote: >>>>>>> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: >>>>>>>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: >>>>>>>> Actually I do wonder if we should reprent a PCI root as two >>>>>>>> <controller> elements, one representing the actual PCI root >>>>>>>> device, and the other representing the host bridge that is >>>>>>>> built-in. >>>>>>>> >>>>>>>> Also we should use the actual model names, not 'pci-root' or >>>>>>>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever >>>>>>>> the q35 model name is. >>>>>>>> >>>>>>>> - One PCI root with built-in PCI bus (ie todays' setup) >>>>>>>> >>>>>>>> <controller type="pci-root" index="0"> >>>>>>>> <model name="i440FX"/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="0"> <!-- Host bridge --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='0''/> >>>>>>> Isn't this saying that the bridge connects to itself? (since bus 0 is >>>>>>> this bus) >>>>>>> >>>>>>> I understand (again, possibly wrongly) that the builtin PCI bus connects >>>>>>> to the chipset using its own slot 0 (that's why it's reserved), but >>>>>>> that's its address on itself. How is this bridge associated with the >>>>>>> pci-root? >>>>>>> >>>>>>> Ah, I *think* I see it - the domain attribute of the pci controller is >>>>>>> matched to the index of the pci-root controller, correct? But there's >>>>>>> still something strange about the <address> of the pci controller being >>>>>>> self-referential. >>>>>> Yes, the index of the pci-root matches the 'domain' of <address> >>>>> Okay, then the way that libvirt differentiates between a pci bridge that >>>>> is connected to the root, and one that is connected to a slot of another >>>>> bridge is 1) the "bus" attribute of the bridge's <address> matches the >>>>> "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct? >>>>> >>>>> (The corollary of this is that if slot == 0 and bus != index, or bus == >>>>> index and slot != 0, it is a configuration error). >>>>> >>>>> I'm still unclear on the usefulness of the pci-root controller though - >>>>> all the necessary information is contained in the pci controller, except >>>>> for the type of root. But in the case of pcie root, I think you're not >>>>> allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge" >>>>> (i82801b11-bridge) >>>> Yes you can connect a pci bridge to pcie-root. >>>> It's represented as a root complex integrated device. >> Is this accurate? Per the PCI express spec, any PCI express device >> needs to have a PCI express capability, which our pci-bridge does not. >> I think this is one of the main differences for our i82801b11-bridge, >> that it exposes itself as a root complex integrated endpoint, so we know >> it's effectively a PCIe-to-PCI bridge. > If it does not have an express link upstream it's not a > PCIe-to-PCI bridge, is it? To my untrained ear it sounds like you're disagreeing with yourself ??? >> We'll be asking for trouble >> if/when we get guest IOMMU support if we are lax about using PCI-to-PCI >> bridges where we should have PCIe-to-PCI bridges. > I recall the spec saying somewhere that integrated endpoints are outside > the root complex hierarchy. I think IOMMU will simply not apply to > these. Correct me if I'm wrong - I think libvirt can ignore this bit of debate other than to use its result to determine which devices are allowed to connect to which other devices, right?

>> There are plenty of >> examples to the contrary of root complex integrated endpoints without an >> express capability, but that doesn't make it correct to the spec. > Is there something in the spec explicitly forbidding this? I merely > find: The PCI Express Capability structure is required for PCI Express > device Functions. > So if it's not an express device it does not have to have > an express capability? > > Maybe we should send an example dump to pci sig and ask them... > >>> ARGHH!! Just when I think I'm starting to understand *something* about >>> these devices... >>> >>> (later edit: after some coaching on IRC, I *think* I've got a bit better >>> handle on it.) (But I guess not good enough :-P) >>> >>>>>>>> </controller> >>>>>>>> <interface type='direct'> >>>>>>>> ... >>>>>>>> <address type='pci' domain='0' bus='0' slot='3'/> >>>>>>>> </controller> >>>>>>>> >>>>>>>> - One PCI root with built-in PCI bus and extra PCI bridge >>>>>>>> >>>>>>>> <controller type="pci-root" index="0"> >>>>>>>> <model name="i440FX"/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="0"> <!-- Host bridge --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='0'/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="1"> <!-- Additional bridge --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='1'/> >>>>>>>> </controller> >>>>>>>> <interface type='direct'> >>>>>>>> ... >>>>>>>> <address type='pci' domain='0' bus='1' slot='3'/> >>>>>>>> </controller> >>>>>>>> >>>>>>>> - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge >>>>>>>> (ie possible q35 setup) >>>>>>> Why would a q35 machine have an i440FX pci-root? >>>>>> It shouldn't, that's a typo >>>>>> >>>>>>>> <controller type="pci-root" index="0"> >>>>>>>> <model name="i440FX"/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="0"> <!-- Host bridge --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='0'/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="1"> <!-- Additional bridge --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='1'/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="1"> <!-- Additional bridge --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='1'/> >>>>>>>> </controller> >>>>>>> I think you did a cut-paste here and intended to change something, but >>>>>>> didn't - those two bridges are identical. >>>>>> Yep, the slot should be 2 in the second one >>>>>> >>>>>>>> <interface type='direct'> >>>>>>>> ... >>>>>>>> <address type='pci' domain='0' bus='1' slot='3'/> >>>>>>>> </controller> >>>>>>>> >>>>>>>> So if we later allowed for mutiple PCI roots, then we'd have something >>>>>>>> like >>>>>>>> >>>>>>>> <controller type="pci-root" index="0"> >>>>>>>> <model name="i440FX"/> >>>>>>>> </controller> >>>>>>>> <controller type="pci-root" index="1"> >>>>>>>> <model name="i440FX"/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>>>>>>> <address type='pci' domain='0' bus='0' slot='0''/> >>>>>>>> </controller> >>>>>>>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>>>>>>> <address type='pci' domain='1' bus='0' slot='0''/> >>>>>>>> </controller> >>> >>> There is a problem here - within a given controller type, we will now >>> have the possibility of multiple controllers with the same index - the >>> differentiating attribute will be in the <address> subelement, which >>> could create some awkwardness. Maybe instead this should be handled with >>> a different model of pci controller, and we can add a "domain" attribute >>> at the toplevel rather than specifying an <address>? >> On real hardware, the platform can specify the _BBN (Base Bus Number = >> bus) and the _SEG (Segment = domain) of the host bridge. So perhaps you >> want something like: >> >> <controller type="pci-host-bridge"> >> <model name="i440FX"/> >> <address type="pci-host-bridge-addr" domain='1' bus='0'/> >> </controller> The <address> element is intended to specify where a device or controller is connected *to*, not what bus/domain it *provides*. I think you're intending for this to provide domain 1 bus 0, so according to existing convention, you would want that information in the <controller> element attributes (e.g. for all other controller types, the generic "index" attribute is used to indicate a bus number when such a thing is appropriate for that type of controller). Anyway, I've simplified this a bit in my latest iteration - there are no separate "root" and "root bus" controllers, just a "pci-root" (for i440FX) or "pcie-root" (for q35), both of which provide a "pci" bus (I'm using the term loosely here), each with different restrictions about what can be connected. > Yes, we could specify segments, though it's not the same as > a domain as linux guests define it (I assume this is what libvirt wants > to call a domain): if memory serves a segment does not have to be a root > based hierarchy, linux domains are all root based. I'm not exactly sure of the meanings/implications of all those terms, but from the point of view of libvirt, as long as we can represent all possible connections between devices using the domain:bus:slot.function notation, I think it doesn't matter too much. > We are better off not specifying BBN for all buses I think - How would you differentiate between the different buses without some sort of identifier? > it's intended for multi-root support for legacy OSes. > >> "index" is confusing to me. index is being used just because that's been the convention for other controller types - when there are multiple controllers of the same type, each is given an index, and that's used in the "child" devices to indicate which of the parent controllers they connect to. > I'd prefer ID for bus not a number, I'm concerned users will > assume it's bus number and get confused by a mismatch. So you would rather that they were something like this? <controller type='pci' bus='pci.0'> <model type='pci-root'/> </controller> <interface type='blah'> ... <address type='pci' domain='0' bus='pci.0' slot='0' function='0'/> </interface> The problem is that the use of numeric bus IDs is fairly deeply ingrained in libvirt; every existing libvirt guest config has device addresses specifying "bus='0'" Switching to using an alphanumeric ID rather than a simple number would require extra care to maintain backward compatibility with all those existing configs and previous versions of libvirt that might end up being the recipient of xml generated by a newer libvirt. Because of this, at the very least the pci.0 bus must be referred to as bus='0'; once we've done that, we might as well refer to them *all* numerically (anyway, even if names were allowed, I'm sure everybody would just call them '1', '2', (or at the very most "pci.1", "pci.2") etc. anyway. >>>>>>>> <interface type='direct'> <!-- NIC on host bridge 2 --> >>>>>>>> ... >>>>>>>> <address type='pci' domain='1' bus='0' slot='3'/> >>>>>>>> </controller> >>>>>>>> >>>>>>>> >>>>>>>> NB this means that 'index' values can be reused against the >>>>>>>> <controller>, provided they are setup on different pci-roots. >>>>>>>> >>>>>>>>> (also note that it might happen that the bus number in libvirt's config >>>>>>>>> will correspond to the bus numbering that shows up in the guest OS, but >>>>>>>>> that will just be a happy coincidence) >>>>>>>>> >>>>>>>>> Does this make sense? >>>>>>>> Yep, I think we're fairly close. >>>>>>> What about the other types of pci controllers that are used by PCIe? We >>>>>>> should make sure they fit in this model before we settle on it. >>>>>> What do they do ? >>> (The descriptions of different models below tell what each of these >>> other devices does; in short, they're all just some sort of electronic >>> Lego to help connect PCI and PCIe devices into a tree). >>> >>> Okay, I'll make yet another attempt at understanding these devices, and >>> suggesting how they can all be described in the XML. I'm thinking that >>> *all* of the express hubs, switch ports, bridges, etc can be described >>> in xml in the manner above, i.e.: >>> >>> <controller type='pci' index='n'> >>> <model type='xxx'/> >>> </controller> >>> >>> and that the method for connecting a device to any of them would be by >>> specifying: >>> >>> <address type='pci' domain='n' bus='n' slot='n' function='n'/> >>> >>> Any limitations about which devices/controllers can connect to which >>> controllers, and how many devices can connect to any particular >>> controller will be derived from the <model type='xxx'/>. (And, as we've >>> said before, although qemu doesn't assign each of these controllers a >>> numeric bus id, and although we can make no guarantee that the bus id we >>> use for a particular controller is what will be used by the guest >>> BIOS/OS, it's still a convenient notation and works well with other >>> hypervisors as well as qemu. I'll also note that when I run lspci on an >>> X58-based machine I have here, *all* of the relationships between all >>> the devices listed below are described with simple bus:slot.function >>> numbers.) >>> >>> Here is a list of the pci controller model types and their restrictions >>> (thanks to mst and aw for repeating these over and over to me; I'm sure >>> I still have made mistakes, but at least it's getting closer). >>> >>> >>> <controller type='pci-root'> >>> ============================ >>> >>> Upstream: nothing >>> Downstream: only a single pci-root-bus (implied) >>> qemu commandline: nothing (it's implied in the q35 machinetype) >>> >>> Explanation: >>> >>> Each machine will have a different controller called "pci-root" as >>> outlined above by Daniel. Two types of pci-root will be supported: >>> i440FX and q35. If a pci-root is not spelled out in the config, one will >>> be auto-added (depending on machinetype). >>> >>> An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and >>> any bridge that has an address of slot='0' on its own bus is, by >>> definition, connected to a pci-root controller - the two are matched by >>> setting "domain" in the address of the pci-bridge to "index" of the >>> pci-root). This bridge can only have PCI devices added. >>> >>> A q35 pci-root also implies a different kind of pci-bridge device - one >>> that can only have PCIe devices/controllers attached, but is otherwise >>> identical to the pci-bridge added for i440FX. This bus will be called >>> "root-bus" (Note that there are generally followed conventions for what >>> can be connected to which slot on this bus, and we will probably follow >>> those conventions when building a machine, *but* we will not hardcode >>> this convention into libvirt; each q35 machine will be an empty slate) >>> >>> >>> <controller type='pci'> >>> ======================= >>> >>> This will be used for *all* of the following controller devices >>> supported by qemu: >>> >>> <model type='pcie-root-bus'/> (implicit/integrated) >>> ---------------------------- >>> >>> Upstream: connect to pci-root controller *only* >>> Downstream: 32 slots, PCIe devices only, no hotplug. >>> qemu commandline: nothing (implicit in the q35-* machinetype) >>> >>> This controller is the bus described above that connects to a q35's >>> pci-root, and provides places for PCIe devices to connect. Examples are >>> root-ports, dmi-to-pci-bridges sata controllers, integrated >>> sound/usb/ethernet devices (do any of those that can be connected to the >>> pcie-root-bus exist yet?). >>> >>> There is only one of these controllers, and it will *always* be >>> index='0', and will always have the following address: >>> >>> <address type='pci' domain='0' bus='0' slot='0' function='0'/> >> Implicit devices make me nervous, why wouldn't this just be a pcie-root >> (or pcie-host-bridge)? If we want to support multiple host bridges, >> there can certainly be more than one, so the index='0' assumption seems >> to fall apart. That's when we need to start talking about a "domain" attribute, like this: <controller type='pci' domain='1' index='0'> <model type='pcie-root-bus'/> </controller> >>> <model type='root-port'/> (ioh3420) >>> ------------------------- >>> >>> Upstream: PCIe, connect to pcie-root-bus *only* (?) >> yes >> >>> Downstream: 1 slot, PCIe devices only (?) >> yes >> >>> qemu commandline: -device ioh3420,... >>> >>> These can only connect to the "pcie-root-bus" of of a q35 (implying that >>> this bus will need to have a different model name than the simple >>> "pci-bridge" >>> >>> >>> <model type='dmi-to-pci-bridge'/> (i82801b11-bridge) >> I'm worried this name is either too specific or too generic. What >> happens when we add a generic pcie-bridge and want to use that instead >> of the i82801b11-bridge? The guest really only sees this as a >> PCIe-to-PCI bridge, it just happens that on q35 this attaches at the DMI >> port of the MCH. Hehe. Just using the name you (Alex) suggested :-) My use of the "generic" device *type* names rather than exact hardware model names is based on the idea that any given machinetype will have a set of these "building block" devices available, and as long as you use everything from the same "set" on a given machine, it doesn't really matter which set you use. Is this a valid assumption? >> >>> --------------------------------- >>> >>> (btw, what does "dmi" mean?) >> http://en.wikipedia.org/wiki/Direct_Media_Interface >> >>> Upstream: pcie-root-bus *only* >> And only to a specific q35 slot (1e.0) for the i82801b11-bridge. >> >>> Downstream: 32 slots, any PCI device, no hotplug (?) >> Yet, but I think this is where we want to implement ACPI based hotplug. Okay, but for now libvirt can just refrain from auto-addressing any user-created devices to that bus; we'll just make sure that there is always a "pci-bridge" plugged into it, and auto-addressed devices will all be put there. In the meantime if someone explicitly addresses a device to connect to the i82801b11-bridge, we'll let them do it, but if they try to hot-unplug it they will get an error. >> >>> qemu commandline: -device i82801b11-bridge,... >>> >>> >>> <model type='upstream-switch-port'/> (x3130-upstream) >>> ------------------------------------ >>> >>> Upstream: PCIe, connect to pcie-root-bus, root-port, or >>> downstream-switch-port (?) >> yes >> >>> Downstream: 32 slots, connect *only* to downstream-switch-port >> I can't verify that there are 32 slots, mst? I've only setup downstream >> ports within slot 0. According to a discussion with Don Dutile on IRC yesterday, the downstream side of an upstream-switch-port has 32 "slots" with 8 "functions" each, and each of these functions can have a downstream-switch-port connected. That said, he told me that in every case he's seen in the real world, all the downstream-switch-ports were connected to "function 0", effectively limiting it to 32 downstreams/upstream. >> >>> qemu-commandline: -device x3130-upstream >>> >>> >>> This is the upper side of a switch that can multiplex multiple devices >>> onto a single port. It's only useful when one or more downstream switch >>> ports are connected to it. >>> >>> <model type='downstream-switch-port'/> (xio3130-downstream) >>> -------------------------------------- >>> >>> Upstream: connect *only* to upstream-switch-port >>> Downstream: 1 slot, any PCIe device >>> qemu commandline: -device xio3130-downstream >>> >>> You can connect one or more of these to an upstream-switch-port in order >>> to effectively plug multiple devices into a single PCIe port. >>> >>> <model type='pci-bridge'/> (pci-bridge) >>> -------------------------- >>> >>> Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3) >>> another pci-bridge >>> Downstream: any PCI device, 32 slots >>> qemu commandline: -device pci-bridge,... >>> >>> This differs from dmi-to-pci-bridge in that its upstream connection is >>> PCI rather than PCIe (so it will work on an i440FX system, which has no >>> root PCIe bus) and that hotplug is supported. In general, if a guest >>> will have any PCI devices, one of these controllers should be added, and >>> >>> =============================================================== >>> >>> >>> Comment: I'm not quite convinced that we really need the separate >>> "pci-root" device. Since 1) every pci-root will *always* have either a >>> pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will >>> only ever be connected to the pci-root, and 3) the pci-bridge that >>> connects to it will need special handling within the pci-bridge case >>> anyway, why not: >>> >>> 1) eliminate the separate pci-root controller type >>> >>> 2) within <controller type='pci'>, a new <model type='pci-root-bus'/> >>> will be added. >>> >>> 3) a pcie-root-bus will automatically be added for q35 machinetypes, and >>> pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*") >>> >>> 4) model type='pci-root-bus' will behave like pci-bridge, except that it >>> will be an implicit device (nothing on qemu commandline) and it won't >>> need an <address> element (neither will pcie-root-bus). >> I think they should both have a domain + bus address to make it possible >> to build multi-domain/multi-host bridge systems. They do not use any >> slots through. Yes. I think I agree with that. But we don't have to implement the multiple-domain stuff today (since qemu doesn't support it yet), and when we do, I think we can just add a "domain" attribute to the main element of pci-root and pcie-root controllers. >>> 5) to support multiple domains, we can simply add a "domain" attribute >>> to the toplevel of controller. >>> >> Or this Wouldn't even be unnecessary if we supported a 'pci-root-addr' >> address type for the above with the default being domain=0, bus=0? I >> suppose it doesn't matter whether it's a separate attribute or new >> address type though. Thanks, I think you're mixing up the purpose of the <address> element vs the "index" attribute in the main <controller> element. To clarify, take this example: <controller type='pci' index='3'> <model type='pci-bridge'/> <address domain='0' bus='1' slot='9' function='0'/> </controller> This controller is connected to slot 9 of the already-existing bus 1. It provides a bus 3 for other devices to connect to. If we wanted to start up a domain 1, we would do something like this: <controller type='pci' domain='1' index='0'> <model type='pci-root'/> </controller> This would give us a PCI bus 0 in domain 1. You could then connect a pci-bridge to it like this: <controller type='pci' domain='1' index='1'> <model type='pci-bridge'/> <address type='pci' domain='1' bus='0' slot='1' function='0'/> </controller> The <address> tells us that this new bus connects to slot 1 of PCI bus 0 in domain 1. The <controller domain='1' index='1'> tells us that there is now a new bus other devices can connect to that is at domain='1' bus='1'. > Also AFAIK there's nothing in the spec that requires bus=0 > to be root. The _BBN hack above is used sometimes to give !=0 > bus numbers to roots. I don't really understand that, but do you think that 1) qemu would ever want/be able to model that, or that 2) anyone would ever have a practical reason for wanting to? It's really cool and all to be able to replicate any possible esoteric hardware configuration in a virtual machine, but it seems like the only practical use of replicating something like that would be for someone wanting to test what their OS does when there's no domain=0 in the hardware...

On Mon, 2013-04-08 at 12:37 -0400, Laine Stump wrote: > On 04/05/2013 03:26 PM, Alex Williamson wrote: > >> Sure. I just mean "pci bus 0" (hmm, but actually this does point out a > >> problem with my logic - the same namespace (well, "numbering space") is > >> used for both pcie and pci buses, so on a q35 system, bus=0 is already > >> taken by pcie.0; that means that the first pci bus would need to use a > >> different bus number anyway, so it wouldn't be so easy to switch an > >> existing domain from pc to q35 - every PCI device would need to have its > >> bus number modified. I suppose that's reasonable to expect, though. > > I would think you'd want to differentiate PCI from PCIe anyway. PCI is > > a bus and you have 32 slots per bus to fill. PCIe is a point-to-point > > link and you really only have slot 0 available. Perhaps that puts them > > in different number spaces already. > > Are you saying that it's okay to have a bus=0 for pci and a different > bus=0 for pcie? In bus=<identifier> the identifier needs to be unique, but it's not a bus #, it's just an identifier.

> I was hoping that what is used in libvirt's config could mirror as > closely as possible the numbering that you see in the output of lspci on > the guest, but it sounds like that numbering is something done at the > whim of the guest, with no basis in (standard) reality, is that right? Correct, the the BIOS determines the initial bus numbers and it can do it however it wants. Most guests won't renumber buses, but they can if they want. It's a lost cause to expect any correlation between the libvirt bus identifier and the actual bus number.

> I think we're starting to get closer to the concrete problem that's > bothering me. As I understand it (and again - "what I understand" has > repeatedly been shown to be incorrect in this thread :-): > > * Ihere are multiple different types of devices that provide a bus with > 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the > e1000 network device, etc) can be plugged into. > > * In the config for those devices, there is a required (auto-generated > if not explicitly provided) <address> element that indicates what > controller that device is plugged into e.g.: > > <interface type='direct'> > ... > <address type='pci' domain='0' bus='0' slot='3' function='0'/> > ... > </interface> The above is not really how QEMU works. QEMU PCI devices take an addr= parameter that specifies the "slot.function". The bus= option is not numeric. That's the identifier value. So if you create a bus with: -device i82801b11-bridge,id=dmi-to-pci-bridge,addr=1e.0 Then to put a device on that bus you'd do: -device e1000,id=e1000-net-0,bus=dmi-to-pci-bridge,addr=0.0

We don't have a way to generate new domains yet, but I imagine it would require a PCI host bridge device and be a parameter to that. For instance: - device pci-host-bridge,domain=1,id=domain1-pcie.0 Which would create a new "pci-root". You would then specify a device in the domain using the same bus= notation. An e1000 on bus=pcie.0 is in domain0, but a nic on bus=domain1-pcie.0 is in domain1. This is all just speculation though. > * domain is always hardcoded to 0, and in the past bus was also always > hardcoded to 0 because until now there has only been a single place > where PCI devices could be connected - the builtin pci.0 bus, which is a > part of the basic "pc" (and some others) virtual machine and provides 32 > slots. Again, bus= is an identifier and I would guess that it will implicitly specify the domain when we get that far. Libvirt specifying a numerical domain:bus:slot.fn and expecting the device to appear there to the guest is a flawed concept.