On 04/04/2013 11:25 AM, Laine Stump wrote:

On 04/03/2013 11:50 AM, Ján Tomko wrote:

...

Change QEMU_PCI_ADDRESS_LAST_SLOT to the number of slots in the bus, not the maximum slot value, to match QEMU_PCI_ADDRESS_LAST_FUNCTION. ---

If you want to be *really* consistent, you should rename these to VIR_QEMU_PCI_ADDRESS_SLOT_LAST and VIR_QEMU_PCI_ADDRESS_FUNCTION_LAST :-)

(i.e. start with "VIR_" and put "LAST" at the end)

A VIR_ prefix might not be necessary, since this define is local to a qemu_ file; but yes, the bit about _LAST being a suffix in our various enum tail markers is a valid point. -- Eric Blake eblake redhat com +1-919-301-3266 Libvirt virtualization library http://libvirt.org

On 04/05/2013 01:38 PM, Daniel P. Berrange wrote:

On Fri, Apr 05, 2013 at 12:32:04PM -0400, Laine Stump wrote:

...

On 04/03/2013 11:50 AM, Ján Tomko wrote:

...

From: liguang <lig.fnst@cn.fujitsu.com>

add a new controller type, then one can define a pci-bridge controller like this: <controller type='pci-bridge' index='0'/> In the next patch we're prohibiting exactly this config (index='0') because the pre-existing pci bus on the "pc-*" machinetypes is already named pci.0. If we don't allow it, we shouldn't include it as an example in the commit log :-) NB, it isn't always named 'pci.0' - on many arches it is merely 'pci'.

Yeah, I'm just using that as a convenient shorthand. The final decision on whether to use pci.0 or pci happens down in the qemuBuildCommandline().

...

More on this - one of the things this points out is that there is no representation in the config of the pci.0 bus, it's just assumed to always be there. That is the case for pc-* machinetypes (and probably several others with PCI buses), but for q35, there is no pci.0 bus in the basic machine, only a pcie.0; if you want a pci.0 on q35 (which *will* be necessary in order to attach any pci devices, so I imagine we will always want one), you have to attach a pcie->pci bridge, which is the device "i82801b11-bridge", to pcie.0. The reason I bring this up here, is I'm wondering:

1) should we have some representation of the default pci.0 bus in the config, even though it is just "always there" for the pc machinetypes and there is no way to disable it, and nothing on the commandline that specifies its existence? Yep, we should be aiming for the XML to fully describe the machine hardware. So since we're adding the concept of PCI controllers/bridges etc to the XML, we should be auto-adding the default bus to the XML.

...

2) For the q35 machinetype, should we just always add an i82801b11-bridge device and name it pci.0? Or should that need to be present in the xml? We've been burnt before auto-adding stuff that ought to have been optional. So I'd tend towards only having the minimal config that is required. If the users want this, let them explicitly ask for the bridge

Also from the apps POV the QEMU device name is irrelevant. The XML config works off the PCI addresses. So there's no need to force/specialcase a i82801b11-bridge to use the name 'pci.0'.

Sure. I just mean "pci bus 0" (hmm, but actually this does point out a problem with my logic - the same namespace (well, "numbering space") is used for both pcie and pci buses, so on a q35 system, bus=0 is already taken by pcie.0; that means that the first pci bus would need to use a different bus number anyway, so it wouldn't be so easy to switch an existing domain from pc to q35 - every PCI device would need to have its bus number modified. I suppose that's reasonable to expect, though.

...

3) Most important - depending on the answers to (1) and (2), should we maybe name this device "pci", and use a different backend depending on index and machinetype? (or alternately explicitly specifiable with a <driver> subelement). To be specific, we would have:

<controller type='pci' index='0'/>

which on pc machinetypes would just be a placeholder in the config (and always inserted if it wasn't there, for machinetypes that have a pci bus). On the q35 machinetype, that same line would equate to adding an i82801b11-bridge device (with source defaulting to bus=pcie.0,addr=1e.0). This would serve several purposes:

a) on pc machinetypes, it would be a visual aid indicating that pci.0 exists, and that index='0' isn't available for a new pci controller.

b) it would make switching a domain config from pc to q35 simpler, since pci.0 would always already be in place for attaching pci devices (including pci.1, pci.2, etc)

c) it would make the config a true complete description of the machine being created.

(I've suggested naming the controller "pci" rather than "pci-bridge" because in the case of a "root" bus like pci.0 it seems to not be a "bridge", but maybe the name "pci-bridge" is always appropriate, even when it's a root bus. Maybe someone with better pci/pcie knowledge can provide an opinion on this) I think "pci" is a little too generic - how about we call it 'pci-root'

Okay, so a separate "pci-root" device along with "pci-bridge"? What I was really hoping was to have all PCI buses represented in a common way in the config. How about a controller called "pci" with different types, "root" and "bridge"? And since they use the same numbering space as pcie buses, maybe the pcie controllers (including the root and the hubs and ???) would be different types of PCI controllers. That would make it easier (i.e. *possible*) to avoid collisions in use of bus numbers. Alex or mst, any advice/opinions on how to represent all the different q35 devices that consume bus numbers in a succinct fashion?

On 04/05/2013 03:26 PM, Alex Williamson wrote:

On Fri, 2013-04-05 at 14:42 -0400, Laine Stump wrote:

...

On 04/05/2013 01:38 PM, Daniel P. Berrange wrote:

...

On Fri, Apr 05, 2013 at 12:32:04PM -0400, Laine Stump wrote:

...

On 04/03/2013 11:50 AM, Ján Tomko wrote:

...

From: liguang <lig.fnst@cn.fujitsu.com>

add a new controller type, then one can define a pci-bridge controller like this: <controller type='pci-bridge' index='0'/> In the next patch we're prohibiting exactly this config (index='0') because the pre-existing pci bus on the "pc-*" machinetypes is already named pci.0. If we don't allow it, we shouldn't include it as an example in the commit log :-) NB, it isn't always named 'pci.0' - on many arches it is merely 'pci'. Yeah, I'm just using that as a convenient shorthand. The final decision on whether to use pci.0 or pci happens down in the qemuBuildCommandline().

...

...

More on this - one of the things this points out is that there is no representation in the config of the pci.0 bus, it's just assumed to always be there. That is the case for pc-* machinetypes (and probably several others with PCI buses), but for q35, there is no pci.0 bus in the basic machine, only a pcie.0; if you want a pci.0 on q35 (which *will* be necessary in order to attach any pci devices, so I imagine we will always want one), you have to attach a pcie->pci bridge, which is the device "i82801b11-bridge", to pcie.0. The reason I bring this up here, is I'm wondering:

1) should we have some representation of the default pci.0 bus in the config, even though it is just "always there" for the pc machinetypes and there is no way to disable it, and nothing on the commandline that specifies its existence? Yep, we should be aiming for the XML to fully describe the machine hardware. So since we're adding the concept of PCI controllers/bridges etc to the XML, we should be auto-adding the default bus to the XML.

...

2) For the q35 machinetype, should we just always add an i82801b11-bridge device and name it pci.0? Or should that need to be present in the xml? We've been burnt before auto-adding stuff that ought to have been optional. So I'd tend towards only having the minimal config that is required. If the users want this, let them explicitly ask for the bridge

Okay. This makes for a larger burden on the user/virt-manager/boxes/libvirt-designer, but does prevent us from setting up an undesirable default that we can't rescue ourselves from :-)

...

...

Also from the apps POV the QEMU device name is irrelevant. The XML config works off the PCI addresses. So there's no need to force/specialcase a i82801b11-bridge to use the name 'pci.0'.

Sure. I just mean "pci bus 0" (hmm, but actually this does point out a problem with my logic - the same namespace (well, "numbering space") is used for both pcie and pci buses, so on a q35 system, bus=0 is already taken by pcie.0; that means that the first pci bus would need to use a different bus number anyway, so it wouldn't be so easy to switch an existing domain from pc to q35 - every PCI device would need to have its bus number modified. I suppose that's reasonable to expect, though. I would think you'd want to differentiate PCI from PCIe anyway. PCI is a bus and you have 32 slots per bus to fill. PCIe is a point-to-point link and you really only have slot 0 available. Perhaps that puts them in different number spaces already.

Are you saying that it's okay to have a bus=0 for pci and a different bus=0 for pcie? I was hoping that what is used in libvirt's config could mirror as closely as possible the numbering that you see in the output of lspci on the guest, but it sounds like that numbering is something done at the whim of the guest, with no basis in (standard) reality, is that right?

...

...

...

3) Most important - depending on the answers to (1) and (2), should we maybe name this device "pci", and use a different backend depending on index and machinetype? (or alternately explicitly specifiable with a <driver> subelement). To be specific, we would have:

<controller type='pci' index='0'/>

which on pc machinetypes would just be a placeholder in the config (and always inserted if it wasn't there, for machinetypes that have a pci bus). On the q35 machinetype, that same line would equate to adding an i82801b11-bridge device (with source defaulting to bus=pcie.0,addr=1e.0). This would serve several purposes:

a) on pc machinetypes, it would be a visual aid indicating that pci.0 exists, and that index='0' isn't available for a new pci controller.

b) it would make switching a domain config from pc to q35 simpler, since pci.0 would always already be in place for attaching pci devices (including pci.1, pci.2, etc)

c) it would make the config a true complete description of the machine being created.

(I've suggested naming the controller "pci" rather than "pci-bridge" because in the case of a "root" bus like pci.0 it seems to not be a "bridge", but maybe the name "pci-bridge" is always appropriate, even when it's a root bus. Maybe someone with better pci/pcie knowledge can provide an opinion on this) I think "pci" is a little too generic - how about we call it 'pci-root' Okay, so a separate "pci-root" device along with "pci-bridge"? What I was really hoping was to have all PCI buses represented in a common way in the config. How about a controller called "pci" with different types, "root" and "bridge"? And since they use the same numbering space as pcie buses, maybe the pcie controllers (including the root and the hubs and ???) would be different types of PCI controllers. That would make it easier (i.e. *possible*) to avoid collisions in use of bus numbers.

Alex or mst, any advice/opinions on how to represent all the different q35 devices that consume bus numbers in a succinct fashion? Note that none of these are really bus numbers, they're just bus identifiers. The BIOS and the guest running define the bus numbers. "root" also has special meaning in PCI, so for instance I wouldn't name a bus behind the i82801b11-bridge "pci-root". Somehow we also need to deal with what can be attached where. For instance a pci-bridge is a PCI device and can only go on a PCI bus. The equivalent structure on PCIe is an upstream switch port with some number of downstream switch ports. Each of those are specific to the bus type.

I think we're starting to get closer to the concrete problem that's bothering me. As I understand it (and again - "what I understand" has repeatedly been shown to be incorrect in this thread :-): * Ihere are multiple different types of devices that provide a bus with 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the e1000 network device, etc) can be plugged into. * In the config for those devices, there is a required (auto-generated if not explicitly provided) <address> element that indicates what controller that device is plugged into e.g.: <interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3' function='0'/> ... </interface> * domain is always hardcoded to 0, and in the past bus was also always hardcoded to 0 because until now there has only been a single place where PCI devices could be connected - the builtin pci.0 bus, which is a part of the basic "pc" (and some others) virtual machine and provides 32 slots. * Now we are adding the ability to define new PCI buses, for now just a single kind - a pci-bridge controller, which itself must connect to an existing PCI slot, and provides 32 new PCI slots. But in the future there will be more different types of controllers that provide one or more PCI slots where PCI devices/controllers can be plugged in. * In these patches adding support for pci-bridge, we are making the assumption that there is a 1:1 correspondence between the "index='n'" attribute of the pci-bridge controller and the "bus='n'" attribute of the <address> element in devices that will be plugged into that controller. So for example if we have: <controller type='pci-bridge' index='1'> <address type='pci' domain='0' bus='0' slot='10' function='0'/> </controller> and then change the <interface> definition above to say "bus='1'", that interface device will plug into this new bus at slot 3. * So let's assume that we add a new controller called "dmi-to-pci-bridge: <controller type='dmi-to-pci-bridge' index='0'/> Ignoring for now the question of what address we give in the definition of *this* device (which is itself problematic - do we need a new "pcie" address type?), if some device is then defined with <address type='pci bus='0' .../> How do we differentiate between that meaning "the pci-ptp controller that is index='0'" and "the pci-bridge controller that is index='0'"? Do we need to expand our <address> element further? If, as I think you suggest, we have multiple different kinds of controllers that provide PCI slots, each with its own namespace, the current pci address element is inadequate to unambiguously describe where a pci device should be plugged in. Perhaps we should be referencing the "<alias name='nnn'/>" element of each controller in the pci address of the target device, e.g.: <controller type='pci-bridge' index='0'> <alias name='pci.0'/> <!-- obviously on a machine with no builtin pci.0! --> </controller/> <controller type='dmi-to-pci-bridge' index='0'> <alias name='dmi-to-pci-bridge.0'/> </controller> <interface type='direct'> ... <address type='pci' controller='dmi-to-pci-bridge.0' slot='3' function='0'/> </interface> (or, since this "controller" attribute really obsolates the numeric "bus" attribute, maybe it could be "bus='dmi-to-pci-bridge.0'", and we could continue to support "bus='0'" for legacy configs). I believe right now the alias name is always auto-generated; we would need to make that so that when explicitly provided it would be guaranteed to never change (and if that's not possible to do in a backward compatible manner, then we need to come up with some new attribute to use in this manner) Alternately, we could add new types to address, one for each new type of controller, then define the devices like this: <interface type='direct'> <address type='pci-bridge' bus='0' slot='3' function='0'/> <interface <interface type='direct'> <address type='dmi-to-pci-bridge' bus='0' slot='3' function='0'/> </interface> (yes, I know you wouldn't want to plug a network device into the dmi-to-pci-bridge directly, this is just for the sake of example) You'll notice that this makes the bus attribute obsolete. (side note: I know that this discussion has gone far beyond just talking about adding a single new type of controller (pci-bridge), but how we do this device will have implications far beyond, so we need to figure it out now.)

For PCIe, we create new buses for root ports (ioh3420), upstream switch ports (xio3130-upstream), downstream switch ports (xio3130-downstream), and the dmi-to-pci bridge (i82801b11-bridge). For PCI, PCI-to-PCI bridges create new buses (pci-bridge and dec-21154).

One of my goals is to move us away from emulation of specific chips and create more devices like pci-bridge that adhere to the standard, but don't try to emulate a specific device. Then we might have "root-port", "pcie-upstream-switch-port", "pcie-downstream-switch-port", and "dmi-to-pci-bridge" (none of these names have been discussed).

That makes sense to me at the level of libvirt, but in qemu don't you need to "emulate specific devices" anyway, in order for the guest OS to operate properly? If that's the case and there are different chips that implement the same functionality in a different manner, how would you decide which of those should be chosen as "the *only" dmi-to-pci-bridge"?

On 04/08/2013 12:48 PM, Daniel P. Berrange wrote:

...

I think we're starting to get closer to the concrete problem that's bothering me. As I understand it (and again - "what I understand" has repeatedly been shown to be incorrect in this thread :-):

* Ihere are multiple different types of devices that provide a bus with 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the e1000 network device, etc) can be plugged into.

* In the config for those devices, there is a required (auto-generated if not explicitly provided) <address> element that indicates what controller that device is plugged into e.g.:

<interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3' function='0'/> ... </interface>

* domain is always hardcoded to 0, and in the past bus was also always hardcoded to 0 because until now there has only been a single place where PCI devices could be connected - the builtin pci.0 bus, which is a part of the basic "pc" (and some others) virtual machine and provides 32 slots.

* Now we are adding the ability to define new PCI buses, for now just a single kind - a pci-bridge controller, which itself must connect to an existing PCI slot, and provides 32 new PCI slots. But in the future there will be more different types of controllers that provide one or more PCI slots where PCI devices/controllers can be plugged in.

* In these patches adding support for pci-bridge, we are making the assumption that there is a 1:1 correspondence between the "index='n'" attribute of the pci-bridge controller and the "bus='n'" attribute of the <address> element in devices that will be plugged into that controller. So for example if we have:

<controller type='pci-bridge' index='1'> <address type='pci' domain='0' bus='0' slot='10' function='0'/> </controller>

and then change the <interface> definition above to say "bus='1'", that interface device will plug into this new bus at slot 3.

* So let's assume that we add a new controller called "dmi-to-pci-bridge":

<controller type='dmi-to-pci-bridge' index='0'/>

Ignoring for now the question of what address we give in the definition of *this* device (which is itself problematic - do we need a new "pcie" address type?), if some device is then defined with

<address type='pci bus='0' .../>

How do we differentiate between that meaning "the pci-ptp controller that is index='0'" and "the pci-bridge controller that is index='0'"? Do we need to expand our <address> element further? If, as I think you suggest, we have multiple different kinds of controllers that provide PCI slots, each with its own namespace, the current pci address element is inadequate to unambiguously describe where a pci device should be plugged in. Hmm yes, you're right - as long as we only have <adress type='pci'>

On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: then all <controller> elements should use type='pci' too, and we should just distinguish based on the model name of the controller. So ignore my previous suggestion to have 'pci-bridge' and 'pci-root' types, we can only use type='pci' on <controller> elements.

Okay, so that means we preserve the correlation between <controller type='pci' index='1'> and <address type='pci' bus='1' ..../> Should the <controller> device use, e.g. <model type='pci-bridge'/> for the model, as is done for <interface> devices? One notable difference is that in the case of <interface> (with the exception of "<model type='virtio'/>"), the model isn't used for anything except passing directly through to qemu (and very recently validating against a list of known interface models), while in the case of controllers with type='pci', different models will have different rules about what they can connect to and what can connect to them, and they will affect what is valid in other devices. An example on a "pc" machinetype that has the builtin PCI bus, one extra pci-pci bridge, and an interface device plugged into slot 3 of the pci-bridge: <controller type='pci' index='0'> <model type='pci-root'/> <!-- builtin pci bus --> </controller> <controller type='pci' index='1'> <model type='pci-bridge'/> </controller> <interface type='direct'> ... <address type='pci' bus='1' slot='3'/> </controller> And for a q35 machinetype that has the root pcie, an i82801b11-bridge connected to slot 1e of that, a pci bridge connected to slot 1 of the i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge: <controller type='pci' index='0'> <model type='pcie-root'/> </controller> <controller type='pci' index='1'> <model type='i82801b11-bridge'/> <!-- [*] --> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci' index='2'> <model type='pci-bridge'/> <address type='pci' bus='1' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' bus='2' slot='3'/> </controller> (note that controllers with model='(pci|pcie)-root' will not have any <address> element, because they exist in the basic machine so we don't need to connect them to anywhere.) (also note that it might happen that the bus number in libvirt's config will correspond to the bus numbering that shows up in the guest OS, but that will just be a happy coincidence) Does this make sense?

On 04/08/2013 03:37 PM, Michael S. Tsirkin wrote:

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:

...

On 04/08/2013 12:48 PM, Daniel P. Berrange wrote:

...

...

I think we're starting to get closer to the concrete problem that's bothering me. As I understand it (and again - "what I understand" has repeatedly been shown to be incorrect in this thread :-):

* Ihere are multiple different types of devices that provide a bus with 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the e1000 network device, etc) can be plugged into.

* In the config for those devices, there is a required (auto-generated if not explicitly provided) <address> element that indicates what controller that device is plugged into e.g.:

<interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3' function='0'/> ... </interface>

* domain is always hardcoded to 0, and in the past bus was also always hardcoded to 0 because until now there has only been a single place where PCI devices could be connected - the builtin pci.0 bus, which is a part of the basic "pc" (and some others) virtual machine and provides 32 slots.

* Now we are adding the ability to define new PCI buses, for now just a single kind - a pci-bridge controller, which itself must connect to an existing PCI slot, and provides 32 new PCI slots. But in the future there will be more different types of controllers that provide one or more PCI slots where PCI devices/controllers can be plugged in.

* In these patches adding support for pci-bridge, we are making the assumption that there is a 1:1 correspondence between the "index='n'" attribute of the pci-bridge controller and the "bus='n'" attribute of the <address> element in devices that will be plugged into that controller. So for example if we have:

<controller type='pci-bridge' index='1'> <address type='pci' domain='0' bus='0' slot='10' function='0'/> </controller>

and then change the <interface> definition above to say "bus='1'", that interface device will plug into this new bus at slot 3.

* So let's assume that we add a new controller called "dmi-to-pci-bridge":

<controller type='dmi-to-pci-bridge' index='0'/>

Ignoring for now the question of what address we give in the definition of *this* device (which is itself problematic - do we need a new "pcie" address type?), if some device is then defined with

<address type='pci bus='0' .../>

How do we differentiate between that meaning "the pci-ptp controller that is index='0'" and "the pci-bridge controller that is index='0'"? Do we need to expand our <address> element further? If, as I think you suggest, we have multiple different kinds of controllers that provide PCI slots, each with its own namespace, the current pci address element is inadequate to unambiguously describe where a pci device should be plugged in. Hmm yes, you're right - as long as we only have <adress type='pci'>

On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: then all <controller> elements should use type='pci' too, and we should just distinguish based on the model name of the controller. So ignore my previous suggestion to have 'pci-bridge' and 'pci-root' types, we can only use type='pci' on <controller> elements. Okay, so that means we preserve the correlation between

<controller type='pci' index='1'>

and

<address type='pci' bus='1' ..../>

Should the <controller> device use, e.g. <model type='pci-bridge'/> for the model, as is done for <interface> devices? One notable difference is that in the case of <interface> (with the exception of "<model type='virtio'/>"), the model isn't used for anything except passing directly through to qemu (and very recently validating against a list of known interface models), while in the case of controllers with type='pci', different models will have different rules about what they can connect to and what can connect to them, and they will affect what is valid in other devices.

An example on a "pc" machinetype that has the builtin PCI bus, one extra pci-pci bridge, and an interface device plugged into slot 3 of the pci-bridge:

<controller type='pci' index='0'> <model type='pci-root'/> <!-- builtin pci bus --> </controller> <controller type='pci' index='1'> <model type='pci-bridge'/> </controller> <interface type='direct'> ... <address type='pci' bus='1' slot='3'/> </controller>

And for a q35 machinetype that has the root pcie, an i82801b11-bridge connected to slot 1e of that, a pci bridge connected to slot 1 of the i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge:

<controller type='pci' index='0'> <model type='pcie-root'/> </controller> <controller type='pci' index='1'> <model type='i82801b11-bridge'/> <!-- [*] --> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci' index='2'> <model type='pci-bridge'/> <address type='pci' bus='1' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' bus='2' slot='3'/> </controller>

(note that controllers with model='(pci|pcie)-root' will not have any <address> element, because they exist in the basic machine so we don't need to connect them to anywhere.)

(also note that it might happen that the bus number in libvirt's config will correspond to the bus numbering that shows up in the guest OS, but that will just be a happy coincidence)

Does this make sense? Confused. So why are you using bus numbers at all?

I'm kind of wondering that too :-) That's why I suggested naming each controller that provided a connection point/bus and using the name instead of a number. I think initially the pci address element was given a bus number because it made the config in libvirt look similar to the view of the devices from the guest (and also that's what is necessary for a PCI address on the host side, e.g. when doing PCI passthrough); maybe now we're only considering keeping it for backwards compatibility and consistency with the hostside address element? Or does using numbers fit better with the paradigms for other hypervisors? (doesn't seem like that should be a problem, because the name can be completely internal to libvirt, and translated into what each different hypervisor needs to have).

On Tue, Apr 09, 2013 at 09:59:50AM +0100, Daniel P. Berrange wrote:

On Mon, Apr 08, 2013 at 10:37:45PM +0300, Michael S. Tsirkin wrote:

...

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:

...

On 04/08/2013 12:48 PM, Daniel P. Berrange wrote:

...

...

I think we're starting to get closer to the concrete problem that's bothering me. As I understand it (and again - "what I understand" has repeatedly been shown to be incorrect in this thread :-):

* Ihere are multiple different types of devices that provide a bus with 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the e1000 network device, etc) can be plugged into.

* In the config for those devices, there is a required (auto-generated if not explicitly provided) <address> element that indicates what controller that device is plugged into e.g.:

<interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3' function='0'/> ... </interface>

* domain is always hardcoded to 0, and in the past bus was also always hardcoded to 0 because until now there has only been a single place where PCI devices could be connected - the builtin pci.0 bus, which is a part of the basic "pc" (and some others) virtual machine and provides 32 slots.

* Now we are adding the ability to define new PCI buses, for now just a single kind - a pci-bridge controller, which itself must connect to an existing PCI slot, and provides 32 new PCI slots. But in the future there will be more different types of controllers that provide one or more PCI slots where PCI devices/controllers can be plugged in.

* In these patches adding support for pci-bridge, we are making the assumption that there is a 1:1 correspondence between the "index='n'" attribute of the pci-bridge controller and the "bus='n'" attribute of the <address> element in devices that will be plugged into that controller. So for example if we have:

<controller type='pci-bridge' index='1'> <address type='pci' domain='0' bus='0' slot='10' function='0'/> </controller>

and then change the <interface> definition above to say "bus='1'", that interface device will plug into this new bus at slot 3.

* So let's assume that we add a new controller called "dmi-to-pci-bridge":

<controller type='dmi-to-pci-bridge' index='0'/>

Ignoring for now the question of what address we give in the definition of *this* device (which is itself problematic - do we need a new "pcie" address type?), if some device is then defined with

<address type='pci bus='0' .../>

How do we differentiate between that meaning "the pci-ptp controller that is index='0'" and "the pci-bridge controller that is index='0'"? Do we need to expand our <address> element further? If, as I think you suggest, we have multiple different kinds of controllers that provide PCI slots, each with its own namespace, the current pci address element is inadequate to unambiguously describe where a pci device should be plugged in. Hmm yes, you're right - as long as we only have <adress type='pci'>

On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: then all <controller> elements should use type='pci' too, and we should just distinguish based on the model name of the controller. So ignore my previous suggestion to have 'pci-bridge' and 'pci-root' types, we can only use type='pci' on <controller> elements.

Okay, so that means we preserve the correlation between

<controller type='pci' index='1'>

and

<address type='pci' bus='1' ..../>

Should the <controller> device use, e.g. <model type='pci-bridge'/> for the model, as is done for <interface> devices? One notable difference is that in the case of <interface> (with the exception of "<model type='virtio'/>"), the model isn't used for anything except passing directly through to qemu (and very recently validating against a list of known interface models), while in the case of controllers with type='pci', different models will have different rules about what they can connect to and what can connect to them, and they will affect what is valid in other devices.

An example on a "pc" machinetype that has the builtin PCI bus, one extra pci-pci bridge, and an interface device plugged into slot 3 of the pci-bridge:

<controller type='pci' index='0'> <model type='pci-root'/> <!-- builtin pci bus --> </controller> <controller type='pci' index='1'> <model type='pci-bridge'/> </controller> <interface type='direct'> ... <address type='pci' bus='1' slot='3'/> </controller>

And for a q35 machinetype that has the root pcie, an i82801b11-bridge connected to slot 1e of that, a pci bridge connected to slot 1 of the i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge:

<controller type='pci' index='0'> <model type='pcie-root'/> </controller> <controller type='pci' index='1'> <model type='i82801b11-bridge'/> <!-- [*] --> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci' index='2'> <model type='pci-bridge'/> <address type='pci' bus='1' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' bus='2' slot='3'/> </controller>

(note that controllers with model='(pci|pcie)-root' will not have any <address> element, because they exist in the basic machine so we don't need to connect them to anywhere.)

(also note that it might happen that the bus number in libvirt's config will correspond to the bus numbering that shows up in the guest OS, but that will just be a happy coincidence)

Does this make sense?

Confused. So why are you using bus numbers at all? It's just wrong.

They are not wrong. We use to link the <address> element to the <controller> element.

Daniel

I see so it's just a number, not bus number in pci sense? Maybe name it bus_id or something.

-- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On 04/09/2013 04:58 AM, Daniel P. Berrange wrote:

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote:

...

On 04/08/2013 12:48 PM, Daniel P. Berrange wrote:

...

...

I think we're starting to get closer to the concrete problem that's bothering me. As I understand it (and again - "what I understand" has repeatedly been shown to be incorrect in this thread :-):

* Ihere are multiple different types of devices that provide a bus with 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the e1000 network device, etc) can be plugged into.

* In the config for those devices, there is a required (auto-generated if not explicitly provided) <address> element that indicates what controller that device is plugged into e.g.:

<interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3' function='0'/> ... </interface>

* domain is always hardcoded to 0, and in the past bus was also always hardcoded to 0 because until now there has only been a single place where PCI devices could be connected - the builtin pci.0 bus, which is a part of the basic "pc" (and some others) virtual machine and provides 32 slots.

* Now we are adding the ability to define new PCI buses, for now just a single kind - a pci-bridge controller, which itself must connect to an existing PCI slot, and provides 32 new PCI slots. But in the future there will be more different types of controllers that provide one or more PCI slots where PCI devices/controllers can be plugged in.

* In these patches adding support for pci-bridge, we are making the assumption that there is a 1:1 correspondence between the "index='n'" attribute of the pci-bridge controller and the "bus='n'" attribute of the <address> element in devices that will be plugged into that controller. So for example if we have:

<controller type='pci-bridge' index='1'> <address type='pci' domain='0' bus='0' slot='10' function='0'/> </controller>

and then change the <interface> definition above to say "bus='1'", that interface device will plug into this new bus at slot 3.

* So let's assume that we add a new controller called "dmi-to-pci-bridge":

<controller type='dmi-to-pci-bridge' index='0'/>

Ignoring for now the question of what address we give in the definition of *this* device (which is itself problematic - do we need a new "pcie" address type?), if some device is then defined with

<address type='pci bus='0' .../>

How do we differentiate between that meaning "the pci-ptp controller that is index='0'" and "the pci-bridge controller that is index='0'"? Do we need to expand our <address> element further? If, as I think you suggest, we have multiple different kinds of controllers that provide PCI slots, each with its own namespace, the current pci address element is inadequate to unambiguously describe where a pci device should be plugged in. Hmm yes, you're right - as long as we only have <adress type='pci'>

On Mon, Apr 08, 2013 at 12:37:49PM -0400, Laine Stump wrote: then all <controller> elements should use type='pci' too, and we should just distinguish based on the model name of the controller. So ignore my previous suggestion to have 'pci-bridge' and 'pci-root' types, we can only use type='pci' on <controller> elements. Okay, so that means we preserve the correlation between

<controller type='pci' index='1'>

and

<address type='pci' bus='1' ..../>

Should the <controller> device use, e.g. <model type='pci-bridge'/> for the model, as is done for <interface> devices? One notable difference is that in the case of <interface> (with the exception of "<model type='virtio'/>"), the model isn't used for anything except passing directly through to qemu (and very recently validating against a list of known interface models), while in the case of controllers with type='pci', different models will have different rules about what they can connect to and what can connect to them, and they will affect what is valid in other devices.

An example on a "pc" machinetype that has the builtin PCI bus, one extra pci-pci bridge, and an interface device plugged into slot 3 of the pci-bridge:

<controller type='pci' index='0'> <model type='pci-root'/> <!-- builtin pci bus --> </controller> <controller type='pci' index='1'> <model type='pci-bridge'/> </controller> <interface type='direct'> ... <address type='pci' bus='1' slot='3'/> </controller>

And for a q35 machinetype that has the root pcie, an i82801b11-bridge connected to slot 1e of that, a pci bridge connected to slot 1 of the i82801b11-bridge, and an interface plugged into slot 3 of the pci-bridge:

<controller type='pci' index='0'> <model type='pcie-root'/> </controller> <controller type='pci' index='1'> <model type='i82801b11-bridge'/> <!-- [*] --> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci' index='2'> <model type='pci-bridge'/> <address type='pci' bus='1' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' bus='2' slot='3'/> </controller>

(note that controllers with model='(pci|pcie)-root' will not have any <address> element, because they exist in the basic machine so we don't need to connect them to anywhere.) Actually I do wonder if we should reprent a PCI root as two <controller> elements, one representing the actual PCI root device, and the other representing the host bridge that is built-in.

Also we should use the actual model names, not 'pci-root' or 'pcie-root' but rather i440FX for "pc" machine type, and whatever the q35 model name is.

- One PCI root with built-in PCI bus (ie todays' setup)

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0''/>

Isn't this saying that the bridge connects to itself? (since bus 0 is this bus) I understand (again, possibly wrongly) that the builtin PCI bus connects to the chipset using its own slot 0 (that's why it's reserved), but that's its address on itself. How is this bridge associated with the pci-root? Ah, I *think* I see it - the domain attribute of the pci controller is matched to the index of the pci-root controller, correct? But there's still something strange about the <address> of the pci controller being self-referential.

</controller> <interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3'/> </controller>

- One PCI root with built-in PCI bus and extra PCI bridge

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller>

- One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge (ie possible q35 setup)

Why would a q35 machine have an i440FX pci-root?

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller>

I think you did a cut-paste here and intended to change something, but didn't - those two bridges are identical.

<interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller>

So if we later allowed for mutiple PCI roots, then we'd have something like

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci-root" index="1"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge 1 --> <address type='pci' domain='0' bus='0' slot='0''/> </controller> <controller type="pci" index="0"> <!-- Host bridge 2 --> <address type='pci' domain='1' bus='0' slot='0''/> </controller> <interface type='direct'> <!-- NIC on host bridge 2 --> ... <address type='pci' domain='1' bus='0' slot='3'/> </controller>

NB this means that 'index' values can be reused against the <controller>, provided they are setup on different pci-roots.

...

(also note that it might happen that the bus number in libvirt's config will correspond to the bus numbering that shows up in the guest OS, but that will just be a happy coincidence)

Does this make sense? Yep, I think we're fairly close.

What about the other types of pci controllers that are used by PCIe? We should make sure they fit in this model before we settle on it.

On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:

On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote:

...

...

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: Actually I do wonder if we should reprent a PCI root as two <controller> elements, one representing the actual PCI root device, and the other representing the host bridge that is built-in.

Also we should use the actual model names, not 'pci-root' or 'pcie-root' but rather i440FX for "pc" machine type, and whatever the q35 model name is.

- One PCI root with built-in PCI bus (ie todays' setup)

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0''/> Isn't this saying that the bridge connects to itself? (since bus 0 is

On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: this bus)

I understand (again, possibly wrongly) that the builtin PCI bus connects to the chipset using its own slot 0 (that's why it's reserved), but that's its address on itself. How is this bridge associated with the pci-root?

Ah, I *think* I see it - the domain attribute of the pci controller is matched to the index of the pci-root controller, correct? But there's still something strange about the <address> of the pci controller being self-referential. Yes, the index of the pci-root matches the 'domain' of <address>

Okay, then the way that libvirt differentiates between a pci bridge that is connected to the root, and one that is connected to a slot of another bridge is 1) the "bus" attribute of the bridge's <address> matches the "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct? (The corollary of this is that if slot == 0 and bus != index, or bus == index and slot != 0, it is a configuration error). I'm still unclear on the usefulness of the pci-root controller though - all the necessary information is contained in the pci controller, except for the type of root. But in the case of pcie root, I think you're not allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge" (i82801b11-bridge)

...

...

</controller> <interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3'/> </controller>

- One PCI root with built-in PCI bus and extra PCI bridge

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller>

- One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge (ie possible q35 setup) Why would a q35 machine have an i440FX pci-root? It shouldn't, that's a typo

...

...

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> I think you did a cut-paste here and intended to change something, but didn't - those two bridges are identical. Yep, the slot should be 2 in the second one

...

...

<interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller>

So if we later allowed for mutiple PCI roots, then we'd have something like

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci-root" index="1"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge 1 --> <address type='pci' domain='0' bus='0' slot='0''/> </controller> <controller type="pci" index="0"> <!-- Host bridge 2 --> <address type='pci' domain='1' bus='0' slot='0''/> </controller> <interface type='direct'> <!-- NIC on host bridge 2 --> ... <address type='pci' domain='1' bus='0' slot='3'/> </controller>

NB this means that 'index' values can be reused against the <controller>, provided they are setup on different pci-roots.

...

(also note that it might happen that the bus number in libvirt's config will correspond to the bus numbering that shows up in the guest OS, but that will just be a happy coincidence)

Does this make sense? Yep, I think we're fairly close. What about the other types of pci controllers that are used by PCIe? We should make sure they fit in this model before we settle on it. What do they do ?

Although I've asked that question, and had it answered, several times now, without a tabular list, I'm still unable to get the answer to that question clear in my mind :-/ Here's the controllers that aw has talked about (in a recent response on a different sub-thread): On 04/05/2013 03:26 PM, Alex Williamson wrote:

For PCIe, we create new buses for root ports (ioh3420), upstream switch ports (xio3130-upstream), downstream switch ports (xio3130-downstream), and the dmi-to-pci bridge (i82801b11-bridge). For PCI, PCI-to-PCI bridges create new buses (pci-bridge and dec-21154).

Alex: what do each of these connect to, and what can be connected to them? root-port (ioh3420) upstream-switch-port (xio3130-upstream) downstream-switch-port (xio3130-downstream) dmi-to-pci-bridge (i82801b11-bridge) We already know/understand this one: pci-bridge; is this identical in behavior/function? dec-21154

One of my goals is to move us away from emulation of specific chips and create more devices like pci-bridge that adhere to the standard, but don't try to emulate a specific device. Then we might have "root-port", "pcie-upstream-switch-port", "pcie-downstream-switch-port", and "dmi-to-pci-bridge" (none of these names have been discussed).

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:

...

On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:

...

On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote:

...

...

On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: Actually I do wonder if we should reprent a PCI root as two <controller> elements, one representing the actual PCI root device, and the other representing the host bridge that is built-in.

Also we should use the actual model names, not 'pci-root' or 'pcie-root' but rather i440FX for "pc" machine type, and whatever the q35 model name is.

- One PCI root with built-in PCI bus (ie todays' setup)

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0''/> Isn't this saying that the bridge connects to itself? (since bus 0 is

On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: this bus)

I understand (again, possibly wrongly) that the builtin PCI bus connects to the chipset using its own slot 0 (that's why it's reserved), but that's its address on itself. How is this bridge associated with the pci-root?

Ah, I *think* I see it - the domain attribute of the pci controller is matched to the index of the pci-root controller, correct? But there's still something strange about the <address> of the pci controller being self-referential. Yes, the index of the pci-root matches the 'domain' of <address>

Okay, then the way that libvirt differentiates between a pci bridge that is connected to the root, and one that is connected to a slot of another bridge is 1) the "bus" attribute of the bridge's <address> matches the "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct?

(The corollary of this is that if slot == 0 and bus != index, or bus == index and slot != 0, it is a configuration error).

I'm still unclear on the usefulness of the pci-root controller though - all the necessary information is contained in the pci controller, except for the type of root. But in the case of pcie root, I think you're not allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge" (i82801b11-bridge) Yes you can connect a pci bridge to pcie-root. It's represented as a root complex integrated device.

ARGHH!! Just when I think I'm starting to understand *something* about these devices... (later edit: after some coaching on IRC, I *think* I've got a bit better handle on it.)

...

...

...

...

</controller> <interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3'/> </controller>

- One PCI root with built-in PCI bus and extra PCI bridge

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller>

- One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge (ie possible q35 setup) Why would a q35 machine have an i440FX pci-root? It shouldn't, that's a typo

...

...

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge --> <address type='pci' domain='0' bus='0' slot='0'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> <controller type="pci" index="1"> <!-- Additional bridge --> <address type='pci' domain='0' bus='0' slot='1'/> </controller> I think you did a cut-paste here and intended to change something, but didn't - those two bridges are identical. Yep, the slot should be 2 in the second one

...

...

<interface type='direct'> ... <address type='pci' domain='0' bus='1' slot='3'/> </controller>

So if we later allowed for mutiple PCI roots, then we'd have something like

<controller type="pci-root" index="0"> <model name="i440FX"/> </controller> <controller type="pci-root" index="1"> <model name="i440FX"/> </controller> <controller type="pci" index="0"> <!-- Host bridge 1 --> <address type='pci' domain='0' bus='0' slot='0''/> </controller> <controller type="pci" index="0"> <!-- Host bridge 2 --> <address type='pci' domain='1' bus='0' slot='0''/> </controller>

There is a problem here - within a given controller type, we will now have the possibility of multiple controllers with the same index - the differentiating attribute will be in the <address> subelement, which could create some awkwardness. Maybe instead this should be handled with a different model of pci controller, and we can add a "domain" attribute at the toplevel rather than specifying an <address>?

...

...

...

...

<interface type='direct'> <!-- NIC on host bridge 2 --> ... <address type='pci' domain='1' bus='0' slot='3'/> </controller>

NB this means that 'index' values can be reused against the <controller>, provided they are setup on different pci-roots.

...

(also note that it might happen that the bus number in libvirt's config will correspond to the bus numbering that shows up in the guest OS, but that will just be a happy coincidence)

Does this make sense? Yep, I think we're fairly close. What about the other types of pci controllers that are used by PCIe? We should make sure they fit in this model before we settle on it. What do they do ?

(The descriptions of different models below tell what each of these other devices does; in short, they're all just some sort of electronic Lego to help connect PCI and PCIe devices into a tree). Okay, I'll make yet another attempt at understanding these devices, and suggesting how they can all be described in the XML. I'm thinking that *all* of the express hubs, switch ports, bridges, etc can be described in xml in the manner above, i.e.: <controller type='pci' index='n'> <model type='xxx'/> </controller> and that the method for connecting a device to any of them would be by specifying: <address type='pci' domain='n' bus='n' slot='n' function='n'/> Any limitations about which devices/controllers can connect to which controllers, and how many devices can connect to any particular controller will be derived from the <model type='xxx'/>. (And, as we've said before, although qemu doesn't assign each of these controllers a numeric bus id, and although we can make no guarantee that the bus id we use for a particular controller is what will be used by the guest BIOS/OS, it's still a convenient notation and works well with other hypervisors as well as qemu. I'll also note that when I run lspci on an X58-based machine I have here, *all* of the relationships between all the devices listed below are described with simple bus:slot.function numbers.) Here is a list of the pci controller model types and their restrictions (thanks to mst and aw for repeating these over and over to me; I'm sure I still have made mistakes, but at least it's getting closer). <controller type='pci-root'> ============================ Upstream: nothing Downstream: only a single pci-root-bus (implied) qemu commandline: nothing (it's implied in the q35 machinetype) Explanation: Each machine will have a different controller called "pci-root" as outlined above by Daniel. Two types of pci-root will be supported: i440FX and q35. If a pci-root is not spelled out in the config, one will be auto-added (depending on machinetype). An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and any bridge that has an address of slot='0' on its own bus is, by definition, connected to a pci-root controller - the two are matched by setting "domain" in the address of the pci-bridge to "index" of the pci-root). This bridge can only have PCI devices added. A q35 pci-root also implies a different kind of pci-bridge device - one that can only have PCIe devices/controllers attached, but is otherwise identical to the pci-bridge added for i440FX. This bus will be called "root-bus" (Note that there are generally followed conventions for what can be connected to which slot on this bus, and we will probably follow those conventions when building a machine, *but* we will not hardcode this convention into libvirt; each q35 machine will be an empty slate) <controller type='pci'> ======================= This will be used for *all* of the following controller devices supported by qemu: <model type='pcie-root-bus'/> (implicit/integrated) ---------------------------- Upstream: connect to pci-root controller *only* Downstream: 32 slots, PCIe devices only, no hotplug. qemu commandline: nothing (implicit in the q35-* machinetype) This controller is the bus described above that connects to a q35's pci-root, and provides places for PCIe devices to connect. Examples are root-ports, dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet devices (do any of those that can be connected to the pcie-root-bus exist yet?). There is only one of these controllers, and it will *always* be index='0', and will always have the following address: <address type='pci' domain='0' bus='0' slot='0' function='0'/> <model type='root-port'/> (ioh3420) ------------------------- Upstream: PCIe, connect to pcie-root-bus *only* (?) Downstream: 1 slot, PCIe devices only (?) qemu commandline: -device ioh3420,... These can only connect to the "pcie-root-bus" of of a q35 (implying that this bus will need to have a different model name than the simple "pci-bridge" <model type='dmi-to-pci-bridge'/> (i82801b11-bridge) --------------------------------- (btw, what does "dmi" mean?) Upstream: pcie-root-bus *only* Downstream: 32 slots, any PCI device, no hotplug (?) qemu commandline: -device i82801b11-bridge,... <model type='upstream-switch-port'/> (x3130-upstream) ------------------------------------ Upstream: PCIe, connect to pcie-root-bus, root-port, or downstream-switch-port (?) Downstream: 32 slots, connect *only* to downstream-switch-port qemu-commandline: -device x3130-upstream This is the upper side of a switch that can multiplex multiple devices onto a single port. It's only useful when one or more downstream switch ports are connected to it. <model type='downstream-switch-port'/> (xio3130-downstream) -------------------------------------- Upstream: connect *only* to upstream-switch-port Downstream: 1 slot, any PCIe device qemu commandline: -device xio3130-downstream You can connect one or more of these to an upstream-switch-port in order to effectively plug multiple devices into a single PCIe port. <model type='pci-bridge'/> (pci-bridge) -------------------------- Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3) another pci-bridge Downstream: any PCI device, 32 slots qemu commandline: -device pci-bridge,... This differs from dmi-to-pci-bridge in that its upstream connection is PCI rather than PCIe (so it will work on an i440FX system, which has no root PCIe bus) and that hotplug is supported. In general, if a guest will have any PCI devices, one of these controllers should be added, and =============================================================== Comment: I'm not quite convinced that we really need the separate "pci-root" device. Since 1) every pci-root will *always* have either a pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will only ever be connected to the pci-root, and 3) the pci-bridge that connects to it will need special handling within the pci-bridge case anyway, why not: 1) eliminate the separate pci-root controller type 2) within <controller type='pci'>, a new <model type='pci-root-bus'/> will be added. 3) a pcie-root-bus will automatically be added for q35 machinetypes, and pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*") 4) model type='pci-root-bus' will behave like pci-bridge, except that it will be an implicit device (nothing on qemu commandline) and it won't need an <address> element (neither will pcie-root-bus). 5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller.

On 04/15/2013 06:29 AM, Daniel P. Berrange wrote:

...

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:

...

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:

...

On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:

...

...

> So if we later allowed for mutiple PCI roots, then we'd have something > like > > <controller type="pci-root" index="0"> > <model name="i440FX"/> > </controller> > <controller type="pci-root" index="1"> > <model name="i440FX"/> > </controller> > <controller type="pci" index="0"> <!-- Host bridge 1 --> > <address type='pci' domain='0' bus='0' slot='0''/> > </controller> > <controller type="pci" index="0"> <!-- Host bridge 2 --> > <address type='pci' domain='1' bus='0' slot='0''/> > </controller>

There is a problem here - within a given controller type, we will now have the possibility of multiple controllers with the same index - the differentiating attribute will be in the <address> subelement, which could create some awkwardness. Maybe instead this should be handled with a different model of pci controller, and we can add a "domain" attribute at the toplevel rather than specifying an <address>? IIUC there is a limit on the number of PCI buses you can create per domain, due to fixed size of PCI addresses. Google suggests to me

On Fri, Apr 12, 2013 at 11:46:15AM -0400, Laine Stump wrote: the limit is 256. So for domain 1, we could just start index at 256, and domain 2 at 512, etc

Okay. Whether we choose that method, or a separate domain attribute, I'm satisfied that we'll be able to find a way to solve it when the time comes (and it hasn't yet), so we can ignore that problem for now.

...

Comment: I'm not quite convinced that we really need the separate "pci-root" device. Since 1) every pci-root will *always* have either a pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will only ever be connected to the pci-root, and 3) the pci-bridge that connects to it will need special handling within the pci-bridge case anyway, why not:

1) eliminate the separate pci-root controller type Ok, lets leave it out - we can always add it later if desired.

Okay.

...

2) within <controller type='pci'>, a new <model type='pci-root-bus'/> will be added.

3) a pcie-root-bus will automatically be added for q35 machinetypes, and pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")

4) model type='pci-root-bus' will behave like pci-bridge, except that it will be an implicit device (nothing on qemu commandline) and it won't need an <address> element (neither will pcie-root-bus).

5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller. Or use index numbers modulo 256 to identify domain numbers.

Right. One or the other. But we can defer that discussion.

One note on q35 - we need to make sure whatever we do in terms of creating default <controller>s in the XML 'just works' for applications. eg if they define a guest using <type machine="q35">hvm</type>, and then add a <interface>, it should do the right thing wrt PCI addressing/connectivity. We must not require applications to manually add <controller> elements for q35 for things to work. Adding <controller>s must purely be an opt-in for apps which have the detailed knowledge rquired & need full control over bus layout.

Yep. What I see happening is that the place where we currently add default controllers will, in the future, automatically add this for machinetype pc* and rhel-*: <controller type='pci'> <!-- implied index='0' --> <model type='pci-root'/> </controller> and for machinetype q35* it will add (something like): <controller type='pci'> <!-- index='0' --> <model type='pcie-root'/> </controller> <controller type='pci'> <!-- index='1' --> <model type='dmi-to-pci-bridge'/> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci'> <!-- index='2' --> <model type='pci-bridge'> <address type='pci' bus='1' slot='1'/> </controller> The slot-auto-reserve code will look through all pci controllers and only auto-reserve slots on controllers appropriate for the given device - controller 0 is already inappropriate for PCI devices, and we can mark the dmi-to-pci-bridge type as being inappropriate for auto-reserve (since, if I recall correctly, I was told that you can't hotplug devices on that bus). So, all new PCI devices in the config will get addresses with bus='2'. Of course this means that it will not be possible to switch an existing domain config from pc to q35 simply by changing the machinetype - the bus number in the address of all devices will need to be changed from 0 to 2. But this is another case of "opt in", and already requires editing the domain config anyway. If someone creates a brand new q35 machine though, all PCI devices will get added with bus='whatever is the bus number of the first pci-root or pci-bridge controller' (in this case, '2'). So, here are the proposed pci controller types cleaned up an re-summarized, followed by an example. <controller type='pci'> ======================= This will be used for *all* of the following PCI controller devices supported by qemu: <model type='pci-root'/> (implicit/integrated) ------------------------ Upstream: implicit connection to the host Downstream: 32 slots (slot 0 reserved), PCI devices only qemu commandline: nothing (implicit in the pc-* etc. machinetypes) This controller represents a pc* (or rhel-*) machine's integrated PCI bus (pci.0) and provides places for PCI devices to connect (including the "pci-bridge" type of PCI controller). There is only one of these controllers, and it will *always* be index='0', and will have no <address> element. <model type='pcie-root'/> (implicit/integrated) ------------------------- Upstream: implicit connection to the host Downstream: 32 slots (slot 0 reserved), PCIe devices only, no hotplug. qemu commandline: nothing (implicit in the q35-* machinetype) This controller represents a q35's PCI "root complex", and provides places for PCIe devices to connect. Examples are root-ports, dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet devices (do any of those integrated devices that can be connected to the pcie-root-bus exist yet?). There is only one of these controllers, and it will *always* be index='0', and will have no <address> element. <model type='root-port'/> (ioh3420) ------------------------- Upstream: PCIe, connect to pcie-root-bus *only* (?) Downstream: 1 slot, PCIe devices only (?) qemu commandline: -device ioh3420,... These can only connect to the "pcie-root" of a q35. Any PCIe devices can connect to it, including an upstream-switch-port. <model type='upstream-switch-port'/> (x3130-upstream) ------------------------------------ Upstream: PCIe, connect to pcie-root-bus, root-port, or downstream-switch-port (?) Downstream: 32 slots, connect *only* to downstream-switch-port qemu-commandline: -device x3130-upstream This is the upper side of a switch that can multiplex multiple devices onto a single port. It's only useful when one or more downstream switch ports are connected to it. <model type='downstream-switch-port'/> (xio3130-downstream) -------------------------------------- Upstream: connect *only* to upstream-switch-port Downstream: 1 slot, any PCIe device qemu commandline: -device xio3130-downstream You can connect one or more of these to an upstream-switch-port in order to effectively plug multiple devices into a single PCIe port. <model type='dmi-to-pci-bridge'/> (i82801b11-bridge) --------------------------------- (btw, what does "dmi" mean?) Upstream: pcie-root *only* Downstream: 32 slots, any PCI device (including "pci-bridge"), no hotplug (?) qemu commandline: -device i82801b11-bridge,... This is the gateway to the world of standard old PCI. <model type='pci-bridge'/> (pci-bridge) -------------------------- Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge 3) another pci-bridge Downstream: any PCI device, 32 slots qemu commandline: -device pci-bridge,... This differs from dmi-to-pci-bridge in that its upstream connection is PCI rather than PCIe (so it will work on an i440FX system, which has a pci-root rather than pcie-root) and that hotplug is supported. In general, if a guest will have any PCI devices, one of these controllers should be added, and the PCI devices connected to it rather than to the dmi-to-pci-bridge. ************************************ (For q35, we *may* decide to always auto-add a dmi-to-pci-bridge at 00:1E.0, and a pci-bridge on slot 1 of the dmi-to-pci-bridge. This will allow a continuation of the tradition of simply adding new devices to the config without worrying about where they connect.) ============================================================================ Just to make sure this config model will work, here is the XML to replicate the layout (only the ones involved in the PCI tree, along with 3 ethernet devices as examples) of the X58 hardware I have sitting under my desk (I've attached lspci and virsh nodedev-list --tree output from that machine): <controller type='pci' index='0'> <model type='pcie-root'/> </controller> <controller type='pci' index='1'> <model type='root-port'/> <address type='pci' bus='0' slot='1'/> </controller> ( there is a scsi controller connected to bus='1') <controller type='pci' index='2'> <model type='root-port'/> <address type='pci' bus='0' slot='3'/> </controller> (the VGA controller is connected to bus='2') <controller type='pci' index='3'> <model type='root-port'/> <address type='pci' bus='0' slot='7'/> </controller> (PCIe SRIOV network card (in external PCIe slot) connected to bus='3') <controller type='pci' index='4'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='0'/> </controller> (unused PCIe slot available on bus='4') <!-- pcie-root (0:1c.4) -> root-port (5:0.0) -> onboard ethernet -> <controller type='pci' index='5'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='4'/> </controller> <interface type='blah'> ... <mac address='00:27:13:53:db:76'/> <address type='pci' bus='5' slot='0' function='0'/> </interface> <!-- more complicated connection to 2nd systemboard ethernet --> <!-- pcie-root ->(0:1c:5)root-port -> (6:0.0)upstream-switch-port -> (7:3.0)downstream-switch-port -> (9:0.0)ethernet --> <controller type='pci' index='6'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='5'/> </controller> <controller type='pci' index='7'> <model type='upstream-switch-port'/> <address type='pci' bus='6' slot='0' function='0'/> </controller> <controller type='pci' index='8'> <model type='downstream-switch-port'/> <address type='pci' bus='7' slot='2' function='0'/> </controller> <controller type='pci' index='9'> <model type='downstream-switch-port'/> <address type='pci' bus='7' slot='3' function='0'/> </controller> <interface type='blah'> ... <mac address='00:27:13:53:db:77'/> <address type='pci' bus='9' slot='0' function='0'/> </interface> <!-- old-fashioned PCI ethernet in an external PCI slot --> <controller type='pci' index='0x0a'> <model type='dmi-to-pci-bridge'/> <address type='pci' bus='0x1e' slot='0' function='0'/> </controller> <interface type='blah'> ... <mac address='00:03:47:7b:63:e6'/> <address type='pci' bus='0x0a' slot='0x0e' function='0'/> </interface> So I think this will all work. Does anyone see any problems? If not, then we can draw it all back to the *current* patchset - support for multiple PCI buses using the pci-bridge device. For *that*, we only need to implement the follow bits of the above: 1) There will be a new <controller type='pci'> device, with a <model type='xyz'/> subelement. Initially we will support types "pci-root" and "pci-bridge" (all the other types discussed above can be added later). pci-root will have *no <address>* element (and will generate nothing on the qemu commandline, but will create a 32 slot "bus='0'" to plug PCI devices into). pci-bridge will have an <address> element, will generate a -device option on the qemu commandline, and will also create a 32 slot "bus='n'" to plug PCI devices into. 2) for machinetypes that have a PCI bus, the config should have this controller auto-added: <controller type='pci'> <model type='pci-root'/> </controller> This will make bus='0' available (but add nothing to the qemu commandline). Any attempt to add a PCI device when there is no bus available should be an error. 3) The way to add more buses will be to add a controller like this: <controller type='pci'> <model type='pci-bridge'/> </controller> 4) When <controller type='usb'> was added, resulting in auto-generated devices, that caused problems when migrating from a host with newer libvirt to one with older libvirt. We need to make sure we don't suffer the same problem this time. See the following two BZes for details (unless you have a better memory than me! :-): https://bugzilla.redhat.com/show_bug.cgi?id=815503 https://bugzilla.redhat.com/show_bug.cgi?id=856864 (and note how danpb eerily prophesied the current pending situation :-) I think everything else about Jan's/Liguang's pci-bridge patches can remain.

On 04/15/2013 06:14 PM, Don Dutile wrote:

On 04/15/2013 04:09 PM, Laine Stump wrote:

...

On 04/15/2013 06:29 AM, Daniel P. Berrange wrote:

...

...

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:

...

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:

...

On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: >>> So if we later allowed for mutiple PCI roots, then we'd have >>> something >>> like >>> >>> <controller type="pci-root" index="0"> >>> <model name="i440FX"/> >>> </controller> >>> <controller type="pci-root" index="1"> >>> <model name="i440FX"/> >>> </controller> >>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>> <address type='pci' domain='0' bus='0' slot='0''/> >>> </controller> >>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>> <address type='pci' domain='1' bus='0' slot='0''/> >>> </controller>

There is a problem here - within a given controller type, we will now have the possibility of multiple controllers with the same index - the differentiating attribute will be in the<address> subelement, which could create some awkwardness. Maybe instead this should be handled with a different model of pci controller, and we can add a "domain" attribute at the toplevel rather than specifying an<address>? IIUC there is a limit on the number of PCI buses you can create per domain, due to fixed size of PCI addresses. Google suggests to me

On Fri, Apr 12, 2013 at 11:46:15AM -0400, Laine Stump wrote: the limit is 256. So for domain 1, we could just start index at 256, and domain 2 at 512, etc

Okay. Whether we choose that method, or a separate domain attribute, I'm satisfied that we'll be able to find a way to solve it when the time comes (and it hasn't yet), so we can ignore that problem for now.

*PLEASE* don't create a new/competing naming/numbering scheme for differentiating PCI domains.... as much as I dislike the overuse of the term 'domain', it's what is used. No sane person is going to look to assign PCI bus numbers > 256 in order to get new/different domains. The name sucks, but that's what it's called in the code, and what customers are use to.

I infer from this that you're okay with: <controller type='pci' domain='n' index='n'> when defining a new controller (using "index" instead of "bus" is a bit bothersome, but it is following current convention; should we reconsider and just call it "bus" this time?), and: <address domain='n bus='n' slot='n' function='n'/> For specifying where to connect a PCI/PCIe device (including PCI[e] controllers).

...

...

...

Comment: I'm not quite convinced that we really need the separate "pci-root" device. Since 1) every pci-root will *always* have either a pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will only ever be connected to the pci-root, and 3) the pci-bridge that connects to it will need special handling within the pci-bridge case anyway, why not:

1) eliminate the separate pci-root controller type Ok, lets leave it out - we can always add it later if desired.

Okay.

Not so fast.... something that represents the PCI Root Complex might be handy -- error handling and embedded devices (like IOMMUs, intr-remapping table) come to mind... ACPI tables (if they get duped from real systems) may need unconventional naming schemes for qemu if an RC isn't modelled.

If I understand it correctly, that's what I intend <model type='pcie-root'/> to be - it is what shows up as "bus 0" in the output of lspci. Everything, including embedded/integrated devices, is shown as connected to domain 0 bus 0, isn't it? What Dan had originally suggested was a separate "root" that only specified "i440FX" or "q35", and that the devices I'm naming "pci-root" and "pcie-root" in this latest draft would just magically connect to that "root" by giving their own connection address as "slot='0'" of their own bus. That seemed redundant to I suggested removing it and just having a special kind of PCI controller that had no connection address.

...

...

...

2) within<controller type='pci'>, a new<model type='pci-root-bus'/> will be added.

3) a pcie-root-bus will automatically be added for q35 machinetypes, and pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")

4) model type='pci-root-bus' will behave like pci-bridge, except that it will be an implicit device (nothing on qemu commandline) and it won't need an<address> element (neither will pcie-root-bus).

5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller. Or use index numbers modulo 256 to identify domain numbers.

Right. One or the other. But we can defer that discussion.

Just say 'domain' .... again! ;-)

Okay, I'm on board with you (and we have quite awhile to convince anyone who isn't :-)

...

...

One note on q35 - we need to make sure whatever we do in terms of creating default<controller>s in the XML 'just works' for applications. eg if they define a guest using<type machine="q35">hvm</type>, and then add a <interface>, it should do the right thing wrt PCI addressing/connectivity. We must not require applications to manually add<controller> elements for q35 for things to work. Adding<controller>s must purely be an opt-in for apps which have the detailed knowledge rquired& need full control over bus layout.

Yep. What I see happening is that the place where we currently add default controllers will, in the future, automatically add this for machinetype pc* and rhel-*:

<controller type='pci'> <!-- implied index='0' --> <model type='pci-root'/> </controller>

and for machinetype q35* it will add (something like):

<controller type='pci'> <!-- index='0' --> <model type='pcie-root'/> </controller> <controller type='pci'> <!-- index='1' --> <model type='dmi-to-pci-bridge'/> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci'> <!-- index='2' --> <model type='pci-bridge'> <address type='pci' bus='1' slot='1'/> </controller>

The slot-auto-reserve code will look through all pci controllers and only auto-reserve slots on controllers appropriate for the given device - controller 0 is already inappropriate for PCI devices, and we can mark the dmi-to-pci-bridge type as being inappropriate for auto-reserve (since, if I recall correctly, I was told that you can't hotplug devices on that bus). So, all new PCI devices in the config will get addresses with bus='2'.

Of course this means that it will not be possible to switch an existing domain config from pc to q35 simply by changing the machinetype - the bus number in the address of all devices will need to be changed from 0 to 2. But this is another case of "opt in", and already requires editing the domain config anyway. If someone creates a brand new q35 machine though, all PCI devices will get added with bus='whatever is the bus number of the first pci-root or pci-bridge controller' (in this case, '2').

So, here are the proposed pci controller types cleaned up an re-summarized, followed by an example.

<controller type='pci'> =======================

This will be used for *all* of the following PCI controller devices supported by qemu:

<model type='pci-root'/> (implicit/integrated) ------------------------

Upstream: implicit connection to the host Downstream: 32 slots (slot 0 reserved), PCI devices only qemu commandline: nothing (implicit in the pc-* etc. machinetypes)

This controller represents a pc* (or rhel-*) machine's integrated PCI bus (pci.0) and provides places for PCI devices to connect (including the "pci-bridge" type of PCI controller).

There is only one of these controllers, and it will *always* be index='0', and will have no<address> element.

ok.

...

<model type='pcie-root'/> (implicit/integrated) -------------------------

Upstream: implicit connection to the host Downstream: 32 slots (slot 0 reserved), PCIe devices only, no hotplug. qemu commandline: nothing (implicit in the q35-* machinetype)

This controller represents a q35's PCI "root complex", and provides places for PCIe devices to connect. Examples are root-ports, dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet devices (do any of those integrated devices that can be connected to the pcie-root-bus exist yet?).

There is only one of these controllers, and it will *always* be index='0', and will have no<address> element.

ok.

...

<model type='root-port'/> (ioh3420) -------------------------

Upstream: PCIe, connect to pcie-root-bus *only* (?) Downstream: 1 slot, PCIe devices only (?) qemu commandline: -device ioh3420,...

These can only connect to the "pcie-root" of a q35. Any PCIe devices can connect to it, including an upstream-switch-port.

ioh on q35; ich9/10/xx for other intel chipsets

...

<model type='upstream-switch-port'/> (x3130-upstream) ------------------------------------

Upstream: PCIe, connect to pcie-root-bus, root-port, or downstream-switch-port (?) Downstream: 32 slots, connect *only* to downstream-switch-port qemu-commandline: -device x3130-upstream

This is the upper side of a switch that can multiplex multiple devices onto a single port. It's only useful when one or more downstream switch ports are connected to it.

<model type='downstream-switch-port'/> (xio3130-downstream) --------------------------------------

Upstream: connect *only* to upstream-switch-port Downstream: 1 slot, any PCIe device qemu commandline: -device xio3130-downstream

You can connect one or more of these to an upstream-switch-port in order to effectively plug multiple devices into a single PCIe port.

ugh! one cannot have 3130-downstream w/o 3130upstream; simplify: PCIe-PPB-up; PCIe-PPB-down -- then it can be anything (not TI, not IDT, not Intel, etc.).

Are you just saying that the upstream and downstream ports must be a matching set? That was the intent with naming them (up|down)stream-switch-port - since I've been led to believe that all the different chipsets end up providing the same externally-visible functionality (as long as all the building blocks are used from the same set), I've just used functional names rather than specific chipset/device names, so that the appropriate choice can be made at domain start time, based on what's available in the qemu being run (or the machine type that is chosen). (the "qemu commandline" line merely tells what would be used to implement this controller *with existing qemu 1.4 devices*) (Hmmm - do you think that changing something like the type of upstream and downstream switch ports would lead to Windows requiring a license re-activation? If so, we may need to rethink this and hardcode the specific device that's used :-/)

...

<model type='dmi-to-pci-bridge'/> (i82801b11-bridge) ---------------------------------

Based on Alex's feedback, do we maybe want to name this device "pcie-to-pci-bridge" instead? (or maybe just "pcie-to-pci"?)

...

(btw, what does "dmi" mean?)

Upstream: pcie-root *only* Downstream: 32 slots, any PCI device (including "pci-bridge"), no hotplug (?) qemu commandline: -device i82801b11-bridge,...

This is the gateway to the world of standard old PCI.

why needed?

My understanding is that this is the only type of bridge that can be directly connected to pcie-root (the "root complex") and provide plain old PCI slots.

...

<model type='pci-bridge'/> (pci-bridge) --------------------------

For consistency with the above "pcie-to-pci-bridge" type, should we name this "pci-to-pci-bridge" instead? (or maybe just "pci-to-pci"?)

...

Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge 3) another pci-bridge Downstream: any PCI device, 32 slots qemu commandline: -device pci-bridge,...

This differs from dmi-to-pci-bridge in that its upstream connection is PCI rather than PCIe (so it will work on an i440FX system, which has a pci-root rather than pcie-root) and that hotplug is supported. In general, if a guest will have any PCI devices, one of these controllers should be added, and the PCI devices connected to it rather than to the dmi-to-pci-bridge.

************************************ (For q35, we *may* decide to always auto-add a dmi-to-pci-bridge at 00:1E.0, and a pci-bridge on slot 1 of the dmi-to-pci-bridge. This will allow a continuation of the tradition of simply adding new devices to the config without worrying about where they connect.)

============================================================================

Just to make sure this config model will work, here is the XML to replicate the layout (only the ones involved in the PCI tree, along with 3 ethernet devices as examples) of the X58 hardware I have sitting under my desk (I've attached lspci and virsh nodedev-list --tree output from that machine):

<controller type='pci' index='0'> <model type='pcie-root'/> </controller>

<controller type='pci' index='1'> <model type='root-port'/> <address type='pci' bus='0' slot='1'/> </controller>

( there is a scsi controller connected to bus='1')

<controller type='pci' index='2'> <model type='root-port'/> <address type='pci' bus='0' slot='3'/> </controller>

(the VGA controller is connected to bus='2')

<controller type='pci' index='3'> <model type='root-port'/> <address type='pci' bus='0' slot='7'/> </controller>

(PCIe SRIOV network card (in external PCIe slot) connected to bus='3')

<controller type='pci' index='4'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='0'/> </controller>

(unused PCIe slot available on bus='4')

<!-- pcie-root (0:1c.4) -> root-port (5:0.0) -> onboard ethernet -> <controller type='pci' index='5'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='4'/> </controller> <interface type='blah'> ... <mac address='00:27:13:53:db:76'/> <address type='pci' bus='5' slot='0' function='0'/> </interface>

<!-- more complicated connection to 2nd systemboard ethernet --> <!-- pcie-root ->(0:1c:5)root-port -> (6:0.0)upstream-switch-port -> (7:3.0)downstream-switch-port -> (9:0.0)ethernet --> <controller type='pci' index='6'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='5'/> </controller> <controller type='pci' index='7'> <model type='upstream-switch-port'/> <address type='pci' bus='6' slot='0' function='0'/> </controller> <controller type='pci' index='8'> <model type='downstream-switch-port'/> <address type='pci' bus='7' slot='2' function='0'/> </controller> <controller type='pci' index='9'> <model type='downstream-switch-port'/> <address type='pci' bus='7' slot='3' function='0'/> </controller> <interface type='blah'> ... <mac address='00:27:13:53:db:77'/> <address type='pci' bus='9' slot='0' function='0'/> </interface>

<!-- old-fashioned PCI ethernet in an external PCI slot --> <controller type='pci' index='0x0a'> <model type='dmi-to-pci-bridge'/> <address type='pci' bus='0x1e' slot='0' function='0'/> </controller> <interface type='blah'> ... <mac address='00:03:47:7b:63:e6'/> <address type='pci' bus='0x0a' slot='0x0e' function='0'/> </interface>

So I think this will all work. Does anyone see any problems?

If not, then we can draw it all back to the *current* patchset - support for multiple PCI buses using the pci-bridge device. For *that*, we only need to implement the follow bits of the above:

1) There will be a new<controller type='pci'> device, with a<model type='xyz'/> subelement. Initially we will support types "pci-root" and "pci-bridge" (all the other types discussed above can be added later). pci-root will have *no<address>* element (and will generate nothing on the qemu commandline, but will create a 32 slot "bus='0'" to plug PCI devices into). pci-bridge will have an<address> element, will generate a -device option on the qemu commandline, and will also create a 32 slot "bus='n'" to plug PCI devices into.

2) for machinetypes that have a PCI bus, the config should have this controller auto-added:

<controller type='pci'> <model type='pci-root'/> </controller>

This will make bus='0' available (but add nothing to the qemu commandline). Any attempt to add a PCI device when there is no bus available should be an error.

3) The way to add more buses will be to add a controller like this:

<controller type='pci'> <model type='pci-bridge'/> </controller>

for legacy PCI, yes;

Right. When I said "current patchset", that's what I was referring to. I started this whole design discussion as a sub-thread of a patcheset that just adds support for the "pci-bridge" device to the existing pc-*/rhel-* machinetypes. I only got into the discussion of all the PCIe controller types because I wanted to make sure that the new controller type we added to support that would be logically expandable to support all of the controllers needed for q35/PCIe support (which is in the queue, but will happen later).

but for PCIe, one needs PCIe-PPB-up & at least one PCI-PPB-down One _cannot_ have just a single pci-bridge except as driving bridge from a root-complex port.

I had thought that the possibilities for a bridge that provides a legacy PCI bus were: 1) create an i82801b11-bridge device connected to the root complex (or to any root-port or downstream-switch-port). This new bus will provide 32 legacy PCI slots (slot 0 reserved), but devices (currently) can't be hot-plugged. 2) create a pci-bridge device connected to any existing legacy PCI slot (e.g. one of the slots of an i82801b11-bridge). This new bus will provide 32 legacy PCI slots (slot 0 reserved) and hot-plug *is* supported. The (up|down)stream-switch-port based "buses" provide PCIe slots, not PCI.

...

4) When<controller type='usb'> was added, resulting in auto-generated devices, that caused problems when migrating from a host with newer libvirt to one with older libvirt. We need to make sure we don't suffer the same problem this time. See the following two BZes for details (unless you have a better memory than me! :-):

https://bugzilla.redhat.com/show_bug.cgi?id=815503 https://bugzilla.redhat.com/show_bug.cgi?id=856864

(and note how danpb eerily prophesied the current pending situation :-)

I think everything else about Jan's/Liguang's pci-bridge patches can remain.

On Tue, Apr 16, 2013 at 12:35:29PM -0400, Laine Stump wrote:

On 04/15/2013 06:14 PM, Don Dutile wrote:

...

On 04/15/2013 04:09 PM, Laine Stump wrote:

...

On 04/15/2013 06:29 AM, Daniel P. Berrange wrote:

...

...

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:

...

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote: > On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: >>>> So if we later allowed for mutiple PCI roots, then we'd have >>>> something >>>> like >>>> >>>> <controller type="pci-root" index="0"> >>>> <model name="i440FX"/> >>>> </controller> >>>> <controller type="pci-root" index="1"> >>>> <model name="i440FX"/> >>>> </controller> >>>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>>> <address type='pci' domain='0' bus='0' slot='0''/> >>>> </controller> >>>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>>> <address type='pci' domain='1' bus='0' slot='0''/> >>>> </controller>

There is a problem here - within a given controller type, we will now have the possibility of multiple controllers with the same index - the differentiating attribute will be in the<address> subelement, which could create some awkwardness. Maybe instead this should be handled with a different model of pci controller, and we can add a "domain" attribute at the toplevel rather than specifying an<address>? IIUC there is a limit on the number of PCI buses you can create per domain, due to fixed size of PCI addresses. Google suggests to me

On Fri, Apr 12, 2013 at 11:46:15AM -0400, Laine Stump wrote: the limit is 256. So for domain 1, we could just start index at 256, and domain 2 at 512, etc

Okay. Whether we choose that method, or a separate domain attribute, I'm satisfied that we'll be able to find a way to solve it when the time comes (and it hasn't yet), so we can ignore that problem for now.

*PLEASE* don't create a new/competing naming/numbering scheme for differentiating PCI domains.... as much as I dislike the overuse of the term 'domain', it's what is used. No sane person is going to look to assign PCI bus numbers > 256 in order to get new/different domains. The name sucks, but that's what it's called in the code, and what customers are use to.

I infer from this that you're okay with:

<controller type='pci' domain='n' index='n'>

when defining a new controller (using "index" instead of "bus" is a bit bothersome, but it is following current convention; should we reconsider and just call it "bus" this time?), and:

<address domain='n bus='n' slot='n' function='n'/>

For specifying where to connect a PCI/PCIe device (including PCI[e] controllers).

I think I prefer index to bus, since bus would make people think it's the bus number. I'm still confused as to what does domain mean exactly. If you identify where device is connected using the index value, why do you need a separate domain?

...

...

...

...

Comment: I'm not quite convinced that we really need the separate "pci-root" device. Since 1) every pci-root will *always* have either a pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will only ever be connected to the pci-root, and 3) the pci-bridge that connects to it will need special handling within the pci-bridge case anyway, why not:

1) eliminate the separate pci-root controller type Ok, lets leave it out - we can always add it later if desired.

Okay.

Not so fast.... something that represents the PCI Root Complex might be handy -- error handling and embedded devices (like IOMMUs, intr-remapping table) come to mind... ACPI tables (if they get duped from real systems) may need unconventional naming schemes for qemu if an RC isn't modelled.

If I understand it correctly, that's what I intend <model type='pcie-root'/> to be - it is what shows up as "bus 0" in the output of lspci. Everything, including embedded/integrated devices, is shown as connected to domain 0 bus 0, isn't it?

There does not have to be a bus 0 in the output of lspci, even though current guests enumerate bus numbers from 0 so there currently usually is one, this might not hold in the future. I am not sure whether libvirt uses any magic values itself but I think it's preferable not to have magic numbers. In fact, we only use numbers for the index field for compatibility.

What Dan had originally suggested was a separate "root" that only specified "i440FX" or "q35", and that the devices I'm naming "pci-root" and "pcie-root" in this latest draft would just magically connect to that "root" by giving their own connection address as "slot='0'" of their own bus. That seemed redundant to I suggested removing it and just having a special kind of PCI controller that had no connection address.

Maybe for the root it makes sense to optionally specify domain instead of index in the address.

...

...

...

...

2) within<controller type='pci'>, a new<model type='pci-root-bus'/> will be added.

3) a pcie-root-bus will automatically be added for q35 machinetypes, and pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")

4) model type='pci-root-bus' will behave like pci-bridge, except that it will be an implicit device (nothing on qemu commandline) and it won't need an<address> element (neither will pcie-root-bus).

5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller. Or use index numbers modulo 256 to identify domain numbers.

Right. One or the other. But we can defer that discussion.

Just say 'domain' .... again! ;-)

Okay, I'm on board with you (and we have quite awhile to convince anyone who isn't :-)

Looks sane. So there shouldn't be any need for the domain attribute anywhere as long as you only have one root, right?

...

...

...

One note on q35 - we need to make sure whatever we do in terms of creating default<controller>s in the XML 'just works' for applications. eg if they define a guest using<type machine="q35">hvm</type>, and then add a <interface>, it should do the right thing wrt PCI addressing/connectivity. We must not require applications to manually add<controller> elements for q35 for things to work. Adding<controller>s must purely be an opt-in for apps which have the detailed knowledge rquired& need full control over bus layout.

Yep. What I see happening is that the place where we currently add default controllers will, in the future, automatically add this for machinetype pc* and rhel-*:

<controller type='pci'> <!-- implied index='0' --> <model type='pci-root'/> </controller>

and for machinetype q35* it will add (something like):

<controller type='pci'> <!-- index='0' --> <model type='pcie-root'/> </controller> <controller type='pci'> <!-- index='1' --> <model type='dmi-to-pci-bridge'/> <address type='pci' bus='0' slot='0x1e'/> </controller> <controller type='pci'> <!-- index='2' --> <model type='pci-bridge'> <address type='pci' bus='1' slot='1'/> </controller>

The slot-auto-reserve code will look through all pci controllers and only auto-reserve slots on controllers appropriate for the given device - controller 0 is already inappropriate for PCI devices, and we can mark the dmi-to-pci-bridge type as being inappropriate for auto-reserve (since, if I recall correctly, I was told that you can't hotplug devices on that bus). So, all new PCI devices in the config will get addresses with bus='2'.

Of course this means that it will not be possible to switch an existing domain config from pc to q35 simply by changing the machinetype - the bus number in the address of all devices will need to be changed from 0 to 2. But this is another case of "opt in", and already requires editing the domain config anyway. If someone creates a brand new q35 machine though, all PCI devices will get added with bus='whatever is the bus number of the first pci-root or pci-bridge controller' (in this case, '2').

So, here are the proposed pci controller types cleaned up an re-summarized, followed by an example.

<controller type='pci'> =======================

This will be used for *all* of the following PCI controller devices supported by qemu:

<model type='pci-root'/> (implicit/integrated) ------------------------

Upstream: implicit connection to the host Downstream: 32 slots (slot 0 reserved), PCI devices only qemu commandline: nothing (implicit in the pc-* etc. machinetypes)

This controller represents a pc* (or rhel-*) machine's integrated PCI bus (pci.0) and provides places for PCI devices to connect (including the "pci-bridge" type of PCI controller).

There is only one of these controllers, and it will *always* be index='0', and will have no<address> element.

ok.

...

<model type='pcie-root'/> (implicit/integrated) -------------------------

Upstream: implicit connection to the host Downstream: 32 slots (slot 0 reserved), PCIe devices only, no hotplug. qemu commandline: nothing (implicit in the q35-* machinetype)

This controller represents a q35's PCI "root complex", and provides places for PCIe devices to connect. Examples are root-ports, dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet devices (do any of those integrated devices that can be connected to the pcie-root-bus exist yet?).

There is only one of these controllers, and it will *always* be index='0', and will have no<address> element.

ok.

...

<model type='root-port'/> (ioh3420) -------------------------

Upstream: PCIe, connect to pcie-root-bus *only* (?) Downstream: 1 slot, PCIe devices only (?) qemu commandline: -device ioh3420,...

These can only connect to the "pcie-root" of a q35. Any PCIe devices can connect to it, including an upstream-switch-port.

ioh on q35; ich9/10/xx for other intel chipsets

...

<model type='upstream-switch-port'/> (x3130-upstream) ------------------------------------

Upstream: PCIe, connect to pcie-root-bus, root-port, or downstream-switch-port (?) Downstream: 32 slots, connect *only* to downstream-switch-port qemu-commandline: -device x3130-upstream

This is the upper side of a switch that can multiplex multiple devices onto a single port. It's only useful when one or more downstream switch ports are connected to it.

<model type='downstream-switch-port'/> (xio3130-downstream) --------------------------------------

Upstream: connect *only* to upstream-switch-port Downstream: 1 slot, any PCIe device qemu commandline: -device xio3130-downstream

You can connect one or more of these to an upstream-switch-port in order to effectively plug multiple devices into a single PCIe port.

ugh! one cannot have 3130-downstream w/o 3130upstream; simplify: PCIe-PPB-up; PCIe-PPB-down -- then it can be anything (not TI, not IDT, not Intel, etc.).

Are you just saying that the upstream and downstream ports must be a matching set? That was the intent with naming them (up|down)stream-switch-port - since I've been led to believe that all the different chipsets end up providing the same externally-visible functionality (as long as all the building blocks are used from the same set), I've just used functional names rather than specific chipset/device names, so that the appropriate choice can be made at domain start time, based on what's available in the qemu being run (or the machine type that is chosen).

(the "qemu commandline" line merely tells what would be used to implement this controller *with existing qemu 1.4 devices*)

(Hmmm - do you think that changing something like the type of upstream and downstream switch ports would lead to Windows requiring a license re-activation? If so, we may need to rethink this and hardcode the specific device that's used :-/)

...

...

<model type='dmi-to-pci-bridge'/> (i82801b11-bridge) ---------------------------------

Based on Alex's feedback, do we maybe want to name this device "pcie-to-pci-bridge" instead? (or maybe just "pcie-to-pci"?)

...

...

(btw, what does "dmi" mean?)

Upstream: pcie-root *only* Downstream: 32 slots, any PCI device (including "pci-bridge"), no hotplug (?) qemu commandline: -device i82801b11-bridge,...

This is the gateway to the world of standard old PCI.

why needed?

My understanding is that this is the only type of bridge that can be directly connected to pcie-root (the "root complex") and provide plain old PCI slots.

...

...

<model type='pci-bridge'/> (pci-bridge) --------------------------

For consistency with the above "pcie-to-pci-bridge" type, should we name this "pci-to-pci-bridge" instead? (or maybe just "pci-to-pci"?)

...

...

Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge 3) another pci-bridge Downstream: any PCI device, 32 slots qemu commandline: -device pci-bridge,...

This differs from dmi-to-pci-bridge in that its upstream connection is PCI rather than PCIe (so it will work on an i440FX system, which has a pci-root rather than pcie-root) and that hotplug is supported. In general, if a guest will have any PCI devices, one of these controllers should be added, and the PCI devices connected to it rather than to the dmi-to-pci-bridge.

************************************ (For q35, we *may* decide to always auto-add a dmi-to-pci-bridge at 00:1E.0, and a pci-bridge on slot 1 of the dmi-to-pci-bridge. This will allow a continuation of the tradition of simply adding new devices to the config without worrying about where they connect.)

============================================================================

Just to make sure this config model will work, here is the XML to replicate the layout (only the ones involved in the PCI tree, along with 3 ethernet devices as examples) of the X58 hardware I have sitting under my desk (I've attached lspci and virsh nodedev-list --tree output from that machine):

<controller type='pci' index='0'> <model type='pcie-root'/> </controller>

<controller type='pci' index='1'> <model type='root-port'/> <address type='pci' bus='0' slot='1'/> </controller>

( there is a scsi controller connected to bus='1')

<controller type='pci' index='2'> <model type='root-port'/> <address type='pci' bus='0' slot='3'/> </controller>

(the VGA controller is connected to bus='2')

<controller type='pci' index='3'> <model type='root-port'/> <address type='pci' bus='0' slot='7'/> </controller>

(PCIe SRIOV network card (in external PCIe slot) connected to bus='3')

<controller type='pci' index='4'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='0'/> </controller>

(unused PCIe slot available on bus='4')

<!-- pcie-root (0:1c.4) -> root-port (5:0.0) -> onboard ethernet -> <controller type='pci' index='5'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='4'/> </controller> <interface type='blah'> ... <mac address='00:27:13:53:db:76'/> <address type='pci' bus='5' slot='0' function='0'/> </interface>

<!-- more complicated connection to 2nd systemboard ethernet --> <!-- pcie-root ->(0:1c:5)root-port -> (6:0.0)upstream-switch-port -> (7:3.0)downstream-switch-port -> (9:0.0)ethernet --> <controller type='pci' index='6'> <model type='root-port'/> <address type='pci' bus='0' slot='0x1c' function='5'/> </controller> <controller type='pci' index='7'> <model type='upstream-switch-port'/> <address type='pci' bus='6' slot='0' function='0'/> </controller> <controller type='pci' index='8'> <model type='downstream-switch-port'/> <address type='pci' bus='7' slot='2' function='0'/> </controller> <controller type='pci' index='9'> <model type='downstream-switch-port'/> <address type='pci' bus='7' slot='3' function='0'/> </controller> <interface type='blah'> ... <mac address='00:27:13:53:db:77'/> <address type='pci' bus='9' slot='0' function='0'/> </interface>

<!-- old-fashioned PCI ethernet in an external PCI slot --> <controller type='pci' index='0x0a'> <model type='dmi-to-pci-bridge'/> <address type='pci' bus='0x1e' slot='0' function='0'/> </controller> <interface type='blah'> ... <mac address='00:03:47:7b:63:e6'/> <address type='pci' bus='0x0a' slot='0x0e' function='0'/> </interface>

So I think this will all work. Does anyone see any problems?

If not, then we can draw it all back to the *current* patchset - support for multiple PCI buses using the pci-bridge device. For *that*, we only need to implement the follow bits of the above:

1) There will be a new<controller type='pci'> device, with a<model type='xyz'/> subelement. Initially we will support types "pci-root" and "pci-bridge" (all the other types discussed above can be added later). pci-root will have *no<address>* element (and will generate nothing on the qemu commandline, but will create a 32 slot "bus='0'" to plug PCI devices into). pci-bridge will have an<address> element, will generate a -device option on the qemu commandline, and will also create a 32 slot "bus='n'" to plug PCI devices into.

2) for machinetypes that have a PCI bus, the config should have this controller auto-added:

<controller type='pci'> <model type='pci-root'/> </controller>

This will make bus='0' available (but add nothing to the qemu commandline). Any attempt to add a PCI device when there is no bus available should be an error.

3) The way to add more buses will be to add a controller like this:

<controller type='pci'> <model type='pci-bridge'/> </controller>

for legacy PCI, yes;

Right. When I said "current patchset", that's what I was referring to. I started this whole design discussion as a sub-thread of a patcheset that just adds support for the "pci-bridge" device to the existing pc-*/rhel-* machinetypes. I only got into the discussion of all the PCIe controller types because I wanted to make sure that the new controller type we added to support that would be logically expandable to support all of the controllers needed for q35/PCIe support (which is in the queue, but will happen later).

...

but for PCIe, one needs PCIe-PPB-up & at least one PCI-PPB-down One _cannot_ have just a single pci-bridge except as driving bridge from a root-complex port.

I had thought that the possibilities for a bridge that provides a legacy PCI bus were:

1) create an i82801b11-bridge device connected to the root complex (or to any root-port or downstream-switch-port). This new bus will provide 32 legacy PCI slots (slot 0 reserved), but devices (currently) can't be hot-plugged.

2) create a pci-bridge device connected to any existing legacy PCI slot (e.g. one of the slots of an i82801b11-bridge). This new bus will provide 32 legacy PCI slots (slot 0 reserved) and hot-plug *is* supported.

The (up|down)stream-switch-port based "buses" provide PCIe slots, not PCI.

...

...

4) When<controller type='usb'> was added, resulting in auto-generated devices, that caused problems when migrating from a host with newer libvirt to one with older libvirt. We need to make sure we don't suffer the same problem this time. See the following two BZes for details (unless you have a better memory than me! :-):

https://bugzilla.redhat.com/show_bug.cgi?id=815503 https://bugzilla.redhat.com/show_bug.cgi?id=856864

(and note how danpb eerily prophesied the current pending situation :-)

I think everything else about Jan's/Liguang's pci-bridge patches can remain.

On Mon, Apr 15, 2013 at 11:27:03AM -0600, Alex Williamson wrote:

On Fri, 2013-04-12 at 11:46 -0400, Laine Stump wrote:

...

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:

...

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:

...

On 04/10/2013 05:26 AM, Daniel P. Berrange wrote:

...

On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote:

...

On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: > On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: > Actually I do wonder if we should reprent a PCI root as two > <controller> elements, one representing the actual PCI root > device, and the other representing the host bridge that is > built-in. > > Also we should use the actual model names, not 'pci-root' or > 'pcie-root' but rather i440FX for "pc" machine type, and whatever > the q35 model name is. > > - One PCI root with built-in PCI bus (ie todays' setup) > > <controller type="pci-root" index="0"> > <model name="i440FX"/> > </controller> > <controller type="pci" index="0"> <!-- Host bridge --> > <address type='pci' domain='0' bus='0' slot='0''/> Isn't this saying that the bridge connects to itself? (since bus 0 is this bus)

I understand (again, possibly wrongly) that the builtin PCI bus connects to the chipset using its own slot 0 (that's why it's reserved), but that's its address on itself. How is this bridge associated with the pci-root?

Ah, I *think* I see it - the domain attribute of the pci controller is matched to the index of the pci-root controller, correct? But there's still something strange about the <address> of the pci controller being self-referential. Yes, the index of the pci-root matches the 'domain' of <address>

Okay, then the way that libvirt differentiates between a pci bridge that is connected to the root, and one that is connected to a slot of another bridge is 1) the "bus" attribute of the bridge's <address> matches the "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct?

(The corollary of this is that if slot == 0 and bus != index, or bus == index and slot != 0, it is a configuration error).

I'm still unclear on the usefulness of the pci-root controller though - all the necessary information is contained in the pci controller, except for the type of root. But in the case of pcie root, I think you're not allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge" (i82801b11-bridge) Yes you can connect a pci bridge to pcie-root. It's represented as a root complex integrated device.

Is this accurate? Per the PCI express spec, any PCI express device needs to have a PCI express capability, which our pci-bridge does not. I think this is one of the main differences for our i82801b11-bridge, that it exposes itself as a root complex integrated endpoint, so we know it's effectively a PCIe-to-PCI bridge.

If it does not have an express link upstream it's not a PCIe-to-PCI bridge, is it?

We'll be asking for trouble if/when we get guest IOMMU support if we are lax about using PCI-to-PCI bridges where we should have PCIe-to-PCI bridges.

I recall the spec saying somewhere that integrated endpoints are outside the root complex hierarchy. I think IOMMU will simply not apply to these.

There are plenty of examples to the contrary of root complex integrated endpoints without an express capability, but that doesn't make it correct to the spec.

Is there something in the spec explicitly forbidding this? I merely find: The PCI Express Capability structure is required for PCI Express device Functions. So if it's not an express device it does not have to have an express capability? Maybe we should send an example dump to pci sig and ask them...

...

ARGHH!! Just when I think I'm starting to understand *something* about these devices...

(later edit: after some coaching on IRC, I *think* I've got a bit better handle on it.)

...

...

...

...

> </controller> > <interface type='direct'> > ... > <address type='pci' domain='0' bus='0' slot='3'/> > </controller> > > - One PCI root with built-in PCI bus and extra PCI bridge > > <controller type="pci-root" index="0"> > <model name="i440FX"/> > </controller> > <controller type="pci" index="0"> <!-- Host bridge --> > <address type='pci' domain='0' bus='0' slot='0'/> > </controller> > <controller type="pci" index="1"> <!-- Additional bridge --> > <address type='pci' domain='0' bus='0' slot='1'/> > </controller> > <interface type='direct'> > ... > <address type='pci' domain='0' bus='1' slot='3'/> > </controller> > > - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge > (ie possible q35 setup) Why would a q35 machine have an i440FX pci-root? It shouldn't, that's a typo

...

> <controller type="pci-root" index="0"> > <model name="i440FX"/> > </controller> > <controller type="pci" index="0"> <!-- Host bridge --> > <address type='pci' domain='0' bus='0' slot='0'/> > </controller> > <controller type="pci" index="1"> <!-- Additional bridge --> > <address type='pci' domain='0' bus='0' slot='1'/> > </controller> > <controller type="pci" index="1"> <!-- Additional bridge --> > <address type='pci' domain='0' bus='0' slot='1'/> > </controller> I think you did a cut-paste here and intended to change something, but didn't - those two bridges are identical. Yep, the slot should be 2 in the second one

...

> <interface type='direct'> > ... > <address type='pci' domain='0' bus='1' slot='3'/> > </controller> > > So if we later allowed for mutiple PCI roots, then we'd have something > like > > <controller type="pci-root" index="0"> > <model name="i440FX"/> > </controller> > <controller type="pci-root" index="1"> > <model name="i440FX"/> > </controller> > <controller type="pci" index="0"> <!-- Host bridge 1 --> > <address type='pci' domain='0' bus='0' slot='0''/> > </controller> > <controller type="pci" index="0"> <!-- Host bridge 2 --> > <address type='pci' domain='1' bus='0' slot='0''/> > </controller>

There is a problem here - within a given controller type, we will now have the possibility of multiple controllers with the same index - the differentiating attribute will be in the <address> subelement, which could create some awkwardness. Maybe instead this should be handled with a different model of pci controller, and we can add a "domain" attribute at the toplevel rather than specifying an <address>?

On real hardware, the platform can specify the _BBN (Base Bus Number = bus) and the _SEG (Segment = domain) of the host bridge. So perhaps you want something like:

<controller type="pci-host-bridge"> <model name="i440FX"/> <address type="pci-host-bridge-addr" domain='1' bus='0'/> </controller>

Yes, we could specify segments, though it's not the same as a domain as linux guests define it (I assume this is what libvirt wants to call a domain): if memory serves a segment does not have to be a root based hierarchy, linux domains are all root based. We are better off not specifying BBN for all buses I think - it's intended for multi-root support for legacy OSes.

"index" is confusing to me.

I'd prefer ID for bus not a number, I'm concerned users will assume it's bus number and get confused by a mismatch.

...

...

...

...

...

> <interface type='direct'> <!-- NIC on host bridge 2 --> > ... > <address type='pci' domain='1' bus='0' slot='3'/> > </controller> > > > NB this means that 'index' values can be reused against the > <controller>, provided they are setup on different pci-roots. > >> (also note that it might happen that the bus number in libvirt's config >> will correspond to the bus numbering that shows up in the guest OS, but >> that will just be a happy coincidence) >> >> Does this make sense? > Yep, I think we're fairly close. What about the other types of pci controllers that are used by PCIe? We should make sure they fit in this model before we settle on it. What do they do ?

(The descriptions of different models below tell what each of these other devices does; in short, they're all just some sort of electronic Lego to help connect PCI and PCIe devices into a tree).

Okay, I'll make yet another attempt at understanding these devices, and suggesting how they can all be described in the XML. I'm thinking that *all* of the express hubs, switch ports, bridges, etc can be described in xml in the manner above, i.e.:

<controller type='pci' index='n'> <model type='xxx'/> </controller>

and that the method for connecting a device to any of them would be by specifying:

<address type='pci' domain='n' bus='n' slot='n' function='n'/>

Any limitations about which devices/controllers can connect to which controllers, and how many devices can connect to any particular controller will be derived from the <model type='xxx'/>. (And, as we've said before, although qemu doesn't assign each of these controllers a numeric bus id, and although we can make no guarantee that the bus id we use for a particular controller is what will be used by the guest BIOS/OS, it's still a convenient notation and works well with other hypervisors as well as qemu. I'll also note that when I run lspci on an X58-based machine I have here, *all* of the relationships between all the devices listed below are described with simple bus:slot.function numbers.)

Here is a list of the pci controller model types and their restrictions (thanks to mst and aw for repeating these over and over to me; I'm sure I still have made mistakes, but at least it's getting closer).

<controller type='pci-root'> ============================

Upstream: nothing Downstream: only a single pci-root-bus (implied) qemu commandline: nothing (it's implied in the q35 machinetype)

Explanation:

Each machine will have a different controller called "pci-root" as outlined above by Daniel. Two types of pci-root will be supported: i440FX and q35. If a pci-root is not spelled out in the config, one will be auto-added (depending on machinetype).

An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and any bridge that has an address of slot='0' on its own bus is, by definition, connected to a pci-root controller - the two are matched by setting "domain" in the address of the pci-bridge to "index" of the pci-root). This bridge can only have PCI devices added.

A q35 pci-root also implies a different kind of pci-bridge device - one that can only have PCIe devices/controllers attached, but is otherwise identical to the pci-bridge added for i440FX. This bus will be called "root-bus" (Note that there are generally followed conventions for what can be connected to which slot on this bus, and we will probably follow those conventions when building a machine, *but* we will not hardcode this convention into libvirt; each q35 machine will be an empty slate)

<controller type='pci'> =======================

This will be used for *all* of the following controller devices supported by qemu:

<model type='pcie-root-bus'/> (implicit/integrated) ----------------------------

Upstream: connect to pci-root controller *only* Downstream: 32 slots, PCIe devices only, no hotplug. qemu commandline: nothing (implicit in the q35-* machinetype)

This controller is the bus described above that connects to a q35's pci-root, and provides places for PCIe devices to connect. Examples are root-ports, dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet devices (do any of those that can be connected to the pcie-root-bus exist yet?).

There is only one of these controllers, and it will *always* be index='0', and will always have the following address:

<address type='pci' domain='0' bus='0' slot='0' function='0'/>

Implicit devices make me nervous, why wouldn't this just be a pcie-root (or pcie-host-bridge)? If we want to support multiple host bridges, there can certainly be more than one, so the index='0' assumption seems to fall apart.

...

<model type='root-port'/> (ioh3420) -------------------------

Upstream: PCIe, connect to pcie-root-bus *only* (?)

yes

...

Downstream: 1 slot, PCIe devices only (?)

yes

...

qemu commandline: -device ioh3420,...

These can only connect to the "pcie-root-bus" of of a q35 (implying that this bus will need to have a different model name than the simple "pci-bridge"

<model type='dmi-to-pci-bridge'/> (i82801b11-bridge)

I'm worried this name is either too specific or too generic. What happens when we add a generic pcie-bridge and want to use that instead of the i82801b11-bridge? The guest really only sees this as a PCIe-to-PCI bridge, it just happens that on q35 this attaches at the DMI port of the MCH.

...

---------------------------------

(btw, what does "dmi" mean?)

http://en.wikipedia.org/wiki/Direct_Media_Interface

...

Upstream: pcie-root-bus *only*

And only to a specific q35 slot (1e.0) for the i82801b11-bridge.

...

Downstream: 32 slots, any PCI device, no hotplug (?)

Yet, but I think this is where we want to implement ACPI based hotplug.

...

qemu commandline: -device i82801b11-bridge,...

<model type='upstream-switch-port'/> (x3130-upstream) ------------------------------------

Upstream: PCIe, connect to pcie-root-bus, root-port, or downstream-switch-port (?)

yes

...

Downstream: 32 slots, connect *only* to downstream-switch-port

I can't verify that there are 32 slots, mst? I've only setup downstream ports within slot 0.

...

qemu-commandline: -device x3130-upstream

This is the upper side of a switch that can multiplex multiple devices onto a single port. It's only useful when one or more downstream switch ports are connected to it.

<model type='downstream-switch-port'/> (xio3130-downstream) --------------------------------------

Upstream: connect *only* to upstream-switch-port Downstream: 1 slot, any PCIe device qemu commandline: -device xio3130-downstream

You can connect one or more of these to an upstream-switch-port in order to effectively plug multiple devices into a single PCIe port.

<model type='pci-bridge'/> (pci-bridge) --------------------------

Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3) another pci-bridge Downstream: any PCI device, 32 slots qemu commandline: -device pci-bridge,...

This differs from dmi-to-pci-bridge in that its upstream connection is PCI rather than PCIe (so it will work on an i440FX system, which has no root PCIe bus) and that hotplug is supported. In general, if a guest will have any PCI devices, one of these controllers should be added, and

===============================================================

Comment: I'm not quite convinced that we really need the separate "pci-root" device. Since 1) every pci-root will *always* have either a pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will only ever be connected to the pci-root, and 3) the pci-bridge that connects to it will need special handling within the pci-bridge case anyway, why not:

1) eliminate the separate pci-root controller type

2) within <controller type='pci'>, a new <model type='pci-root-bus'/> will be added.

3) a pcie-root-bus will automatically be added for q35 machinetypes, and pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")

4) model type='pci-root-bus' will behave like pci-bridge, except that it will be an implicit device (nothing on qemu commandline) and it won't need an <address> element (neither will pcie-root-bus).

I think they should both have a domain + bus address to make it possible to build multi-domain/multi-host bridge systems. They do not use any slots through.

...

5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller.

Or this Wouldn't even be unnecessary if we supported a 'pci-root-addr' address type for the above with the default being domain=0, bus=0? I suppose it doesn't matter whether it's a separate attribute or new address type though. Thanks,

Alex

Also AFAIK there's nothing in the spec that requires bus=0 to be root. The _BBN hack above is used sometimes to give !=0 bus numbers to roots. -- MST

On Tue, Apr 16, 2013 at 12:05:40PM -0400, Laine Stump wrote:

On 04/15/2013 05:58 PM, Michael S. Tsirkin wrote:

...

On Mon, Apr 15, 2013 at 11:27:03AM -0600, Alex Williamson wrote:

...

On Fri, 2013-04-12 at 11:46 -0400, Laine Stump wrote:

...

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote: it's intended for multi-root support for legacy OSes.

...

"index" is confusing to me.

index is being used just because that's been the convention for other controller types - when there are multiple controllers of the same type, each is given an index, and that's used in the "child" devices to indicate which of the parent controllers they connect to.

...

I'd prefer ID for bus not a number, I'm concerned users will assume it's bus number and get confused by a mismatch.

So you would rather that they were something like this?

<controller type='pci' bus='pci.0'> <model type='pci-root'/> </controller> <interface type='blah'> ... <address type='pci' domain='0' bus='pci.0' slot='0' function='0'/> </interface>

The problem is that the use of numeric bus IDs is fairly deeply ingrained in libvirt; every existing libvirt guest config has device addresses specifying "bus='0'" Switching to using an alphanumeric ID rather than a simple number would require extra care to maintain backward compatibility with all those existing configs and previous versions of libvirt that might end up being the recipient of xml generated by a newer libvirt. Because of this, at the very least the pci.0 bus must be referred to as bus='0'; once we've done that, we might as well refer to them *all* numerically (anyway, even if names were allowed, I'm sure everybody would just call them '1', '2', (or at the very most "pci.1", "pci.2") etc. anyway.

We explicitly do *NOT* want to use something like 'pci.1' because that is a QEMU specific naming concept. The use of numeric IDs was a delibrate choice for libvirt addressing and not something we are going to change.

...

...

...

5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller.

Or this Wouldn't even be unnecessary if we supported a 'pci-root-addr' address type for the above with the default being domain=0, bus=0? I suppose it doesn't matter whether it's a separate attribute or new address type though. Thanks,

I think you're mixing up the purpose of the <address> element vs the "index" attribute in the main <controller> element. To clarify, take this example:

<controller type='pci' index='3'> <model type='pci-bridge'/> <address domain='0' bus='1' slot='9' function='0'/> </controller>

This controller is connected to slot 9 of the already-existing bus 1. It provides a bus 3 for other devices to connect to. If we wanted to start up a domain 1, we would do something like this:

<controller type='pci' domain='1' index='0'> <model type='pci-root'/> </controller>

This would give us a PCI bus 0 in domain 1. You could then connect a pci-bridge to it like this:

<controller type='pci' domain='1' index='1'> <model type='pci-bridge'/> <address type='pci' domain='1' bus='0' slot='1' function='0'/> </controller>

The <address> tells us that this new bus connects to slot 1 of PCI bus 0 in domain 1. The <controller domain='1' index='1'> tells us that there is now a new bus other devices can connect to that is at domain='1' bus='1'.

Can we just leave multi-domains out of this discussion for now. IMHO they are just causing more confusion to an already complicated problem. Lets focus on just solving the problem of PCI bridges which is what we actually need right now. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Tue, Apr 16, 2013 at 12:05:40PM -0400, Laine Stump wrote:

On 04/15/2013 05:58 PM, Michael S. Tsirkin wrote:

...

On Mon, Apr 15, 2013 at 11:27:03AM -0600, Alex Williamson wrote:

...

...

On 04/11/2013 07:23 AM, Michael S. Tsirkin wrote:

...

On Thu, Apr 11, 2013 at 07:03:56AM -0400, Laine Stump wrote:

...

On 04/10/2013 05:26 AM, Daniel P. Berrange wrote: > On Tue, Apr 09, 2013 at 04:06:06PM -0400, Laine Stump wrote: >> On 04/09/2013 04:58 AM, Daniel P. Berrange wrote: >>> On Mon, Apr 08, 2013 at 03:32:07PM -0400, Laine Stump wrote: >>> Actually I do wonder if we should reprent a PCI root as two >>> <controller> elements, one representing the actual PCI root >>> device, and the other representing the host bridge that is >>> built-in. >>> >>> Also we should use the actual model names, not 'pci-root' or >>> 'pcie-root' but rather i440FX for "pc" machine type, and whatever >>> the q35 model name is. >>> >>> - One PCI root with built-in PCI bus (ie todays' setup) >>> >>> <controller type="pci-root" index="0"> >>> <model name="i440FX"/> >>> </controller> >>> <controller type="pci" index="0"> <!-- Host bridge --> >>> <address type='pci' domain='0' bus='0' slot='0''/> >> Isn't this saying that the bridge connects to itself? (since bus 0 is >> this bus) >> >> I understand (again, possibly wrongly) that the builtin PCI bus connects >> to the chipset using its own slot 0 (that's why it's reserved), but >> that's its address on itself. How is this bridge associated with the >> pci-root? >> >> Ah, I *think* I see it - the domain attribute of the pci controller is >> matched to the index of the pci-root controller, correct? But there's >> still something strange about the <address> of the pci controller being >> self-referential. > Yes, the index of the pci-root matches the 'domain' of <address> Okay, then the way that libvirt differentiates between a pci bridge that is connected to the root, and one that is connected to a slot of another bridge is 1) the "bus" attribute of the bridge's <address> matches the "index" attribute of the bridge itself, and 2) "slot" is always 0. Correct?

(The corollary of this is that if slot == 0 and bus != index, or bus == index and slot != 0, it is a configuration error).

I'm still unclear on the usefulness of the pci-root controller though - all the necessary information is contained in the pci controller, except for the type of root. But in the case of pcie root, I think you're not allowed to connect a standard bridge to it, only a "dmi-to-pci-bridge" (i82801b11-bridge) Yes you can connect a pci bridge to pcie-root. It's represented as a root complex integrated device. Is this accurate? Per the PCI express spec, any PCI express device needs to have a PCI express capability, which our pci-bridge does not. I think this is one of the main differences for our i82801b11-bridge,

On Fri, 2013-04-12 at 11:46 -0400, Laine Stump wrote: that it exposes itself as a root complex integrated endpoint, so we know it's effectively a PCIe-to-PCI bridge. If it does not have an express link upstream it's not a PCIe-to-PCI bridge, is it?

To my untrained ear it sounds like you're disagreeing with yourself ???

...

...

We'll be asking for trouble if/when we get guest IOMMU support if we are lax about using PCI-to-PCI bridges where we should have PCIe-to-PCI bridges. I recall the spec saying somewhere that integrated endpoints are outside the root complex hierarchy. I think IOMMU will simply not apply to these.

Correct me if I'm wrong - I think libvirt can ignore this bit of debate other than to use its result to determine which devices are allowed to connect to which other devices, right?

Yes.

...

...

There are plenty of examples to the contrary of root complex integrated endpoints without an express capability, but that doesn't make it correct to the spec. Is there something in the spec explicitly forbidding this? I merely find: The PCI Express Capability structure is required for PCI Express device Functions. So if it's not an express device it does not have to have an express capability?

Maybe we should send an example dump to pci sig and ask them...

...

...

ARGHH!! Just when I think I'm starting to understand *something* about these devices...

(later edit: after some coaching on IRC, I *think* I've got a bit better handle on it.)

(But I guess not good enough :-P)

...

...

...

...

...

>>> </controller> >>> <interface type='direct'> >>> ... >>> <address type='pci' domain='0' bus='0' slot='3'/> >>> </controller> >>> >>> - One PCI root with built-in PCI bus and extra PCI bridge >>> >>> <controller type="pci-root" index="0"> >>> <model name="i440FX"/> >>> </controller> >>> <controller type="pci" index="0"> <!-- Host bridge --> >>> <address type='pci' domain='0' bus='0' slot='0'/> >>> </controller> >>> <controller type="pci" index="1"> <!-- Additional bridge --> >>> <address type='pci' domain='0' bus='0' slot='1'/> >>> </controller> >>> <interface type='direct'> >>> ... >>> <address type='pci' domain='0' bus='1' slot='3'/> >>> </controller> >>> >>> - One PCI root with built-in PCI bus, PCI-E bus and and extra PCI bridge >>> (ie possible q35 setup) >> Why would a q35 machine have an i440FX pci-root? > It shouldn't, that's a typo > >>> <controller type="pci-root" index="0"> >>> <model name="i440FX"/> >>> </controller> >>> <controller type="pci" index="0"> <!-- Host bridge --> >>> <address type='pci' domain='0' bus='0' slot='0'/> >>> </controller> >>> <controller type="pci" index="1"> <!-- Additional bridge --> >>> <address type='pci' domain='0' bus='0' slot='1'/> >>> </controller> >>> <controller type="pci" index="1"> <!-- Additional bridge --> >>> <address type='pci' domain='0' bus='0' slot='1'/> >>> </controller> >> I think you did a cut-paste here and intended to change something, but >> didn't - those two bridges are identical. > Yep, the slot should be 2 in the second one > >>> <interface type='direct'> >>> ... >>> <address type='pci' domain='0' bus='1' slot='3'/> >>> </controller> >>> >>> So if we later allowed for mutiple PCI roots, then we'd have something >>> like >>> >>> <controller type="pci-root" index="0"> >>> <model name="i440FX"/> >>> </controller> >>> <controller type="pci-root" index="1"> >>> <model name="i440FX"/> >>> </controller> >>> <controller type="pci" index="0"> <!-- Host bridge 1 --> >>> <address type='pci' domain='0' bus='0' slot='0''/> >>> </controller> >>> <controller type="pci" index="0"> <!-- Host bridge 2 --> >>> <address type='pci' domain='1' bus='0' slot='0''/> >>> </controller>

There is a problem here - within a given controller type, we will now have the possibility of multiple controllers with the same index - the differentiating attribute will be in the <address> subelement, which could create some awkwardness. Maybe instead this should be handled with a different model of pci controller, and we can add a "domain" attribute at the toplevel rather than specifying an <address>?

On real hardware, the platform can specify the _BBN (Base Bus Number = bus) and the _SEG (Segment = domain) of the host bridge. So perhaps you want something like:

<controller type="pci-host-bridge"> <model name="i440FX"/> <address type="pci-host-bridge-addr" domain='1' bus='0'/> </controller>

The <address> element is intended to specify where a device or controller is connected *to*, not what bus/domain it *provides*. I think you're intending for this to provide domain 1 bus 0, so according to existing convention, you would want that information in the <controller> element attributes (e.g. for all other controller types, the generic "index" attribute is used to indicate a bus number when such a thing is appropriate for that type of controller).

Anyway, I've simplified this a bit in my latest iteration - there are no separate "root" and "root bus" controllers, just a "pci-root" (for i440FX) or "pcie-root" (for q35), both of which provide a "pci" bus (I'm using the term loosely here), each with different restrictions about what can be connected.

...

Yes, we could specify segments, though it's not the same as a domain as linux guests define it (I assume this is what libvirt wants to call a domain): if memory serves a segment does not have to be a root based hierarchy, linux domains are all root based.

I'm not exactly sure of the meanings/implications of all those terms, but from the point of view of libvirt, as long as we can represent all possible connections between devices using the domain:bus:slot.function notation, I think it doesn't matter too much.

...

We are better off not specifying BBN for all buses I think -

How would you differentiate between the different buses without some sort of identifier?

...

it's intended for multi-root support for legacy OSes.

...

"index" is confusing to me.

index is being used just because that's been the convention for other controller types - when there are multiple controllers of the same type, each is given an index, and that's used in the "child" devices to indicate which of the parent controllers they connect to.

...

I'd prefer ID for bus not a number, I'm concerned users will assume it's bus number and get confused by a mismatch.

So you would rather that they were something like this?

<controller type='pci' bus='pci.0'> <model type='pci-root'/> </controller> <interface type='blah'> ... <address type='pci' domain='0' bus='pci.0' slot='0' function='0'/> </interface>

The problem is that the use of numeric bus IDs is fairly deeply ingrained in libvirt; every existing libvirt guest config has device addresses specifying "bus='0'" Switching to using an alphanumeric ID rather than a simple number would require extra care to maintain backward compatibility with all those existing configs and previous versions of libvirt that might end up being the recipient of xml generated by a newer libvirt. Because of this, at the very least the pci.0 bus must be referred to as bus='0'; once we've done that, we might as well refer to them *all* numerically (anyway, even if names were allowed, I'm sure everybody would just call them '1', '2', (or at the very most "pci.1", "pci.2") etc. anyway.

...

...

...

...

...

>>> <interface type='direct'> <!-- NIC on host bridge 2 --> >>> ... >>> <address type='pci' domain='1' bus='0' slot='3'/> >>> </controller> >>> >>> >>> NB this means that 'index' values can be reused against the >>> <controller>, provided they are setup on different pci-roots. >>> >>>> (also note that it might happen that the bus number in libvirt's config >>>> will correspond to the bus numbering that shows up in the guest OS, but >>>> that will just be a happy coincidence) >>>> >>>> Does this make sense? >>> Yep, I think we're fairly close. >> What about the other types of pci controllers that are used by PCIe? We >> should make sure they fit in this model before we settle on it. > What do they do ? (The descriptions of different models below tell what each of these other devices does; in short, they're all just some sort of electronic Lego to help connect PCI and PCIe devices into a tree).

Okay, I'll make yet another attempt at understanding these devices, and suggesting how they can all be described in the XML. I'm thinking that *all* of the express hubs, switch ports, bridges, etc can be described in xml in the manner above, i.e.:

<controller type='pci' index='n'> <model type='xxx'/> </controller>

and that the method for connecting a device to any of them would be by specifying:

<address type='pci' domain='n' bus='n' slot='n' function='n'/>

Any limitations about which devices/controllers can connect to which controllers, and how many devices can connect to any particular controller will be derived from the <model type='xxx'/>. (And, as we've said before, although qemu doesn't assign each of these controllers a numeric bus id, and although we can make no guarantee that the bus id we use for a particular controller is what will be used by the guest BIOS/OS, it's still a convenient notation and works well with other hypervisors as well as qemu. I'll also note that when I run lspci on an X58-based machine I have here, *all* of the relationships between all the devices listed below are described with simple bus:slot.function numbers.)

Here is a list of the pci controller model types and their restrictions (thanks to mst and aw for repeating these over and over to me; I'm sure I still have made mistakes, but at least it's getting closer).

<controller type='pci-root'> ============================

Upstream: nothing Downstream: only a single pci-root-bus (implied) qemu commandline: nothing (it's implied in the q35 machinetype)

Explanation:

Each machine will have a different controller called "pci-root" as outlined above by Daniel. Two types of pci-root will be supported: i440FX and q35. If a pci-root is not spelled out in the config, one will be auto-added (depending on machinetype).

An i440FX pci-root has an implicitly added pci-bridge at 0:0:0.0 (and any bridge that has an address of slot='0' on its own bus is, by definition, connected to a pci-root controller - the two are matched by setting "domain" in the address of the pci-bridge to "index" of the pci-root). This bridge can only have PCI devices added.

A q35 pci-root also implies a different kind of pci-bridge device - one that can only have PCIe devices/controllers attached, but is otherwise identical to the pci-bridge added for i440FX. This bus will be called "root-bus" (Note that there are generally followed conventions for what can be connected to which slot on this bus, and we will probably follow those conventions when building a machine, *but* we will not hardcode this convention into libvirt; each q35 machine will be an empty slate)

<controller type='pci'> =======================

This will be used for *all* of the following controller devices supported by qemu:

<model type='pcie-root-bus'/> (implicit/integrated) ----------------------------

Upstream: connect to pci-root controller *only* Downstream: 32 slots, PCIe devices only, no hotplug. qemu commandline: nothing (implicit in the q35-* machinetype)

This controller is the bus described above that connects to a q35's pci-root, and provides places for PCIe devices to connect. Examples are root-ports, dmi-to-pci-bridges sata controllers, integrated sound/usb/ethernet devices (do any of those that can be connected to the pcie-root-bus exist yet?).

There is only one of these controllers, and it will *always* be index='0', and will always have the following address:

<address type='pci' domain='0' bus='0' slot='0' function='0'/> Implicit devices make me nervous, why wouldn't this just be a pcie-root (or pcie-host-bridge)? If we want to support multiple host bridges, there can certainly be more than one, so the index='0' assumption seems to fall apart.

That's when we need to start talking about a "domain" attribute, like this:

<controller type='pci' domain='1' index='0'> <model type='pcie-root-bus'/> </controller>

...

...

...

<model type='root-port'/> (ioh3420) -------------------------

Upstream: PCIe, connect to pcie-root-bus *only* (?) yes

...

Downstream: 1 slot, PCIe devices only (?) yes

...

qemu commandline: -device ioh3420,...

These can only connect to the "pcie-root-bus" of of a q35 (implying that this bus will need to have a different model name than the simple "pci-bridge"

<model type='dmi-to-pci-bridge'/> (i82801b11-bridge) I'm worried this name is either too specific or too generic. What happens when we add a generic pcie-bridge and want to use that instead of the i82801b11-bridge? The guest really only sees this as a PCIe-to-PCI bridge, it just happens that on q35 this attaches at the DMI port of the MCH.

Hehe. Just using the name you (Alex) suggested :-)

My use of the "generic" device *type* names rather than exact hardware model names is based on the idea that any given machinetype will have a set of these "building block" devices available, and as long as you use everything from the same "set" on a given machine, it doesn't really matter which set you use. Is this a valid assumption?

...

...

...

---------------------------------

(btw, what does "dmi" mean?) http://en.wikipedia.org/wiki/Direct_Media_Interface

...

Upstream: pcie-root-bus *only* And only to a specific q35 slot (1e.0) for the i82801b11-bridge.

...

Downstream: 32 slots, any PCI device, no hotplug (?) Yet, but I think this is where we want to implement ACPI based hotplug.

Okay, but for now libvirt can just refrain from auto-addressing any user-created devices to that bus; we'll just make sure that there is always a "pci-bridge" plugged into it, and auto-addressed devices will all be put there.

In the meantime if someone explicitly addresses a device to connect to the i82801b11-bridge, we'll let them do it, but if they try to hot-unplug it they will get an error.

...

...

...

qemu commandline: -device i82801b11-bridge,...

<model type='upstream-switch-port'/> (x3130-upstream) ------------------------------------

Upstream: PCIe, connect to pcie-root-bus, root-port, or downstream-switch-port (?) yes

...

Downstream: 32 slots, connect *only* to downstream-switch-port I can't verify that there are 32 slots, mst? I've only setup downstream ports within slot 0.

According to a discussion with Don Dutile on IRC yesterday, the downstream side of an upstream-switch-port has 32 "slots" with 8 "functions" each, and each of these functions can have a downstream-switch-port connected. That said, he told me that in every case he's seen in the real world, all the downstream-switch-ports were connected to "function 0", effectively limiting it to 32 downstreams/upstream.

...

...

...

qemu-commandline: -device x3130-upstream

This is the upper side of a switch that can multiplex multiple devices onto a single port. It's only useful when one or more downstream switch ports are connected to it.

<model type='downstream-switch-port'/> (xio3130-downstream) --------------------------------------

Upstream: connect *only* to upstream-switch-port Downstream: 1 slot, any PCIe device qemu commandline: -device xio3130-downstream

You can connect one or more of these to an upstream-switch-port in order to effectively plug multiple devices into a single PCIe port.

<model type='pci-bridge'/> (pci-bridge) --------------------------

Upstream: PCI, connect to 1) pci-root, 2) dmi-to-pci-bridge, 3) another pci-bridge Downstream: any PCI device, 32 slots qemu commandline: -device pci-bridge,...

This differs from dmi-to-pci-bridge in that its upstream connection is PCI rather than PCIe (so it will work on an i440FX system, which has no root PCIe bus) and that hotplug is supported. In general, if a guest will have any PCI devices, one of these controllers should be added, and

===============================================================

Comment: I'm not quite convinced that we really need the separate "pci-root" device. Since 1) every pci-root will *always* have either a pcie-root-bus or a pci-bridge connected to it, 2) the pci-root-bus will only ever be connected to the pci-root, and 3) the pci-bridge that connects to it will need special handling within the pci-bridge case anyway, why not:

1) eliminate the separate pci-root controller type

2) within <controller type='pci'>, a new <model type='pci-root-bus'/> will be added.

3) a pcie-root-bus will automatically be added for q35 machinetypes, and pci-root-bus for any machinetype that supports a PCI bus (e.g. "pc-*")

4) model type='pci-root-bus' will behave like pci-bridge, except that it will be an implicit device (nothing on qemu commandline) and it won't need an <address> element (neither will pcie-root-bus). I think they should both have a domain + bus address to make it possible to build multi-domain/multi-host bridge systems. They do not use any slots through.

Yes. I think I agree with that. But we don't have to implement the multiple-domain stuff today (since qemu doesn't support it yet), and when we do, I think we can just add a "domain" attribute to the main element of pci-root and pcie-root controllers.

...

...

...

5) to support multiple domains, we can simply add a "domain" attribute to the toplevel of controller.

Or this Wouldn't even be unnecessary if we supported a 'pci-root-addr' address type for the above with the default being domain=0, bus=0? I suppose it doesn't matter whether it's a separate attribute or new address type though. Thanks,

I think you're mixing up the purpose of the <address> element vs the "index" attribute in the main <controller> element. To clarify, take this example:

<controller type='pci' index='3'> <model type='pci-bridge'/> <address domain='0' bus='1' slot='9' function='0'/> </controller>

This controller is connected to slot 9 of the already-existing bus 1. It provides a bus 3 for other devices to connect to. If we wanted to start up a domain 1, we would do something like this:

<controller type='pci' domain='1' index='0'> <model type='pci-root'/> </controller>

This would give us a PCI bus 0 in domain 1. You could then connect a pci-bridge to it like this:

<controller type='pci' domain='1' index='1'> <model type='pci-bridge'/> <address type='pci' domain='1' bus='0' slot='1' function='0'/> </controller>

The <address> tells us that this new bus connects to slot 1 of PCI bus 0 in domain 1. The <controller domain='1' index='1'> tells us that there is now a new bus other devices can connect to that is at domain='1' bus='1'.

...

Also AFAIK there's nothing in the spec that requires bus=0 to be root. The _BBN hack above is used sometimes to give !=0 bus numbers to roots.

I don't really understand that, but do you think that 1) qemu would ever want/be able to model that, or that 2) anyone would ever have a practical reason for wanting to? It's really cool and all to be able to replicate any possible esoteric hardware configuration in a virtual machine, but it seems like the only practical use of replicating something like that would be for someone wanting to test what their OS does when there's no domain=0 in the hardware...

On Mon, Apr 08, 2013 at 04:26:19PM -0600, Alex Williamson wrote:

On Mon, 2013-04-08 at 12:37 -0400, Laine Stump wrote:

...

On 04/05/2013 03:26 PM, Alex Williamson wrote:

...

...

Sure. I just mean "pci bus 0" (hmm, but actually this does point out a problem with my logic - the same namespace (well, "numbering space") is used for both pcie and pci buses, so on a q35 system, bus=0 is already taken by pcie.0; that means that the first pci bus would need to use a different bus number anyway, so it wouldn't be so easy to switch an existing domain from pc to q35 - every PCI device would need to have its bus number modified. I suppose that's reasonable to expect, though. I would think you'd want to differentiate PCI from PCIe anyway. PCI is a bus and you have 32 slots per bus to fill. PCIe is a point-to-point link and you really only have slot 0 available. Perhaps that puts them in different number spaces already.

Are you saying that it's okay to have a bus=0 for pci and a different bus=0 for pcie?

In bus=<identifier> the identifier needs to be unique, but it's not a bus #, it's just an identifier.

This is mixing up QEMU syntax with libvirt syntax - we're not requiring them to be the same. Libvirt operates solely in terms of bus numbers, and will turn those into QEMU bus identifiers at the time it generates the CLI args. Indeed we explicitly do *not* want to follow QEMU's syntax in the libvirt XML, since this is no good for non-QEMU hypervisors.

...

I was hoping that what is used in libvirt's config could mirror as closely as possible the numbering that you see in the output of lspci on the guest, but it sounds like that numbering is something done at the whim of the guest, with no basis in (standard) reality, is that right?

Correct, the the BIOS determines the initial bus numbers and it can do it however it wants. Most guests won't renumber buses, but they can if they want. It's a lost cause to expect any correlation between the libvirt bus identifier and the actual bus number.

I don't think libvirt needs to require that its XML bus numbers match the guest OS' view of bus numbers. If they do match, great, otherwise so be it.

...

I think we're starting to get closer to the concrete problem that's bothering me. As I understand it (and again - "what I understand" has repeatedly been shown to be incorrect in this thread :-):

* Ihere are multiple different types of devices that provide a bus with 1 or more "slots" that PCI devices (e.g., the virtio-net-pci device, the e1000 network device, etc) can be plugged into.

* In the config for those devices, there is a required (auto-generated if not explicitly provided) <address> element that indicates what controller that device is plugged into e.g.:

<interface type='direct'> ... <address type='pci' domain='0' bus='0' slot='3' function='0'/> ... </interface>

The above is not really how QEMU works. QEMU PCI devices take an addr= parameter that specifies the "slot.function". The bus= option is not numeric. That's the identifier value. So if you create a bus with:

-device i82801b11-bridge,id=dmi-to-pci-bridge,addr=1e.0

Then to put a device on that bus you'd do:

-device e1000,id=e1000-net-0,bus=dmi-to-pci-bridge,addr=0.0

As above, that's no problem. Libvirt is perfectly capable of translating from <address type='pci' domain='0' bus='5' slot='3' function='0'/> to to whatever bus=AAA and addr=BBB args that are appropriate for QEMU.

We don't have a way to generate new domains yet, but I imagine it would require a PCI host bridge device and be a parameter to that. For instance:

- device pci-host-bridge,domain=1,id=domain1-pcie.0

Which would create a new "pci-root". You would then specify a device in the domain using the same bus= notation. An e1000 on bus=pcie.0 is in domain0, but a nic on bus=domain1-pcie.0 is in domain1. This is all just speculation though.

...

* domain is always hardcoded to 0, and in the past bus was also always hardcoded to 0 because until now there has only been a single place where PCI devices could be connected - the builtin pci.0 bus, which is a part of the basic "pc" (and some others) virtual machine and provides 32 slots.

Again, bus= is an identifier and I would guess that it will implicitly specify the domain when we get that far. Libvirt specifying a numerical domain:bus:slot.fn and expecting the device to appear there to the guest is a flawed concept.

No, libvirt won't be using addr=domain:bus:slot.fun. it will use whatever bus=AAA and addr=slot.fun pairing is required for QEMU. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|