6:40 a.m.
On Fri, 2016-03-04 at 17:05 -0500, Laine Stump wrote:
> I'm not sure I fully understand all of the above, but
I'll pitch
> in with my own proposal regardless :)
>
> First, we make sure that
>
> <controller type='pci' index='0'
model='pcie-root'/>
>
> is always added automatically to the domain XML when using the
> mach-virt machine type. Then, if
>
> <controller type='pci' index='1'
model='dmi-to-pci-bridge'/>
> <controller type='pci' index='2'
model='pci-bridge'/>
>
> is present as well we default to virtio-pci, otherwise we use
> the current default of virtio-mmio. This should allow management
> applications, based on knowledge about the guest OS, to easily
> pick between the two address schemes.
>
> Does this sound like a good idea?
... or a variation of that, anyway :-)
What I think: If there are *any* pci controllers *beyond* pcie-root, or
if there are any devices that already have a PCI address, then assign
PCI addresses, else use mmio.
This sounds reasonable.
However, I'm wondering if we shouldn't be even more explicit about
this... From libvirt's point of view we just need to agree on some
sort of "trigger" that causes it to allocate PCI addresses instead
of MMIO addresses, and for that purpose "any PCI controller or any
device with a PCI address" is perfectly fine; looking at that from
the point of view of an user, though? Not sure about it.
What about adding something like
<preferences>
<preference name="defaultAddressType" value="pci"/>
</preferences>
to the domain XML? AFAICT libvirt doesn't have any element that
could be used for this purpose at the moment, but maybe having a
generic way to set domain-wide preferences could be useful in
other situations as well?
To make this more useful, we will want some enhancements:
1) add a "hotpluggable" attribute to <target> for every device (this
will be a bit ugly, because there isn't a unified parser/formatter for
the <target> element :-( )
2) add a "busHint" ("busType", "preferredBus", ??)
attribute too, to
allow specifying pci vs pcie (is it worth adding "mmio" here? Seems it
will be deprecated before long anyway...). It will default to pcie
rather than pci on platforms that support it.
3) change the auto-assign to pay attention to hotpluggable and
preferredBus (damn! I need to think of a better name!)
This way someone can define a new aarch64 domain and just toss a bunch
of pcie-root-ports into it:
<controller type='pci' model='pcie-root-port'/>
<controller type='pci' model='pcie-root-port'/>
<controller type='pci' model='pcie-root-port'/>
<controller type='pci' model='pcie-root-port'/>
...
then add a bunch of devices. The pcie-root-ports will be auto-assigned
to ports on pcie.0, and the devices will be auto-assigned to root-ports.
(alternately, they could toss in a bunch of
pcie-downstream-switch-ports. These would trigger an auto-add of a
pcie-upstream-switch-port, which would trigger an auto-add of a
pcie-root-port, and those would all be connected).
(Ooh! Or instead of hotpluggable and preferredBus in the devices'
<target>, just "preferredConnection" which would be exactly the model
of
pci controller preferred. So you'd do something like this:
<interface type='network'>
<target preferredConnection='pcie-switch-downstream-port'/>
...
</interface>
This would look for an existing pcie-switch-downstream-port. If it
couldn't find one available, it would add one (and all the underlying
controllers necessary).
I'm assuming these examples refer to the case of a guest starting
up and not device hotplug, because neither pcie-root-port nor
pcie-switch-{up,down}stream-port can be added at runtime.
Looks to me like having both 'hotpluggable' and 'preferredBus'
would be more user-friendly, ie. looking at the XML tells you
whether you'll be able to hot-unplug a device without having to
remember what a pcie-switch-downstream-port is.
This still doesn't help when it comes to providing PCI or PCIe
slots suitable for hotplugging: if you want to have three PCIe
devices at startup and still be able to hotplug up to two PCIe
devices later on, my understanding is that you would need to
have something like
<!-- pretend this is repeated five times -->
<controller type='pci' model='pcie-root-port'/>
or
<controller type='pci' model='pcie-switch-upstream-port'/>
<!-- pretend this is repeated five times -->
<controller type='pci' model='pcie-switch-downstream-port'/>
Could this make use of a preferences mechanism such as the one
hinted at above? For example
<preferences>
<preference name='pcieHotplugSlots' value='5'/>
<preference name='pciHotplugSlots' value='0'/>
</preferences>
would generate either of the above PCIe topology, while
<preferences>
<preference name='pcieHotplugSlots' value='0'/>
<preference name='pciHotplugSlots' value='5'/>
</preferences>
would add a dmi-to-pci-bridge and a pci-bridge in order to
present the hotplug-capable PCI slots to the guest.
Cheers.
--
Andrea Bolognani
Software Engineer - Virtualization Team