On Wed, 2016-02-17 at 12:40 -0500, Laine Stump wrote:

On 01/28/2016 04:14 PM, Cole Robinson wrote:

...

  This was discussed here:   https://www.redhat.com/archives/libvir-list/2015-December/msg00217.html   The summary is that apps/users are in a tough spot WRT aarch64 -M virt and PCIe support: qemu has all the plumbing, but most distros don't support it yet. Presently libvirt adds a PCIe controller to the XML for new enough qemu, but this patch drops that behavior. I read back through all of these messages and am more thoroughly  undecided than ever about the proper thing to do.

Same here :)

I guess the end of that thread was that the best thing to do was (as  you're doing in this patch) to temporarily (?) not add a pcie-root  controller to a domain's XML for aarch64 -M virt, and if there is no PCI  controller, then use virtio-mmio for the address of all devices; but if  a pcie-root is manually added, to instead use pci addresses.   My main issue with that is that the XML is then not reflecting the  hardware exactly in cases where the binary is creating a pcie-root  automatically (but we're not using it). The alternative would be to  force specifying the address type of each device though, and since that  is impractical for management apps to deal with, I guess keying off the  presence/absence of a manually-created pcie-root is the less evil of the  options :-P.

This is my concern as well. Not adding automatically the dmi-to-pci-bridge and pci-bridge sounds like a step in the right direction as QEMU will happily work without them, but the PCIe root controller is always added (if my reading of the output of 'info qtree' and poking around the guest is correct) and the domain XML should reflect this fact. Deviating from the actual (virtual) hardware is something that's bound to come back and bite us at some point, I'm afraid, and we should avoid it. Cheers. --  Andrea Bolognani Software Engineer - Virtualization Team

On 01/28/2016 04:14 PM, Cole Robinson wrote:

If a user manually specifies this XML snippet for aarch64 machvirt:

<controller type='pci' index='0' model='pci-root'/>

As you've noted below, this isn't correct. aarch64 machvirt has no implicit pci-root controller (aka "pci.0"). It instead has a pcie-root controller ("pcie.0"). Since a pci[e]-root controller cannot be explicitly added, by definition this couldn't work.

Libvirt will interpret this to mean that the OS supports virtio-pci, and will allocate PCI addresses (instead of virtio-mmio) for virtio devices.

This is a giant hack. Trying to improve it led me into the maze of PCI address code and I gave up for now. Here are the issues:

* I'd prefer that to be model='pcie-root' which matches what qemu-system-aarch64 -M virt actually provides by default... however libvirt isn't happy with a single pcie-root specified by the user, it will error with:

error: unsupported configuration: failed to create PCI bridge on bus 1: too many devices with fixed addresses

That's not the right error, but it's caused by the fact that libvirt wants the pci-bridge device to be plugged into a standard PCI slot, but all the slots of pcie-root are PCIe slots. Since we now know that qemu doesn't mind if any standard PCI device is plugged into a PCIe slot, the decision of how we want to solve this problem depends on whether or not we want the devices in question to be hot-pluggable - the ports of pcie-root do not support hot-plugging devices (at least on Q35), while the ports on pci-bridge do. So if we require that all devices be hot-pluggable, then we have a few choices: 1) create the same PCI controller Frankenstein we currently have for Q35 - a dmi-to-pci-bridge plugged into pcie-root, and a pci-bridge plugged into dmi-to-pci-bridge. This is easiest because it already works, but it does create an extra unnecessary controller. 2) auto-add a pci-bridge in cases when there is a pcie-root but not standard PCI slots. This would take only a slight amount more work. 3) auto-add a pcie-root-port to each port of the pcie-root controller. This would still leave us with PCIe ports, so we would need to teach libvirt that it's okay to plug PCI devices into PCIe ports. If we don't require hot-pluggability, then we can just teach the address-assignment code that PCI devices can plug into non-hotpluggable PCIe ports and we're done. Or we can do a hybrid that's kind of a continuation of the "use PCI if it's available, otherwise mmio" - we could do this: A) If there are any standard PCI slots, then auto-assign to PCI slots (creating new pci-bridge controllers s necessary) B) else if there are any PCIe slots, then auto-assign to hot-pluggable PCIe if available, or straight PCIe if not. C) else use virtio-mmio. ------------------------------------------- Mixed in with all of this discussion is my thinking that we should have some way to specify, in XML, constraints for the address of each device *without specifying the address itself*. Things we need to be able to specify: 1) Is a PCI-only vs. PCIe-only vs. either one (maybe this could be used in the future to constrain to virtio-mmio as well)? 2) Must the device be hot-pluggable? (default would be yes) 3) guest-side NUMA node? (I'm not sure if this needs to be user specifiable - in the case of a vfio-assigned device, I think all we need to to inform the guest which NUMA node the device is on in the host (via putting it on a PXB controller that is configured with that same NUMA node number). For emulated devices - is there any use to putting an *emulated* device on the same controller as a particular vfio-assigned device that is on a specific node? If not, then maybe it will never matter). It would be better if these "address constraints" were in a different part of the XML than the <address> element itself - this would maintain the simplicity of being able to just remove all <address> elements in order to force libvirt to re-assign all device addresses. This isn't something that needs doing immediately, but worth keeping in mind while putting together something that works for aarch64.

Instead this patch uses hacks to make pci-root use the pcie.0 bus for aarch64, since that code path already works.

I think that's a dead-end that we would have to back-track on, so probably not a good solution even temporarily. Here's an attempt at a plan: 1) change the PCI address assignment code so that for aarch64/virt it prefers PCIe addresses, but still requires hot-pluggable (currently it almost always prefers PCI, and requires hot-pluggable). (alternate - if aarch64 doesn't support pcie-root-port or pcie-switch-*-port, then don't require hot-pluggable either). 2) put something on the front of that that checks for existence of pcie-root, and if it's not found, uses virtio-mmio instead (is there something already that auto-adds the virtio-mmio address? I haven't looked and am too lazy to do so now). At this point, as long as you manually add a bunch of pcie-root-port controllers along with the manual pcie-root, everything should just work. Then we would go to step 3: 3) enhance the auto-assign code so that, in addition to auto-adding a pci-bridge when needed, it would auto-add either a single pcie-root-port or a pcie-switch-upstream-port and 32 pcie-switch-downstream-ports anytime a hotpluggable PCIe port was needed and couldn't be found. (the latter assumes that aarch64 supports those controllers). Does that make any sense? I could try to code some of this up if you could test it (or help me get setup to test it myself).

On 02/26/2016 10:13 AM, Andrea Bolognani wrote:

On Wed, 2016-02-17 at 15:03 -0500, Laine Stump wrote:

...

On 01/28/2016 04:14 PM, Cole Robinson wrote:

...

If a user manually specifies this XML snippet for aarch64 machvirt:

<controller type='pci' index='0' model='pci-root'/>

As you've noted below, this isn't correct. aarch64 machvirt has no implicit pci-root controller (aka "pci.0"). It instead has a pcie-root controller ("pcie.0"). Since a pci[e]-root controller cannot be explicitly added, by definition this couldn't work.

...

Libvirt will interpret this to mean that the OS supports virtio-pci, and will allocate PCI addresses (instead of virtio-mmio) for virtio devices.

This is a giant hack. Trying to improve it led me into the maze of PCI address code and I gave up for now. Here are the issues:

* I'd prefer that to be model='pcie-root' which matches what qemu-system-aarch64 -M virt actually provides by default... however libvirt isn't happy with a single pcie-root specified by the user, it will error with:

error: unsupported configuration: failed to create PCI bridge on bus 1: too many devices with fixed addresses

That's not the right error, but it's caused by the fact that libvirt wants the pci-bridge device to be plugged into a standard PCI slot, but all the slots of pcie-root are PCIe slots. Since we now know that qemu doesn't mind if any standard PCI device is plugged into a PCIe slot, Should we rely on this behavior? Isn't this something that might change in the future? Or at least be quite puzzling for users?

Just thinking out loud :)

It was my identical thinking that led to libvirt being initially very strict about plugging PCI into PCI and PCIe into PCIe. I've since received reasonable assurances that qemu will continue to be permissive about plugging PCI things into PCIe, so I allow it, but still default to "purity".

...

the decision of how we want to solve this problem depends on whether or not we want the devices in question to be hot-pluggable - the ports of pcie-root do not support hot-plugging devices (at least on Q35), while the ports on pci-bridge do. So if we require that all devices be hot-pluggable, then we have a few choices:

1) create the same PCI controller Frankenstein we currently have for Q35 - a dmi-to-pci-bridge plugged into pcie-root, and a pci-bridge plugged into dmi-to-pci-bridge. This is easiest because it already works, but it does create an extra unnecessary controller. This is the current situation, right?

qemu-kvm in current aarch64 RHEL doesn't have the i82801b11-bridge device compiled in, by the way. However, since qemu-system-aarch64 in Fedora 23 *does* have it, I assume enabling it would simply be a matter of flipping a build configuration bit.

...

2) auto-add a pci-bridge in cases when there is a pcie-root but not standard PCI slots. This would take only a slight amount more work.

3) auto-add a pcie-root-port to each port of the pcie-root controller. This would still leave us with PCIe ports, so we would need to teach libvirt that it's okay to plug PCI devices into PCIe ports. As mentioned above, I'm not sure this is a good idea. Maybe I'm just afraid of my own shadow though :)

...

If we don't require hot-pluggability, then we can just teach the address-assignment code that PCI devices can plug into non-hotpluggable PCIe ports and we're done.

Or we can do a hybrid that's kind of a continuation of the "use PCI if it's available, otherwise mmio" - we could do this:

A) If there are any standard PCI slots, then auto-assign to PCI slots (creating new pci-bridge controllers s necessary)

B) else if there are any PCIe slots, then auto-assign to hot-pluggable PCIe if available, or straight PCIe if not.

C) else use virtio-mmio.

-------------------------------------------

Mixed in with all of this discussion is my thinking that we should have some way to specify, in XML, constraints for the address of each device *without specifying the address itself*. Things we need to be able to specify:

1) Is a PCI-only vs. PCIe-only vs. either one (maybe this could be used in the future to constrain to virtio-mmio as well)?

2) Must the device be hot-pluggable? (default would be yes)

3) guest-side NUMA node? (I'm not sure if this needs to be user specifiable - in the case of a vfio-assigned device, I think all we need to to inform the guest which NUMA node the device is on in the host (via putting it on a PXB controller that is configured with that same NUMA node number). For emulated devices - is there any use to putting an *emulated* device on the same controller as a particular vfio-assigned device that is on a specific node? If not, then maybe it will never matter).

It would be better if these "address constraints" were in a different part of the XML than the <address> element itself - this would maintain the simplicity of being able to just remove all <address> elements in order to force libvirt to re-assign all device addresses.

This isn't something that needs doing immediately, but worth keeping in mind while putting together something that works for aarch64.

...

Instead this patch uses hacks to make pci-root use the pcie.0 bus for aarch64, since that code path already works.

I think that's a dead-end that we would have to back-track on, so probably not a good solution even temporarily.

Here's an attempt at a plan:

1) change the PCI address assignment code so that for aarch64/virt it prefers PCIe addresses, but still requires hot-pluggable (currently it almost always prefers PCI, and requires hot-pluggable). (alternate - if aarch64 doesn't support pcie-root-port or pcie-switch-*-port, then don't require hot-pluggable either).

2) put something on the front of that that checks for existence of pcie-root, and if it's not found, uses virtio-mmio instead (is there something already that auto-adds the virtio-mmio address? I haven't looked and am too lazy to do so now).

At this point, as long as you manually add a bunch of pcie-root-port controllers along with the manual pcie-root, everything should just work. Then we would go to step 3:

3) enhance the auto-assign code so that, in addition to auto-adding a pci-bridge when needed, it would auto-add either a single pcie-root-port or a pcie-switch-upstream-port and 32 pcie-switch-downstream-ports anytime a hotpluggable PCIe port was needed and couldn't be found. (the latter assumes that aarch64 supports those controllers).

Does that make any sense? I could try to code some of this up if you could test it (or help me get setup to test it myself). I'm not sure I fully understand all of the above, but I'll pitch in with my own proposal regardless :)

First, we make sure that

<controller type='pci' index='0' model='pcie-root'/>

is always added automatically to the domain XML when using the mach-virt machine type. Then, if

<controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/>

is present as well we default to virtio-pci, otherwise we use the current default of virtio-mmio. This should allow management applications, based on knowledge about the guest OS, to easily pick between the two address schemes.

Does this sound like a good idea?

... or a variation of that, anyway :-) What I think: If there are *any* pci controllers *beyond* pcie-root, or if there are any devices that already have a PCI address, then assign PCI addresses, else use mmio. To make this more useful, we will want some enhancements: 1) add a "hotpluggable" attribute to <target> for every device (this will be a bit ugly, because there isn't a unified parser/formatter for the <target> element :-( ) 2) add a "busHint" ("busType", "preferredBus", ??) attribute too, to allow specifying pci vs pcie (is it worth adding "mmio" here? Seems it will be deprecated before long anyway...). It will default to pcie rather than pci on platforms that support it. 3) change the auto-assign to pay attention to hotpluggable and preferredBus (damn! I need to think of a better name!) This way someone can define a new aarch64 domain and just toss a bunch of pcie-root-ports into it: <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> <controller type='pci' model='pcie-root-port'/> ... then add a bunch of devices. The pcie-root-ports will be auto-assigned to ports on pcie.0, and the devices will be auto-assigned to root-ports. (alternately, they could toss in a bunch of pcie-downstream-switch-ports. These would trigger an auto-add of a pcie-upstream-switch-port, which would trigger an auto-add of a pcie-root-port, and those would all be connected). (Ooh! Or instead of hotpluggable and preferredBus in the devices' <target>, just "preferredConnection" which would be exactly the model of pci controller preferred. So you'd do something like this: <interface type='network'> <target preferredConnection='pcie-switch-downstream-port'/> ... </interface> This would look for an existing pcie-switch-downstream-port. If it couldn't find one available, it would add one (and all the underlying controllers necessary). I'm just rambling now, so I'll stop. But I think a good first step is the simple thing in the "What I think" paragraph.

On Wed, Mar 09, 2016 at 01:40:36PM +0100, Andrea Bolognani wrote:

On Fri, 2016-03-04 at 17:05 -0500, Laine Stump wrote:

...

...

I'm not sure I fully understand all of the above, but I'll pitch in with my own proposal regardless :)   First, we make sure that       <controller type='pci' index='0' model='pcie-root'/>   is always added automatically to the domain XML when using the mach-virt machine type. Then, if       <controller type='pci' index='1' model='dmi-to-pci-bridge'/>     <controller type='pci' index='2' model='pci-bridge'/>   is present as well we default to virtio-pci, otherwise we use the current default of virtio-mmio. This should allow management applications, based on knowledge about the guest OS, to easily pick between the two address schemes.   Does this sound like a good idea? ... or a variation of that, anyway :-)   What I think: If there are *any* pci controllers *beyond* pcie-root, or  if there are any devices that already have a PCI address, then assign  PCI addresses, else use mmio.

This sounds reasonable.

However, I'm wondering if we shouldn't be even more explicit about this... From libvirt's point of view we just need to agree on some sort of "trigger" that causes it to allocate PCI addresses instead of MMIO addresses, and for that purpose "any PCI controller or any device with a PCI address" is perfectly fine; looking at that from the point of view of an user, though? Not sure about it.

What about adding something like

  <preferences>     <preference name="defaultAddressType" value="pci"/>   </preferences>

to the domain XML? AFAICT libvirt doesn't have any element that could be used for this purpose at the moment, but maybe having a generic way to set domain-wide preferences could be useful in other situations as well?

From the POV of being able to tell libvirt to assign PCI instead of MMIO addresses for the virt machine, I think it suffices to have

[snip] Looking at this mail and laine's before I really get the impression we are over-thinking this all. The automatic address assignment by libvirt was written such that it matched what QEMU would have done, so that we could introduce the concept of device addressing without breaking existing apps which didn't supply addresses. The automatic addressing logic certainly wasn't ever intended to be able to implement arbitrary policies. As a general rule libvirt has always aimed to stay out of the policy business, and focus on providing the mechanism only. So when we start talking about adding <preferences> to the XML this is a sure sign that we've gone too far into trying to implement policy in libvirt. libvirt accept a simple <address type="pci"/> element (ie with no bus/slot/func filled in). If applictions care about assigning devices to specify PCI buses, ie to distinguish between PCI & PCIe, or to pick a different bus when there is one bus per NUMA node, then really it is time for the application to start specifying the full <address> element with all details, and not try to invent ever more complex policies inside libvirt which will never deal with all application use cases. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Wed, Mar 09, 2016 at 02:09:22PM -0500, Laine Stump wrote:

On 03/09/2016 09:54 AM, Daniel P. Berrange wrote:

...

...

From the POV of being able to tell libvirt to assign PCI instead of MMIO addresses for the virt machine, I think it suffices to have libvirt accept a simple <address type="pci"/> element (ie with no bus/slot/func filled in).

That's just a more limited case of the same type of feature creep though - you're specifying a preference for what type of address you want, just in a different place. (The odd thing is that we're adding it only to remedy what is apparently a temporary problem, and even then it's a problem that wouldn't exist if we just said simply this: "If virtio-mmio is specified, then use virtio-mmio; If no address is specified, use pci". This puts the burden on people insisting on the old / soon-to-be-deprecated (?) virtio-mmio address type to add a little something to the XML (and a very simple little something at that!). For a short time, this could cause problems for people who create new domains using qemu binaries that don't have a pci bus on aarch64/virt yet, but that will be easily solvable (by adding "<address type='virtio-mmio'/>", which is nearly as simple as typing "<address type='pci'/>").

So, I've just seen that QEMU has decided that as of QEMU 2.6, the virt machine type will start to be versioned. This is quite convenient I think as it gives us a nice thing to hook on. ie we see a non-versioned machine type of 'virt' then we use virtio-mmio addressing, however, if we see a versioned virt-X.Y.Z machine type, then we can assume pci by default. Since the long term plan for AArch64 is to use PCI for everything, this gives us nice default behaviour from this point onwards, while not breaking compatibility for existing early adopters. Of course people with "legacy" mmio-only guests will stll have a little pain to run then on new QEMU, but honestly I think that's worth it since it will avoid us long term pain in the world where aarch64 uses pci for everything Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On Thu, Mar 10, 2016 at 02:34:02PM +0100, Andrea Bolognani wrote:

On Thu, 2016-03-10 at 09:56 +0000, Daniel P. Berrange wrote:

...

So, I've just seen that QEMU has decided that as of QEMU 2.6, the virt machine type will start to be versioned.  This is quite convenient I think as it gives us a nice thing to hook on. ie we see a non-versioned machine type of 'virt' then we use virtio-mmio addressing, however, if we see a versioned virt-X.Y.Z machine type, then we can assume pci by default.

Since the long term plan for AArch64 is to use PCI for everything, this gives us nice default behaviour from this point onwards, while not breaking compatibility for existing early adopters.

Of course people with "legacy" mmio-only guests will stll have a little pain to run then on new QEMU, but honestly I think that's worth it since it will avoid us long term pain in the world where aarch64 uses pci for everything

I think it's way too early to flip the switch and default to PCI addresses: my understanding is that guest OS support is expected to be spotty at best for at least a couple more years.

I don't buy that. For a start the only virtio-mmio and/or pci devices we are giving to guests are virtio devices (blk, net, balloon, etc). The only current guests supporting these devices are Linux, and Linux has support for PCI on aarch64. So any Linux distro that ships today is going to support PCI. Other guests (*BSD) which may choose to support aarch64 in future would likely go straight to PCI and not bother with virtio-mmio. Likewise if we did ever get a Windows guest on aarch64 with win-virtio drivers I would expec them to be PCI based. So AFAICT, the only distros which are going to be using virtio-mmio are the Linux aarch64 early-access/tech-preview releases. I don't really see them as an important target. Even if we switch to PCI by default, I'd still expect virt-manager to be capable of being told to override the default and assign mmio addresses instad. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|

On 03/09/2016 02:09 PM, Laine Stump wrote:

On 03/09/2016 09:54 AM, Daniel P. Berrange wrote:

...

On Wed, Mar 09, 2016 at 01:40:36PM +0100, Andrea Bolognani wrote:

...

On Fri, 2016-03-04 at 17:05 -0500, Laine Stump wrote:

...

...

I'm not sure I fully understand all of the above, but I'll pitch in with my own proposal regardless :) First, we make sure that <controller type='pci' index='0' model='pcie-root'/> is always added automatically to the domain XML when using the mach-virt machine type. Then, if <controller type='pci' index='1' model='dmi-to-pci-bridge'/> <controller type='pci' index='2' model='pci-bridge'/> is present as well we default to virtio-pci, otherwise we use the current default of virtio-mmio. This should allow management applications, based on knowledge about the guest OS, to easily pick between the two address schemes. Does this sound like a good idea? ... or a variation of that, anyway :-) What I think: If there are *any* pci controllers *beyond* pcie-root, or if there are any devices that already have a PCI address, then assign PCI addresses, else use mmio. This sounds reasonable.

However, I'm wondering if we shouldn't be even more explicit about this... From libvirt's point of view we just need to agree on some sort of "trigger" that causes it to allocate PCI addresses instead of MMIO addresses, and for that purpose "any PCI controller or any device with a PCI address" is perfectly fine; looking at that from the point of view of an user, though? Not sure about it.

What about adding something like

<preferences> <preference name="defaultAddressType" value="pci"/> </preferences>

to the domain XML? AFAICT libvirt doesn't have any element that could be used for this purpose at the moment, but maybe having a generic way to set domain-wide preferences could be useful in other situations as well? [snip]

Looking at this mail and laine's before I really get the impression we are over-thinking this all. The automatic address assignment by libvirt was written such that it matched what QEMU would have done, so that we could introduce the concept of device addressing without breaking existing apps which didn't supply addresses. The automatic addressing logic certainly wasn't ever intended to be able to implement arbitrary policies.

As a general rule libvirt has always aimed to stay out of the policy business, and focus on providing the mechanism only. So when we start talking about adding <preferences> to the XML this is a sure sign that we've gone too far into trying to implement policy in libvirt.

...

From the POV of being able to tell libvirt to assign PCI instead of MMIO addresses for the virt machine, I think it suffices to have libvirt accept a simple <address type="pci"/> element (ie with no bus/slot/func filled in).

That's just a more limited case of the same type of feature creep though - you're specifying a preference for what type of address you want, just in a different place. (The odd thing is that we're adding it only to remedy what is apparently a temporary problem, and even then it's a problem that wouldn't exist if we just said simply this: "If virtio-mmio is specified, then use virtio-mmio; If no address is specified, use pci". This puts the burden on people insisting on the old / soon-to-be-deprecated (?) virtio-mmio address type to add a little something to the XML (and a very simple little something at that!). For a short time, this could cause problems for people who create new domains using qemu binaries that don't have a pci bus on aarch64/virt yet, but that will be easily solvable (by adding "<address type='virtio-mmio'/>", which is nearly as simple as typing "<address type='pci'/>").

A downside of setting your preference with <address type="pci"/> vs. having the preference in a separate element is that you can no longer cause a "re-addressing" of all the devices by simply removing all the <address> elements (and it's really that that got me thinking about adding hints/preferences in the <target> element). I don't know how important that is; I suppose when the producer of the XML is some other software it's a non-issue, when the producer is a human using virsh edit it would make life much simpler once in awhile (and the suggestion in the previous paragraph would eliminate that problem anyway).

So is there a specific reason why we need to keep virtio-mmio as the default, and require explicitly saying "address type='pci'" in order to get a PCI address?

I explained this in my reply to Dan just now, but basically all currently released distros don't work with virtio-pci. However long term I've suggested from the start that we switch to virtio-pci by default, keying off a new enough -M virt version, as an arbitrary marker in time when hopefully most common distros work with virtio-pci

...

If applictions care about assigning devices to specify PCI buses, ie to distinguish between PCI & PCIe, or to pick a different bus when there is one bus per NUMA node, then really it is time for the application to start specifying the full <address> element with all details, and not try to invent ever more complex policies inside libvirt which will never deal with all application use cases.

First, I agree with your vigilance against unnecessary feature creep. Both because keeping things simple makes it easier to maintain and use, and also because it's very difficult (or even impossible) to change something once it's gone in, in the case that you have second thoughts.

But, expanding beyond just the aarch64/virt mmio vs. pci problem (which I had already done in this thread, and which is what got us into this "feature creep" discussion in the first place :-)...

I remember when we were first adding support for Q35 that we said it should be as easy to create a domain with Q35 machinetype as it is for 440fx, i.e. you should be able to do it without manually specifying any PCI addresses (that's why we have the strange default bus hierarchy, with a dmi-to-pci-bridge and a pci-bridge, and all devices going onto the pci-bridge). But of course 440fx is very simplistic - all slots are standard PCI and all are hotpluggable. On Q35 there could be room for choices though, in particular PCI vs PCIe and hotpluggable vs. not. (and yeah, also what NUMA node the bus is on; I agree that one is "getting out there"). So since there are choices, libvirt has by definition begun implementing policy when it auto-assigns *any* PCI address (current policy is "require all auto-assigned device addresses to be hotpluggable standard PCI). And if libvirt doesn't provide an easy way to choose, it will have implemented a defacto mandatory policy (unless you force full specification of PCI addresses, which goes against the "make it easy" goal).

And the current policy is looking (for Q35 and aarch64/virt at least) less and less like the correct one - in the years since it was put in, 1) we've learned that qemu doesn't care, and will not ever care, that a PCI device has been plugged into a PCIe slot, 2) we now have support for several PCIe controllers we didn't previously have, 3) people have started complaining that their devices are in PCI slots rather than PCIe, 4) even though it was stated at the time I wrote the code to auto-add a pci-bridge to every Q35 domain that pci-bridge's failure to support hotplug was a temporary bug, I just learned yesterday that it apparently still doesn't work, and 5) some platforms (e.g. our favorite aarch64/virt) are emulating a hardware platform that has *never* had standard PCI slots, only PCIe.

Beyond that, there is no place that provides a simple encoding of which type of controller provides which type of slot, and what is allowed to plug into what. If you require the management application/human to manually specify all the PCI addresses as soon as they have a need for one of these basic characteristics, then not only has it become cumbersome to define a domain (because the management app has to maintain a data structure to keep track of which PCI addresses are in use and which aren't), but it means that the management application also needs to know all sorts of rules about which PCI controllers are actually pcie vs. pci, and which accept hotplug devices vs. which don't, as well as things like the min/max slot number for each controller, and which ones can plug into where, e.g. a pcie-root-port can only plug into pcie-root or pcie-expander-bus, and a pcie-switch-downstream-port can only plug into a pcie-switch-upstream-port, etc. Requiring a management app to get all of that right just so that they can pick between a hotpluggable and non-hotpluggable slot seems like an overly large burden (and prone to error).

In the end, if libvirt makes it simple for the management app to specify what kind of slot it wants, rather than requiring it to specify the exact slot, then libvirt isn't making any policy decisions, it's just making it easier for the management app to implement its own policy decisions, without requiring the management app to know all the details about which controller implements what kind of connection etc, and that does seem to fit within libvirt's purpose.

I guess the main thing would be to understand usecases... outside of aarch64 is there a real reason for q35 that apps might need one address policy over another? If it's something super obscure, like only for testing or dev, then I think asking people to do it manually is fine. At least for the arm case I think we can side step the question by adding the <address type='pci'/> allocation request, and flipping the default from virtio-mmio to virtio-pci at some future point - Cole