[PATCH] qemu: allow for 15 subordinate pci controllers under each pcie-expander-bus
From: Laine Stump <laine@redhat.com> A guest was created with PCIe bus and 2 PCIe devices attached to two separate pcie-root-ports that were themselves attached to a pcie-expander-bus. This guest was unable to boot from the 2nd device on the pcie-expander-bus. It turns out this failure was because not enough "bus numbers" were reserved for the pcie-expander-bus in the configuration, and this led to a failure of the system firmware to initialize those devices properly. Background: Each pci[e]-expander-bus has a busNr attribute that can be optionally specified in the <target> subelement; it defines the bus number of the expander bus itself. Any pci controller attached to that expander bus is then assigned a bus number between busNr+1 and the busNr of the next higher expander bus (or 255, if there are no expander buses with a higher busNr. The pci-root (or pcie-root) bus is bus number 0, and any pci controller attached to that bus (or its subordinate buses *except for pci[e]-expander-buses*) will be automatically numbered starting at 1, and continuing up until the lowest numbered expander bus. When libvirt originally added support for the pci-expander-bus, each pci-expander-bus' busNr was initialized by default to be the busNr of the current lowest expander bus - 2 (reserving one for the pci-expander-bus itself, and one for the implicit pci-bridge that is always added (by qemu) along with the pci-expander-bus). When pcie-expander-bus support was added, this logic was left untouched, but that is incorrect - a hotplug-supporting PCIe topology requires that each device have its own pcie-root-port, and so adding multiple hotpluggable devices to a pcie-expander-bus uses up one bus number for each device. So, the solution to the original problem is to just lower the busNr of each pcie-expander-bus by 16 (rather than 2) from the previous. This allows connection of up to 15 pcie-root-port controllers (and thus 15 hotpluggable endpoint devices) to each pcie-expander bus with the default setting of busNr. Resolves: https://redhat.atlassian.net/browse/RHEL-144343 Signed-off-by: Laine Stump <laine@redhat.com> -- Note that I could have chosen 32 or 8 or any other number, and no matter which I chose it would be likely to not work for *somebody*. Anyway, if this scheme ever doesn't work out, manual intervention is always possible, either by manually setting busNr, or by putting multiple endpoint devices on different functions of the same pcie-root-port (which is completely unproblematic if the device will never be unplugged at runtime). Signed-off-by: Laine Stump <laine@redhat.com> --- src/qemu/qemu_domain_address.c | 85 ++++++++++++------- ...e-expander-bus-aarch64.aarch64-latest.args | 2 +- ...ie-expander-bus-aarch64.aarch64-latest.xml | 2 +- 3 files changed, 57 insertions(+), 32 deletions(-) diff --git a/src/qemu/qemu_domain_address.c b/src/qemu/qemu_domain_address.c index a8292a9782..81cf34fe86 100644 --- a/src/qemu/qemu_domain_address.c +++ b/src/qemu/qemu_domain_address.c @@ -2567,43 +2567,63 @@ qemuDomainAddressFindNewTargetIndex(virDomainDef *def) static int -qemuDomainAddressFindNewBusNr(virDomainDef *def) +qemuDomainAddressFindNewBusNr(virDomainDef *def, + virDomainControllerModelPCI model) { /* Try to find a nice default for busNr for a new pci-expander-bus. * This is a bit tricky, since you need to satisfy the following: * - * 1) There need to be enough unused bus numbers between busNr of this - * bus and busNr of the next highest bus for the guest to assign a - * unique bus number to each PCI bus that is a child of this - * bus. Each PCI controller. On top of this, the pxb device (which - * implements the pci-expander-bus) includes a pci-bridge within - * it, and that bridge also uses one bus number (so each pxb device - * requires at least 2 bus numbers). + * 1) There need to be enough unused bus numbers between busNr of + * this bus and busNr of the next highest bus for the guest to + * assign a unique bus number to each PCI bus that is a child + * of this bus. * * 2) There need to be enough bus numbers *below* this for all the - * child controllers of the pci-expander-bus with the next lower + * child controllers of the pci[e]-expander-bus with the next lower * busNr (or the pci-root bus if there are no lower - * pci-expander-buses). + * pci[e]-expander-buses). * * 3) If at all possible, we want to avoid needing to change the busNr * of a bus in the future, as that changes the guest's device ABI, * which could potentially lead to issues with a guest OS that is * picky about such things. * - * Due to the impossibility of predicting what might be added to the - * config in the future, we can't make a foolproof choice, but since - * a pci-expander-bus (pxb) has slots for 32 devices, and the only - * practical use for it is to assign real devices on a particular - * NUMA node in the host, it's reasonably safe to assume it should - * never need any additional child buses (probably only a few of the - * 32 will ever be used). So for pci-expander-bus we find the lowest - * existing busNr, and set this one to the current lowest - 2 (one - * for the pxb, one for the integrated pci-bridge), thus leaving the - * maximum possible bus numbers available for other buses plugged - * into pci-root (i.e. pci-bridges and other - * pci-expander-buses). Anyone who needs more than 32 devices - * descended from one pci-expander-bus should set the busNr manually - * in the config. + * Due to the impossibility of predicting what might be added to + * the config in the future, we can't make a foolproof choice, + * and the parameters are different for a pci-expander-bus and a + * pcie-expander-bus. + * + * pci-expander-bus: the pxb device (which implements the + * pci-expander-bus) includes a pci-bridge within it, and that + * bridge uuses one bus number, so each pxb device requires at + * least 2 bus numbers. That bridge has slots for 32 devices, and + * its only practical use is to assign real physical devices on a + * particular NUMA node in the host. It's reasonably safe to + * assume no guest should never have more than 32 physical + * devices on a single NUMA node that need to be assigned to a + * guest (probably only a few of the 32 will ever be used), and + * so we shouldn't need more than 2 bus numbers for each + * pci-expander-bus. So we find the lowest existing busNr, and + * set busNr of the new pci-expander-bus to that value - + * 2. Anyone who needs more than 32 devices descended from one + * pci-expander-bus should set the busNr manually in the config. + * + * pcie-expander-bus: this is (as you'd expect) a more + * complicated story - if you want to support hotplugging on a + * pcie based machinetype, then each endpoint device needs its + * own pcie-root-port, and each root port uses up one bus number + * in the range available to its parent pcie-expander-bus. Taking + * a very generous example of a guest that has 8 GPUs, 8 NICs, + * and 8 NVME drives split between two NUMA nodes, giving 16 bus + * numbers to each pcie-expander-bus gives plenty of bus number + * space for all the controllers needed for each device in this + * extreme config, while still leaving plenty of overhead for 1) + * lots of devices on NUMA node 0 and 2) more than 2 NUMA nodes + * in the unlikely case that someone was forced to create a guest + * with more than 2 NUMA nodes. Again, if someone has a config + * more extreme than this, they should just set busNr manually + * (after all, the pci addresses of all these devices are already + * all being set manually, so that shouldn't be a huge burden). * * There is room for more error checking here - in particular we * can/should determine the ultimate parent (root-bus) of each PCI @@ -2614,6 +2634,10 @@ qemuDomainAddressFindNewBusNr(virDomainDef *def) size_t i; int lowestBusNr = 256; + int maxChildControllers = 1; /* default for pci-expander-bus */ + + if (model == VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS) + maxChildControllers = 15; for (i = 0; i < def->ncontrollers; i++) { virDomainControllerDef *cont = def->controllers[i]; @@ -2626,14 +2650,15 @@ qemuDomainAddressFindNewBusNr(virDomainDef *def) } } - /* If we already have a busNR = 1, then we can't auto-assign (0 is - * the pci[e]-root, and the others may have been assigned - * purposefully). + /* If we already have a controller with busNR <= + * maxChildControllers+1, then we can't auto-assign busNr to this + * new expander bus (0 is the pci[e]-root, and the others may have + * been assigned purposefully). */ - if (lowestBusNr <= 2) + if (lowestBusNr <= (maxChildControllers + 1)) return -1; - return lowestBusNr - 2; + return lowestBusNr - (maxChildControllers + 1); } @@ -2877,7 +2902,7 @@ qemuDomainAssignPCIAddresses(virDomainDef *def, case VIR_DOMAIN_CONTROLLER_MODEL_PCI_EXPANDER_BUS: case VIR_DOMAIN_CONTROLLER_MODEL_PCIE_EXPANDER_BUS: if (options->busNr == -1) - options->busNr = qemuDomainAddressFindNewBusNr(def); + options->busNr = qemuDomainAddressFindNewBusNr(def, cont->model); if (options->busNr == -1) { virReportError(VIR_ERR_CONFIG_UNSUPPORTED, _("No free busNr lower than current lowest busNr is available to auto-assign to bus %1$d. Must be manually assigned"), diff --git a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args index e76510dbbb..52011865a0 100644 --- a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args +++ b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.args @@ -25,7 +25,7 @@ XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain--1-pcie-expander-bus-te/.config \ -rtc base=utc \ -no-shutdown \ -boot strict=on \ --device '{"driver":"pxb-pcie","bus_nr":254,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \ +-device '{"driver":"pxb-pcie","bus_nr":240,"id":"pci.1","bus":"pcie.0","addr":"0x1"}' \ -audiodev '{"id":"audio1","driver":"none"}' \ -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \ -msg timestamp=on diff --git a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml index ce88da574f..0d0cac7653 100644 --- a/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml +++ b/tests/qemuxmlconfdata/pcie-expander-bus-aarch64.aarch64-latest.xml @@ -20,7 +20,7 @@ <controller type='pci' index='0' model='pcie-root'/> <controller type='pci' index='1' model='pcie-expander-bus'> <model name='pxb-pcie'/> - <target busNr='254'/> + <target busNr='240'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </controller> <audio id='1' type='none'/> -- 2.55.0
On 05/10/2026 05:18, Laine Stump via Devel wrote:
* 3) If at all possible, we want to avoid needing to change the busNr * of a bus in the future, as that changes the guest's device ABI, * which could potentially lead to issues with a guest OS that is * picky about such things.
This change violates this constraint. As a non-backward compatible change to the guest ABI I don't think you can change the default bus allocation method. You may find it more useful for your management plane to evenly distribute bus numbers for the (presumably) per-vNUMA node pcie-expander buses and explicitly specify the busNr for each. David
On 10/5/26 5:40 AM, David Vrabel wrote:
On 05/10/2026 05:18, Laine Stump via Devel wrote:
* 3) If at all possible, we want to avoid needing to change the busNr * of a bus in the future, as that changes the guest's device ABI, * which could potentially lead to issues with a guest OS that is * picky about such things.
(Side note: I was the author of the above comment, made 10 years ago, and I was mainly concerned about runtime changes in ABI, for example during a migration from one host to another where the guest OS doesn't reboot. That would never happen in this case, since the busNr is stored with other attributes of the controller device whether it was set explicitly by the user or auto-set to a default value by libvirt when the domain was defined (or when the controller was added, if that happened later). A change in the default busNr would only have an effect on a newly defined domain (or newly added pcie-expander-bus).
This change violates this constraint.
When a new domain is created and no busNr is given, the default value decided on by libvirt at that time is written into the XML, so subsequent starts of that particular domain using that definition will use the same busNr - only newly created domains will get a different default busNr than they used to. So in that case, this change in default setting has 0 effect. As for migration (and domain save/restore) the change here will never have any effect on runtime ABI at all, since whatever the setting of busNr (whether it was the default value set by libvirt, or explicitly set by the user in the XML config, when the domain was defined) that will be written in the XML that is sent along with the migration data and so the destination end of the migration won't ever be using just "the current default busNr setting"; it will always be using "the exact same busNr that was used when the guest was started". The place where this change *would* have any effect is when the management software doesn't re-use the defined domain for subsequent restarts, but instead just redefines the domain from its original internal format each time it is started. However, note that even in that case the change in busNr will only happen when the guest is completely stopped, shutdown, and booted from scratch (new QEMU process and all). So there will be no case of a running OS having the hardware "change out from under it", there will only be cases where the OS is shut down, and then when it is booted the next time the busNr will be different. Also note that a similar "change in ABI" can happen if, for example, a new device is added to the "raw" XML config of the management software (which is missing PCI addresses), and then that XML is fed back into libvirt - since libvirt internally determines the ordering of the devices (and thus which device is assigned to which PCI address) you can easily end up with, e.g. a network card suddenly appearing at a different PCI address after adding a disk device to the config. If that (changing the PCI address of an endpoint device between OS boots) is acceptable behavior, then I'd say changing the bus number of a PCI controller between guest OS boots is also acceptable. (another example of guest ABI quietly changing between reboots is when a domain is defined using a machinetype of simply "q35". That generic name will be canonicalized into, e.g. pc-q35-10.1 and the guest OS will be booted with that exact machinetype and any migration will convey that exact machinetype to the destination of the migration. But when the guest is stopped, if the management software re-defines the domain from scratch (rather than continuing to use the same domain definition) still using "q35", then the next time it is booted it might come up with a machinetype of pc-q35-11.2 (if the host's QEMU package has been updated).)
As a non-backward compatible change to the guest ABI I don't think you can change the default bus allocation method.
While I agree that the busNr shouldn't be changed during the lifetime of a guest OS (e.g. during migration), this change in default at domain definition time doesn't lead to a busNr changing during migration. If you are aware of any guest OS that cares whether such a change occurs between reboots, then that's very interesting and is something we need to consider.
You may find it more useful for your management plane to evenly distribute bus numbers for the (presumably) per-vNUMA node pcie-expander buses and explicitly specify the busNr for each.
If someone wants to explicitly specify the busNr they're free to do that (as suggested in the comments) just as they are free to manually specify the PCI address of each device. This is just setting a default value at domain definition time that is more likely to be usable in the general case than the original default; the current default has limited value, since it doesn't work if you have more than a single hotpluggable endpoint device on one pcie-expander-bus anyway. In any case libvirt has always saved the auto-assigned default value of busNr (and other obscure things normally auto-set by libvirt, like chassis number) in the domain config so that they would be maintained during migration, or at a subsequent start of the domain using the same domain definition. (Again, if you're aware of an example of busNr changing from one boot to the next of a guest OS, then that would require more caution. Or maybe I'm being too laissez faire about it, but my current opinion is that I'm not :-)).
On 05/10/2026 20:19, Laine Stump wrote:
When a new domain is created and no busNr is given, the default value decided on by libvirt at that time is written into the XML, so subsequent starts of that particular domain using that definition will use the same busNr - only newly created domains will get a different default busNr than they used to. So in that case, this change in default setting has 0 effect.
There are management stacks that re-create the XML for every power on, so every power cycle these VMs would see different SBDFs across the libvirt upgrade and some VMs (e.g., Windows) get upset if SBDFs change (of which I'm sure you're aware). If you think management stacks that do that should also take care of the busNr allocations then I personally find that acceptable (as the management stack I care about already does this), but I'm not sure I'd want to make assumptions about the behaviour of every one. So whilst I agree that the current bus number allocation algorithm isn't very useful, I do think a new algorithm needs to be explicitly requested by the user or management stack. You could have something like: <controller type='pci' index='1' model='pcie-expander-bus'> <model name='pxb-pcie'/> <target busNr='+16'/> ... Where the +16 means reserve 16 "spare" bus numbers for downstream devices/ports, without having to care about exactly which bus numbers are allocated. The default value is the current behaviour of "+2". If you wish to continue with the patch as-is, I think the description could benefit from a clearer description of the current allocation algorithm (e.g., clarify that numbering starts from the highest bus and goes down). David
tl;dr - you are correct and I don't know where my head was. :-) On 10/6/26 12:27 PM, David Vrabel wrote:
On 05/10/2026 20:19, Laine Stump wrote:
When a new domain is created and no busNr is given, the default value decided on by libvirt at that time is written into the XML, so subsequent starts of that particular domain using that definition will use the same busNr - only newly created domains will get a different default busNr than they used to. So in that case, this change in default setting has 0 effect.
There are management stacks that re-create the XML for every power on,
Yeah, I hadn't forgotten that, and was even considering those systems, but...
so every power cycle these VMs would see different SBDFs across the libvirt upgrade
.. my mind was somehow blocking the fact that the attribute we call busNr directly determines the *actual* bus number of the controller and subordinate devices[*]. I mean, it completely makes sense that's what it would be used for (it's right there in the name!), but for some reason as I was looking at it, my brain was thinking of it more like the "chassis" attribute, which (as far as I've been told by lower level guys) is visible to the guest in a register somewhere, but not used to enumerate (and sometimes name) devices the way that SBDF is (it was you saying "SBDF" that finally jogged my memory). [*] as opposed to the "bus" attribute of the <address type='pci'/> element in libvirt, which does get passed directly to QEMU, but is really just an index in a table somewhere in QEMU (I'm assuming), and not necessarily related to the bus number that is visible to the guest OS - we've spent a lot of time trying to explain that over the years to users who expect that when they set "bus='5'" in the XML, the device will always show up in the guest on bus 5, even if they have no bus 4 (for example))
and some VMs (e.g., Windows) get upset if SBDFs change (of which I'm sure you're aware).
Painfully! I don't know if it's still the case, but way back when libvirt began explicitly assigning pci addresses to devices and saving them in the domain XML, I think one of the reasons was that Windows could invalidate the guest OS's license key activation if certain/too many devices' pci addresses changed, and also would pop up a window about a "new network connection" if the pci address of a network device changed (and if you think *that's* annoying, it also pops up that dialog if the MAC address of the next hop default route changes :-/). And you can't even point the finger only at Windows - in modern Linux with "predictable" network device naming, the name of a PCI network device will change when its PCI address changes, thus invalidating any config based on the device name (which I should have noticed when testing to verify this patch fixed the originally reported problem, but was so happy it booted Fedora from the pcie-expander-bus connected disk image that I only looked to see if the pcie-expander-bus connected NIC had gotten a an IP from DHCP and was able to communicate).
If you think management stacks that do that should also take care of the busNr allocations then I personally find that acceptable (as the management stack I care about already does this),
It's a bit of a digression, but really I think the management layer shouldn't need to worry about details like PCI addresses at all - they should be able to just say "make this device hotpluggable, put this device on NUMA node 1 and not hotpluggable" etc, and [someone else] would figure out the details of PCI topology for them. That's a bit outside libvirt's scope though (even though libvirt does automatically assign PCI addresses for all PCI devices, that was originally done as a defense mechanism to assure that PCI addresses remained stable as devices were added and removed, and just encoded what addresses had been previously auto-assigned by QEMU on startup, and grew organically from there).
but I'm not sure I'd want to make assumptions about the behaviour of every one.
So whilst I agree that the current bus number allocation algorithm isn't very useful, I do think a new algorithm needs to be explicitly requested by the user or management stack.
Yeah, after you've jogged my memory about what a change in busNr really leads to, I agree, and am dropping this patch. Thanks for bringing me back to reality :-)
participants (3)
-
David Vrabel -
Laine Stump -
Laine Stump