Re: [libvirt] Analysis of the effect of adding PCIe root ports

On 10/06/2016 11:31 AM, Daniel P. Berrange wrote: > On Thu, Oct 06, 2016 at 12:58:51PM +0200, Andrea Bolognani wrote: > > On Wed, 2016-10-05 at 18:36 +0100, Richard W.M. Jones wrote: > > > > > (b) It would be nice to turn the whole thing off for people who don't > > > > > care about / need hotplugging. > > > > I had contemplated having an "availablePCIeSlots" (or something like > > > > that) that was either an attribute of the config, or an option in > > > > qemu.conf or libvirtd.conf. If we had such a setting, it could be > > > > set to "0". > > I remember some pushback when this was proposed. Maybe we > > should just give up on the idea of providing spare > > hotpluggable PCIe slots by default and ask the user to add > > them explicitly after all. > > > > > Note that changes to libvirt conf files are not usable by libguestfs. > > > The setting would need to go into the XML, and please also make it > > > possible to determine if $random version of libvirt supports the > > > setting, either by a version check or something in capabilities. > > Note that you can avoid using any PCIe root port at all by > > assigning PCI addresses manually. It looks like the overhead > > for the small (I'm assuming) number of devices a libguestfs > > appliance will use is low enough that you will probably not > > want to open that can of worm, though. > For most apps the performance impact of the PCI enumeration > is not a big deal. So having libvirt ensure there's enough > available hotpluggable PCIe slots is reasonable, as long as > we leave a get-out clause for libguestfs. > > This could be as simple as declaring that *if* we see one > or more <controller type="pci"> in the input XML, then libvirt > will honour those and not try to add new controllers to the > guest. > > That way, by default libvirt will just "do the right thing" > and auto-create a suitable number of controllers needed to > boot the guest. > > Apps that want strict control though, can specify the > <controllers> elements themselves. Libvirt can still > auto-assign device addresses onto these controllers. > It simply wouldn't add any further controllers itself > at that point. NB I'm talking cold-boot here. So libguestfs > would specify <controller> itself to the minimal set it wants > to optimize its boot performance. That works for the initial definition of the domain, but as soon as you've saved it once, there will be controllers explicitly in the config, and since we don't have any way of differentiating between auto-added controllers and those specifically requested by the user, we have to assume they were explicitly added, so such a check is then meaningless because you will *always* have PCI controllers.

Say you create a domain definition with no controllers, you would get enough for the devices in the initial config, plus "N" more empty root ports. Let's say you then add 4 more devices (either hotplug or coldplug, doesn't matter). Those devices are placed on the existing unused pcie-root-ports. But now all your ports are full, and since you have PCI controllers in the config, libvirt is going to say "Ah, this user knows what they want to do, so I'm not going to add any extras! I'm so smart!". This would be especially maddening in the case of "coldplug", where libvirt could have easily added a new controller to accomodate the new device, but didn't. Unless we don't care what happens after the initial definition (and then adding of "N" new devices), trying to behave properly purely based on whether or not there are any PCI controllers present in the config isn't going to work.