Re: [libvirt] Analysis of the effect of adding PCIe root ports
On 10/05/2016 11:50 AM, Richard W.M. Jones wrote:
I was asked to look at the impact on boot times of adding (empty)
PCIe
root ports to the device model. The proposal from Laine is to add a
few of these to every guest to allow hotplugging.
Last time I looked into this I found that probing any (legacy) PCI
device is expensive because of the inefficient way that qemu emulates
accesses to PCI config space, requiring IIRC 2 or 4 VMEXITs to access
every word. (PCI slots which are not occupied are basically free, the
problem is PCI devices). At that time I did not look at Q35/PCIe at all.
We generally aim for boot times under 600-700ms. Probing PCI devices
takes a significant fraction of this time.
The detailed analysis is attached. It comes from a program called
'boot-analysis'
(https://github.com/libguestfs/libguestfs/tree/master/utils/boot-analysis).
It is best viewed using 'less -r' so that you can see the colours.
The summary table is:
Number of ports 0 1 2 3 4
bios:overhead 874 884 875 878 935
kernel:entry 159 163 165 174 491
kernel:initcalls-before-userspace 1065 1090 1110 1147 1263
/init:udev-overhead 173 187 185 193 301
insmod virtio_pci 43 41 41 41 74
TOTAL + 51 + 62 + 119 + 750
(All times are in milliseconds.)
A few observations:
(1) For #ports <= 3, each extra port adds around 30-40ms to the boot
time, which is consistent with what I saw when I measured legacy PCI.
Thanks for doing this (and especially on such short notice!). This is
interesting info. I'm wondering now if the discontinuity is due to going
over a threshold in the number of pcie-root-ports, the number of total
PCI controllers, or the total number of devices. This is significant
because a couple of the patches in the "Use more PCIe less legacy PCI"
patchset I sent a two weeks ago will eliminate both the
dmi-to-pci-bridge and the pci-bridge in the default config (as long as
there are no legacy PCI devices). If the "jump" is due to an increase in
total devices, or total PCI controllers, then getting rid of those two
will change the number of root-ports that can be added prior to the jump.
Even if this is the case, it seems like we still want to have as little
fat as possible.
(2) There is a sudden, unexplained and reproducible discontinuity when
going from 3 to 4 ports. (Because libguestfs uses other devices, this
might not actually be 3 to 4 PCIe devices, this is spare ports after
all the others added for libguestfs devices.)
(3) "kernel:entry" covers a lot of initialization that happens before
the kernel prints its version string. This is moderately affected by
adding PCIe ports until we hit the discontinuity.
Based on this analysis I conclude:
(a) Unless we can explain the discontinuity, avoid adding more than
3 root ports.
(b) It would be nice to turn the whole thing off for people who don't
care about / need hotplugging.
I had contemplated having an "availablePCIeSlots" (or something like
that) that was either an attribute of the config, or an option in
qemu.conf or libvirtd.conf. If we had such a setting, it could be set to
"0".
(c) We could make PCI probing a lot faster by having the kernel handle
PCI config space.