[libvirt] Analysis of the effect of adding PCIe root ports

I was asked to look at the impact on boot times of adding (empty) PCIe root ports to the device model. The proposal from Laine is to add a few of these to every guest to allow hotplugging. Last time I looked into this I found that probing any (legacy) PCI device is expensive because of the inefficient way that qemu emulates accesses to PCI config space, requiring IIRC 2 or 4 VMEXITs to access every word. (PCI slots which are not occupied are basically free, the problem is PCI devices). At that time I did not look at Q35/PCIe at all. We generally aim for boot times under 600-700ms. Probing PCI devices takes a significant fraction of this time. The detailed analysis is attached. It comes from a program called 'boot-analysis' (https://github.com/libguestfs/libguestfs/tree/master/utils/boot-analysis). It is best viewed using 'less -r' so that you can see the colours. The summary table is: Number of ports 0 1 2 3 4 bios:overhead 874 884 875 878 935 kernel:entry 159 163 165 174 491 kernel:initcalls-before-userspace 1065 1090 1110 1147 1263 /init:udev-overhead 173 187 185 193 301 insmod virtio_pci 43 41 41 41 74 TOTAL + 51 + 62 + 119 + 750 (All times are in milliseconds.) A few observations: (1) For #ports <= 3, each extra port adds around 30-40ms to the boot time, which is consistent with what I saw when I measured legacy PCI. (2) There is a sudden, unexplained and reproducible discontinuity when going from 3 to 4 ports. (Because libguestfs uses other devices, this might not actually be 3 to 4 PCIe devices, this is spare ports after all the others added for libguestfs devices.) (3) "kernel:entry" covers a lot of initialization that happens before the kernel prints its version string. This is moderately affected by adding PCIe ports until we hit the discontinuity. Based on this analysis I conclude: (a) Unless we can explain the discontinuity, avoid adding more than 3 root ports. (b) It would be nice to turn the whole thing off for people who don't care about / need hotplugging. (c) We could make PCI probing a lot faster by having the kernel handle PCI config space. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org