[libvirt] [PATCH RFC 0/8] qemu: Cache results of parsing qemu help output

Stracing libvirtd shows that the qemu driver is executing 2 different qemu binaries 3 times each to fetch the version, capabilities [supported devices], and cpu models each time VM state is queried. E.g., [lines wrapped]: 6471 17:15:26.561890 execve("/usr/bin/qemu", ["/usr/bin/qemu", "-cpu", "?"], [/* 2 vars */]) = 0 6472 17:15:26.626668 execve("/usr/bin/qemu", ["/usr/bin/qemu", "-help"], [/* 2 vars */]) = 0 6473 17:15:26.698104 execve("/usr/bin/qemu", ["/usr/bin/qemu", "-device", "?", "-device", "pci-assign,?", "-device", "virtio-blk-pci,?"], [/* 2 vars */]) = 0 6484 17:15:27.267770 execve("/usr/bin/qemu-system-x86_64", ["/usr/bin/qemu-system-x86_64", "-cpu", "?"], /* 2 vars */]) = 0 6492 17:15:27.333177 execve("/usr/bin/qemu-system-x86_64", ["/usr/bin/qemu-system-x86_64", "-help"], [/* 2 vars */]) = 0 6496 17:15:27.402280 execve("/usr/bin/qemu-system-x86_64", ["/usr/bin/qemu-system-x86_64", "-device", "?", "-device", "pci-assign,?", "-device", "virtio-blk-pci,?"], [/* 2 vars */]) = 0 ~1sec per libvirt api call. Not a killer, but on a heavily loaded host -- several 10s of VMs -- a periodic query of all VM state, such as from a cloud compute manager, can take a couple of minutes to complete. Because the qemu binaries on the host do not change all that often, the results of parsing the qemu help output from the exec's above can be cached. The qemu driver already does some caching of capabilities, but it does not prevent the execs above. This series is a work in progress. I'm submitting it as an RFC because I saw Eric mention the frequent execing of qemu binaries and I have been working on this to eliminate the overhead shown above. The series caches the parse results of: + qemuCapsExtractVersionInfo + qemuCapsProbeMachineTypes + qemuCapsProbeCPUModels by splitting these functions into two parts. The existing function name fetches the cached parse results for the specified binary and returns them. The other half, named "qemuCapsCacheX", where X is one of ExtractVersionInfo, ProbeMachineTypes, and ProbeCPUModels, exec's the emulator binary and caches the results. The act of fetching the cached results will fill or refresh the cache as necessary in a new function qemuCapsCachedInfoGet(). A few auxilliary function have been added -- e.g., virCapabilitiesDupMachines() to duplicate a cached list of machine types and virBitmapDup() to duplicate cached capabilities flags. The series does not attempt to integrate with nor remove the existing capabilities caching. TBD. The series was developed and tested in the context of the Ubuntu 11.04 natty libvirt_0.8.8-1ubuntu6.7 package using quilt to manage patches in the debian/patches directory. In that context, it builds, passes all "make check" tests [under pbuilder] and some fairly heavy, overlapping VM launch tests where it does eliminate all but a few initial exec's of the various qemu* and kvm binaries. The version here, rebased to libvirt-0.9.10, builds cleanly under mock on Fedora 16 in the context of a modified libvirt-0.9.10-1.fc16 source package. I.e., no errors and warning-for-warning compatible with build of the libvirt-0.9.10 fc16 srpm downloaded from libvirt.org. I placed the modified spec file [applies the patches] and the build logs at: http://downloads.linux.hp.com/~lts/Libvirt/ I have installed the patched libvirt on a fedora 16 system and successfully defined and launched a vm. Testing in progress. I'll place an annotated test log ont the site above when complete. I also need to rebase atop the current mainline sources, but I wanted to get this series out for review to see if the overall approach would be acceptable. Comments?