[libvirt] Quantifying libvirt errors in launching the libguestfs appliance

As people may know, we frequently encounter errors caused by libvirt when running the libguestfs appliance. I wanted to find out exactly how frequently these happen and classify the errors, so I ran the 'virt-df' tool overnight 1700 times. This tool runs several parallel qemu:///session libvirt connections both creating a short-lived appliance guest. Note that I have added Cole's patch to fix https://bugzilla.redhat.com/1271183 "XML-RPC error : Cannot write data: Transport endpoint is not connected" Results: The test failed 538 times (32% of the time), which is pretty dismal. To be fair, virt-df is aggressive about how it launches parallel libvirt connections. Most other virt-* tools use only a single libvirt connection and are consequently more reliable. Of the failures, 518 (96%) were of the form: process exited while connecting to monitor: qemu: could not load kernel '/home/rjones/d/libguestfs/tmp/.guestfs-1000/appliance.d/kernel': Permission denied which is https://bugzilla.redhat.com/921135 or maybe https://bugzilla.redhat.com/1269975. It's not clear to me if these bugs have different causes, but if they do then potentially we're seeing a mix of both since my test has no way to distinguish them. 19 of the failures (4%) were of the form: process exited while connecting to monitor: fread() failed which I believe is a previously unknown bug. I have filed it as https://bugzilla.redhat.com/1298122 Finally there was 1 failure: Unable to read from monitor: Connection reset by peer which I believe is also a new bug. I have filed it as https://bugzilla.redhat.com/1298124 I would be good if libvirt could routinely test the case of multiple parallel launches of qemu:///session, since it still contains bugs even after Cole's fixes. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top