12:47 p.m.
Recently I've been playing with virt autotest, which provides a number of qemu
and libvirt functional tests. Here's some notes on how to get it running:
The super short version of running the libvirt smoke tests on Fedora 18:
yum install autotest-framework
git clone git://github.com/autotest/virt-test.git autotest-virt-tests
cd autotest-virt-tests
./libvirt/get_started.py
./run --type libvirt
For me the default tests take under a minute. Output looks like:
SETUP: PASS (1.00 s)
DATA DIR: /home/crobinso/virt_test
DEBUG LOG:
/home/crobinso/src/autotest-virt-tests/logs/run-2013-02-11-14.21.47/debug.log
TESTS: 28
(1/28) unattended_install.import.import: PASS (23.31 s)
(2/28) virsh_domname.vm_state.vm_running.with_valid_option.domid: PASS (1.59 s)
(3/28) virsh_domname.vm_state.vm_running.with_valid_option.uuid: PASS (0.47 s)
...
(27/28) virsh_domname.with_libvirtd_stop.with_valid_option.uuid: PASS (0.55 s)
(28/28) remove_guest.without_disk: PASS (6.77 s)
TOTAL TIME: 42.30 s
TESTS PASSED: 28
TESTS FAILED: 0
SUCCESS RATE: 100.00 %
Now with a bit more detail.
The first 4 steps are one time setup, though you will want to update
autotest-virt-tests every now and then.
The default set of libvirt tests import a VM with virt-install, verify that we
can SSH in, run a bunch of virsh domname validation, and remove the guest.
Logs are dumped into the autotest-virt-tests/logs directory.
autotest-framework is just the client side piece. autotest also has a server
component which has web UI, handles job scheduling, and tracks results in a
database, but not needed for casual developer usage.
virt-test.git is the repo that actually contains all the qemu and libvirt test
cases. There are also tests for virt-v2v, libguestfs, ovirt, and openvswitch,
but qemu and libvirt have the most tests by far. libvirt has 600 test cases,
but many of these are just parameter validation matrices. libvirt TCK likely
has more functional tests.
All the code is in python. The libvirt test cases actually use virsh and not
the python bindings. This was a deliberate choice to get better virsh
coverage. The code is pretty nice so impl wise dealing with virsh isn't hard
at all.
./libvirt/get_started.py is an interactive script that ensures needed
dependencies are installed, and pulls down a pre made Fedora 17 jeos (just
enough operating system) image. All the tests run against that image by default.
./run is how you'll interact with everything. The important invocations:
# List all libvirt tests
./run --type libvirt --list-tests
# Show all test debugging and stdout to the console
./run --verbose ...
# Run only specific tests. The default libvirt set looks like
./run --type libvirt --tests "unattended_install.import.import virsh_domname
remove_guest.without_disk"
# See all options
./run --help
Autotest has useful and simple infrastructure for logging in to virtual
machines and running commands in there. So for doing end to end tests like
verifying hotplug succeeded or verifying snapshots actually worked it seems
quite handy. It can also assemble a video based on screenshots of the guest,
so you can rewatch an install failure or similar.
A few misc things:
- The default libvirt tests seem to work fine as non-root, though every
libvirt test is marked as 'root only'. I'm not sure what the practical effect
of that is.
- If you interrupt the tests, you'll have to destroy and undefine the
'autotest-vm1' VM, or next run will fail. If the test suite completes, it will
clean up after itself. All this should be fixed.
- There are migration tests for libvirt, but I haven't looked at them.
- All the tests run against system installed libvirt. You can probably get
around this by sticking /path/to/libvirt.git/tools in PATH, but some tests
want to restart libvirtd so it isn't that simple.
- Running the tests creates ~/virt_test and moves the F17 image there. If you
move other media there they can be used for various install tests. You can
override this with VIRT_TEST_DATA_DIR or ./run --data-dir, but IMO there
should be a ~/.autotestconfig or something.
- qemu also has a basic set of smoke tests that test migration, no extra
config required. A custom qemu binary can be used with ./run --qemu-bin.
Probably useful even for libvirt devs that are tracking upstream qemu, if you
suspect qemu.git may be currently be broken.
The virt-test wiki also has much more useful info:
https://github.com/autotest/virt-test/wiki
Thanks,
Cole