[libvirt] libvirt TCK wrapper for autotest review
Hi guys, I was here looking at the autotest wrapper for libvirt TCK and then decided to work on it, as I had the review fresh on my mind. Things that I've worked on: * Fixed some download links, that were already sent to upstream tck and applied (thanks Dan Berrange) * Instead of making all tests output to the same DEBUG log, make them output to separate .tap files on the results directory * Run all tests available for a given item, rather than stopping the test on the first failure * removed capitalization on the wrapper name, since it's project policy * Use os.environ, and some features of the subcommand execution API to execute the tests * Remove usages of error.JobError, as the problems there are more error.TestError, since they are restricted to the libvirt_tck test, not the entire job (in autotest, a job can do more stuff than just a sequence of job.runtest() calls). * Made the error messages more descriptive, with info of all failed tests So, the current output of the tests is like this: $ sudo client/bin/autotest run libvirt_tck 18:29:27 INFO | Writing results to /home/lmr/Code/autotest.lmr/client/results/default 18:29:27 INFO | START ---- ---- timestamp=1329942567 localtime=Feb 22 18:29:27 18:29:27 INFO | START libvirt_tck.domain libvirt_tck.domain timestamp=1329942567 localtime=Feb 22 18:29:27 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['120-disks-stats.t', '205-disk-hotplug-ordering.t'] 18:30:19 INFO | END FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['051-daemon-hook.t', '052-domain-hook.t'] 18:30:19 INFO | END FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.networks libvirt_tck.networks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:28 INFO | GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 completed successfully 18:30:28 INFO | END GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:28 INFO | START libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:32 ERROR| child process failed 18:30:32 INFO | FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 FAIL: ['090-install-image.t', '100-ping-still-working.t', '210-no-mac-spoofing.t', '220-no-ip-spoofing.t', '230-no-mac-broadcast.t', '240-no-arp-spoofing.t', '300-vsitype.t'] 18:30:32 INFO | END FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:32 INFO | START libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:40 ERROR| child process failed 18:30:40 INFO | FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 FAIL: ['205-qcow2-double-backing-file.t'] 18:30:40 INFO | END FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:40 INFO | START libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:49 ERROR| child process failed 18:30:49 INFO | FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 FAIL: ['055-dynamic-base-label.t', '100-static-relabel-no.t'] 18:30:49 INFO | END FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 18:30:49 INFO | START libvirt_tck.storage libvirt_tck.storage timestamp=1329942649 localtime=Feb 22 18:30:49 18:31:24 INFO | GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 completed successfully 18:31:24 INFO | END GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 18:31:24 INFO | END GOOD ---- ---- timestamp=1329942684 localtime=Feb 22 18:31:24 As time allows, I might take a look at the failures and help with libvirt_tck. I've combined all modifications to a single, self contained commit. Also, as the work is self contained, it could be very very easily rebased to the latest upstream tree. I've updated my personal repo and sent a github pull request that you guys can see and review: https://github.com/autotest/autotest/pull/192 I'm still not going to merge this to the upstream tree just yet, since I'd like to hear some feedback from you guys. As for developing together with autotest, I guess you can easily use the clone you have on libvirt now, and on your working directory you can add the following remote: [remote "upstream"] fetch = +refs/heads/*:refs/remotes/origin/* url = git://github.com/autotest/autotest.git Then from time to time, you can just pull from upstream to your master branch: git pull upstream master And then rebase your development trees: git checkout tck git rebase master It should be painless, given the fact that the wrapper is pretty much self contained, and isolated from the rest of the code. Please let me know what you think, Lucas
Hi Lucas, Thanks for your these good modifications. There is one place I noticed where you output each testcase of "*.t" into a separate file with .tap extension. hence, it has a corresponding log file with little content for each testcase. it seem a little harder to check compared to just one log file. The rest of them is perfect for me. Guannan Ren On 02/23/2012 04:45 AM, Lucas Meneghel Rodrigues wrote:
Hi guys,
I was here looking at the autotest wrapper for libvirt TCK and then decided to work on it, as I had the review fresh on my mind. Things that I've worked on:
* Fixed some download links, that were already sent to upstream tck and applied (thanks Dan Berrange) * Instead of making all tests output to the same DEBUG log, make them output to separate .tap files on the results directory * Run all tests available for a given item, rather than stopping the test on the first failure * removed capitalization on the wrapper name, since it's project policy * Use os.environ, and some features of the subcommand execution API to execute the tests * Remove usages of error.JobError, as the problems there are more error.TestError, since they are restricted to the libvirt_tck test, not the entire job (in autotest, a job can do more stuff than just a sequence of job.runtest() calls). * Made the error messages more descriptive, with info of all failed tests
So, the current output of the tests is like this:
$ sudo client/bin/autotest run libvirt_tck 18:29:27 INFO | Writing results to /home/lmr/Code/autotest.lmr/client/results/default 18:29:27 INFO | START ---- ---- timestamp=1329942567 localtime=Feb 22 18:29:27 18:29:27 INFO | START libvirt_tck.domain libvirt_tck.domain timestamp=1329942567 localtime=Feb 22 18:29:27 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['120-disks-stats.t', '205-disk-hotplug-ordering.t'] 18:30:19 INFO | END FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['051-daemon-hook.t', '052-domain-hook.t'] 18:30:19 INFO | END FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.networks libvirt_tck.networks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:28 INFO | GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 completed successfully 18:30:28 INFO | END GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:28 INFO | START libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:32 ERROR| child process failed 18:30:32 INFO | FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 FAIL: ['090-install-image.t', '100-ping-still-working.t', '210-no-mac-spoofing.t', '220-no-ip-spoofing.t', '230-no-mac-broadcast.t', '240-no-arp-spoofing.t', '300-vsitype.t'] 18:30:32 INFO | END FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:32 INFO | START libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:40 ERROR| child process failed 18:30:40 INFO | FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 FAIL: ['205-qcow2-double-backing-file.t'] 18:30:40 INFO | END FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:40 INFO | START libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:49 ERROR| child process failed 18:30:49 INFO | FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 FAIL: ['055-dynamic-base-label.t', '100-static-relabel-no.t'] 18:30:49 INFO | END FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 18:30:49 INFO | START libvirt_tck.storage libvirt_tck.storage timestamp=1329942649 localtime=Feb 22 18:30:49 18:31:24 INFO | GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 completed successfully 18:31:24 INFO | END GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 18:31:24 INFO | END GOOD ---- ---- timestamp=1329942684 localtime=Feb 22 18:31:24
On 02/23/2012 12:27 PM, Guannan Ren wrote:
Hi Lucas,
Thanks for your these good modifications. There is one place I noticed where you output each testcase of "*.t" into a separate file with .tap extension. hence, it has a corresponding log file with little content for each testcase. it seem a little harder to check compared to just one log file. The rest of them is perfect for me.
Guannan Ren
Thanks! Now, feel free to pick this up and modify as you wish, then send me as a pull request/patch to the mailing list. I'll close the example pull request and will be waiting on you, ok? Cheers, Lucas
On 02/24/2012 07:50 PM, Lucas Meneghel Rodrigues wrote:
On 02/23/2012 12:27 PM, Guannan Ren wrote:
Hi Lucas,
Thanks for your these good modifications. There is one place I noticed where you output each testcase of "*.t" into a separate file with .tap extension. hence, it has a corresponding log file with little content for each testcase. it seem a little harder to check compared to just one log file. The rest of them is perfect for me.
Guannan Ren
Thanks!
Now, feel free to pick this up and modify as you wish, then send me as a pull request/patch to the mailing list. I'll close the example pull request and will be waiting on you, ok?
Cheers,
Lucas
That's ok, after I make some modification, I send the patch, thanks. Guannan Ren
On Wed, Feb 22, 2012 at 06:45:58PM -0200, Lucas Meneghel Rodrigues wrote:
* Instead of making all tests output to the same DEBUG log, make them output to separate .tap files on the results directory
So, the current output of the tests is like this:
$ sudo client/bin/autotest run libvirt_tck 18:29:27 INFO | Writing results to /home/lmr/Code/autotest.lmr/client/results/default 18:29:27 INFO | START ---- ---- timestamp=1329942567 localtime=Feb 22 18:29:27 18:29:27 INFO | START libvirt_tck.domain libvirt_tck.domain timestamp=1329942567 localtime=Feb 22 18:29:27 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['120-disks-stats.t', '205-disk-hotplug-ordering.t'] 18:30:19 INFO | END FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['051-daemon-hook.t', '052-domain-hook.t'] 18:30:19 INFO | END FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.networks libvirt_tck.networks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:28 INFO | GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 completed successfully 18:30:28 INFO | END GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:28 INFO | START libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:32 ERROR| child process failed 18:30:32 INFO | FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 FAIL: ['090-install-image.t', '100-ping-still-working.t', '210-no-mac-spoofing.t', '220-no-ip-spoofing.t', '230-no-mac-broadcast.t', '240-no-arp-spoofing.t', '300-vsitype.t'] 18:30:32 INFO | END FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:32 INFO | START libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:40 ERROR| child process failed 18:30:40 INFO | FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 FAIL: ['205-qcow2-double-backing-file.t'] 18:30:40 INFO | END FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:40 INFO | START libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:49 ERROR| child process failed 18:30:49 INFO | FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 FAIL: ['055-dynamic-base-label.t', '100-static-relabel-no.t'] 18:30:49 INFO | END FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 18:30:49 INFO | START libvirt_tck.storage libvirt_tck.storage timestamp=1329942649 localtime=Feb 22 18:30:49 18:31:24 INFO | GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 completed successfully 18:31:24 INFO | END GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 18:31:24 INFO | END GOOD ---- ---- timestamp=1329942684 localtime=Feb 22 18:31:24
FYI, the TCK has a pluggable data output format, configurable via the --format option. By default it outputs in "text" format, which is the standard Perl TAP format. You can change it to 'html' (pretty printed) 'xml' (a custom structured format) or 'junit' (the JUnit standard). More options can be added by subclasses TAP::Formatter - there might be even more on CPAN already I don't know about. So whatever is the optional format for Autotest, we should make the TCK able to generate natively Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
On 02/23/2012 11:08 PM, Daniel P. Berrange wrote:
On Wed, Feb 22, 2012 at 06:45:58PM -0200, Lucas Meneghel Rodrigues wrote:
* Instead of making all tests output to the same DEBUG log, make them output to separate .tap files on the results directory So, the current output of the tests is like this:
$ sudo client/bin/autotest run libvirt_tck 18:29:27 INFO | Writing results to /home/lmr/Code/autotest.lmr/client/results/default 18:29:27 INFO | START ---- ---- timestamp=1329942567 localtime=Feb 22 18:29:27 18:29:27 INFO | START libvirt_tck.domain libvirt_tck.domain timestamp=1329942567 localtime=Feb 22 18:29:27 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['120-disks-stats.t', '205-disk-hotplug-ordering.t'] 18:30:19 INFO | END FAIL libvirt_tck.domain libvirt_tck.domain timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 ERROR| child process failed 18:30:19 INFO | FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 FAIL: ['051-daemon-hook.t', '052-domain-hook.t'] 18:30:19 INFO | END FAIL libvirt_tck.hooks libvirt_tck.hooks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:19 INFO | START libvirt_tck.networks libvirt_tck.networks timestamp=1329942619 localtime=Feb 22 18:30:19 18:30:28 INFO | GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 completed successfully 18:30:28 INFO | END GOOD libvirt_tck.networks libvirt_tck.networks timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:28 INFO | START libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942628 localtime=Feb 22 18:30:28 18:30:32 ERROR| child process failed 18:30:32 INFO | FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 FAIL: ['090-install-image.t', '100-ping-still-working.t', '210-no-mac-spoofing.t', '220-no-ip-spoofing.t', '230-no-mac-broadcast.t', '240-no-arp-spoofing.t', '300-vsitype.t'] 18:30:32 INFO | END FAIL libvirt_tck.nwfilter libvirt_tck.nwfilter timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:32 INFO | START libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942632 localtime=Feb 22 18:30:32 18:30:40 ERROR| child process failed 18:30:40 INFO | FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 FAIL: ['205-qcow2-double-backing-file.t'] 18:30:40 INFO | END FAIL libvirt_tck.qemu libvirt_tck.qemu timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:40 INFO | START libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942640 localtime=Feb 22 18:30:40 18:30:49 ERROR| child process failed 18:30:49 INFO | FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 FAIL: ['055-dynamic-base-label.t', '100-static-relabel-no.t'] 18:30:49 INFO | END FAIL libvirt_tck.selinux libvirt_tck.selinux timestamp=1329942649 localtime=Feb 22 18:30:49 18:30:49 INFO | START libvirt_tck.storage libvirt_tck.storage timestamp=1329942649 localtime=Feb 22 18:30:49 18:31:24 INFO | GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 completed successfully 18:31:24 INFO | END GOOD libvirt_tck.storage libvirt_tck.storage timestamp=1329942684 localtime=Feb 22 18:31:24 18:31:24 INFO | END GOOD ---- ---- timestamp=1329942684 localtime=Feb 22 18:31:24 FYI, the TCK has a pluggable data output format, configurable via the --format option. By default it outputs in "text" format, which is the standard Perl TAP format. You can change it to 'html' (pretty printed) 'xml' (a custom structured format) or 'junit' (the JUnit standard).
More options can be added by subclasses TAP::Formatter - there might be even more on CPAN already I don't know about. So whatever is the optional format for Autotest, we should make the TCK able to generate natively
Daniel
There, I should mention something, the autotest shows the results in its webpage according to the testing result of each testrun, for the well-granulated result-showing sake, I let autotest do the log saving and result checking work instead of TCK itself. If we use TCK to manage the log, the autotest would know a little about each testrun beside the return value from TCK's main command. Another way to make the glue work go well is using TAP parser to parse the TAP file generated by TCK and show the results to webpage, but right now, parser in autotest doesn't support this idea as far as I know. Guannan Ren /// /
participants (3)
-
Daniel P. Berrange -
Guannan Ren -
Lucas Meneghel Rodrigues