[Libvir] [PATCH] Add new testing framework and the first test to use it.

A week or so ago, I read a bug report on this list with a simple reproducer + fix that was just begging for a test suite addition. This change adds framework to make that easy, and adds the first test. I uncovered problems in tests/Makefile.am along the way and just posted a patch for that. This adds most of a portable (bourne-shell-based) test framework that also provides for convenient isolation of individual tests (so we can easily parallelize them without worrying one will tromp on files of another). In a couple years when running "make check" runs hundreds of tests, you'll definitely want them to run in parallel. The mktempd script is not new. I first used it in coreutils and parted, but removed it from coreutils after that package acquired its own C-based implementation. test-lib.sh is from coreutils, but was inspired by the file by the same name in git.git's own t/ (tests) directory. Add new testing framework and the first test to use it. * tests/Makefile.am (test_scripts): Add vcpupin. (EXTRA_DIST): Add test-lib.sh. * tests/test-lib.sh: Testing framework, from coreutils. * tests/vcpupin: New file. * build-aux/mktempd: New file, from gnulib. * bootstrap: Add posix-shell and mktempd to the list of imported modules. * gnulib/m4/posix-shell.m4: New file, from gnulib. This also updates lots of files from gnulib, as you can see from the diffstat output. The diffs below include everything not under gnulib/. --- bootstrap | 2 + build-aux/.cvsignore | 1 + build-aux/mktempd | 132 ++++++++++++++++++++++++++++++++ build-aux/useless-if-before-free | 6 +- gnulib/lib/Makefile.am | 37 +++++++--- gnulib/lib/alloca.in.h | 4 +- gnulib/lib/getaddrinfo.c | 2 +- gnulib/lib/getdelim.c | 6 +- gnulib/lib/unistd.in.h | 22 +++++- gnulib/lib/xsize.h | 2 +- gnulib/m4/absolute-header.m4 | 49 ------------ gnulib/m4/fseeko.m4 | 8 ++- gnulib/m4/gnulib-cache.m4 | 6 +- gnulib/m4/gnulib-comp.m4 | 6 +- gnulib/m4/include_next.m4 | 7 +- gnulib/m4/lib-link.m4 | 66 ++++++++++++----- gnulib/m4/posix-shell.m4 | 58 ++++++++++++++ gnulib/m4/unistd_h.m4 | 6 +- gnulib/tests/Makefile.am | 2 +- tests/Makefile.am | 7 ++- tests/test-lib.sh | 155 ++++++++++++++++++++++++++++++++++++++ tests/vcpupin | 37 +++++++++ 22 files changed, 521 insertions(+), 100 deletions(-) create mode 100755 build-aux/mktempd delete mode 100644 gnulib/m4/absolute-header.m4 create mode 100644 gnulib/m4/posix-shell.m4 create mode 100644 tests/test-lib.sh create mode 100755 tests/vcpupin diff --git a/bootstrap b/bootstrap index d8b79d9..f7b6aec 100755 --- a/bootstrap +++ b/bootstrap @@ -79,6 +79,8 @@ $gnulib_tool \ sys_stat vasprintf strndup \ strsep poll gettext getpass \ useless-if-before-free \ + posix-shell \ + mktempd \ vc-list-files rm -f \ diff --git a/build-aux/.cvsignore b/build-aux/.cvsignore index 1798f40..096cccb 100644 --- a/build-aux/.cvsignore +++ b/build-aux/.cvsignore @@ -7,3 +7,4 @@ install-sh ltmain.sh missing mkinstalldirs +mktempd diff --git a/build-aux/mktempd b/build-aux/mktempd new file mode 100755 index 0000000..7ac914b --- /dev/null +++ b/build-aux/mktempd @@ -0,0 +1,132 @@ +#!/bin/sh +# Create a temporary directory, much like mktemp -d does. + +# Copyright (C) 2007-2008 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +# Written by Jim Meyering. + +# Usage: mktempd /tmp phoey.XXXXXXXXXX + +# First, try to use the mktemp program. +# Failing that, we'll roll our own mktemp-like function: +# - try to get random bytes from /dev/urandom +# - failing that, generate output from a combination of quickly-varying +# sources and gzip. Ignore non-varying gzip header, and extract +# "random" bits from there. +# - given those bits, map to file-name bytes using tr, and try to create +# the desired directory. +# - make only $MAX_TRIES attempts + +ME=`basename "$0"` +die() { echo >&2 "$ME: $@"; exit 1; } + +MAX_TRIES=4 + +rand_bytes() +{ + n=$1 + + chars=abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789 + + dev_rand=/dev/urandom + if test -r "$dev_rand"; then + # Note: 256-length($chars) == 194; 3 copies of $chars is 186 + 8 = 194. + head -c$n "$dev_rand" | tr -c $chars 01234567$chars$chars$chars + return + fi + + cmds='date; date +%N; free; who -a; w; ps auxww; ps ef; netstat -n' + data=` (eval "$cmds") 2>&1 | gzip ` + + n_plus_50=`expr $n + 50` + + # Ensure that $data has length at least 50+$n + while :; do + len=`echo "$data"|wc -c` + test $n_plus_50 -le $len && break; + data=` (echo "$data"; eval "$cmds") 2>&1 | gzip ` + done + + echo "$data" \ + | dd bs=1 skip=50 count=$n 2>/dev/null \ + | tr -c $chars 01234567$chars$chars$chars +} + +mktempd() +{ + case $# in + 2);; + *) die "Usage: $ME DIR TEMPLATE";; + esac + + destdir=$1 + template=$2 + + # Disallow any trailing slash on specified destdir: + # it would subvert the post-mktemp "case"-based destdir test. + case $destdir in + /) ;; + */) die "invalid destination dir: remove trailing slash(es)";; + esac + + case $template in + *XXXX) ;; + *) die "invalid template: $template (must have a suffix of at least 4 X's)";; + esac + + fail=0 + + # First, try to use mktemp. + d=`env -u TMPDIR mktemp -d -t -p "$destdir" "$template" 2>/dev/null` \ + || fail=1 + + # The resulting name must be in the specified directory. + case $d in "$destdir"*);; *) fail=1;; esac + + # It must have created the directory. + test -d "$d" || fail=1 + + # It must have 0700 permissions. Handle sticky "S" bits. + perms=`ls -dgo "$d" 2>/dev/null|tr S -` || fail=1 + case $perms in drwx------*) ;; *) fail=1;; esac + + test $fail = 0 && { + echo "$d" + return + } + + # If we reach this point, we'll have to create a directory manually. + + # Get a copy of the template without its suffix of X's. + base_template=`echo "$template"|sed 's/XX*$//'` + + # Calculate how many X's we've just removed. + nx=`expr length "$template" - length "$base_template"` + + err= + i=1 + while :; do + X=`rand_bytes $nx` + candidate_dir="$destdir/$base_template$X" + err=`mkdir -m 0700 "$candidate_dir" 2>&1` \ + && { echo "$candidate_dir"; return; } + test $MAX_TRIES -le $i && break; + i=`expr $i + 1` + done + die "$err" +} + +mktempd "$@" diff --git a/build-aux/useless-if-before-free b/build-aux/useless-if-before-free index eb18483..626d19a 100755 --- a/build-aux/useless-if-before-free +++ b/build-aux/useless-if-before-free @@ -2,7 +2,7 @@ # Detect instances of "if (p) free (p);". # Likewise for "if (p != NULL) free (p);". And with braces. -my $VERSION = '2008-02-11 08:08'; # UTC +my $VERSION = '2008-03-12 13:06'; # UTC # The definition above must lie within the first 8 lines in order # for the Emacs time-stamp write hook (at end) to update it. # If you change this file with Emacs, please let the write hook @@ -123,8 +123,8 @@ EOF { if ($line =~ /\b(if\s*\(\s*(\S+?)(?:\s*!=\s*NULL)?\s*\) - (?: \s*$regexp\s*\(\s*\2\s*\)| - \s*\{\s*$regexp\s*\(\s*\2\s*\)\s*;\s*\}))/sx) + (?: \s*$regexp\s*\((?:\s*\([^)]+\))\s*\2\s*\)| + \s*\{\s*$regexp\s*\((?:\s*\([^)]+\))\s*\2\s*\)\s*;\s*\}))/sx) { $found_match = 1; $list diff --git a/tests/test-lib.sh b/tests/test-lib.sh new file mode 100644 index 0000000..cdbea5d --- /dev/null +++ b/tests/test-lib.sh @@ -0,0 +1,155 @@ +# source this file; set up for tests + +# Skip this test if the shell lacks support for functions. +unset function_test +eval 'function_test() { return 11; }; function_test' +if test $? != 11; then + echo "$0: /bin/sh lacks support for functions; skipping this test." 1>&2 + (exit 77); exit 77 +fi + +skip_test_() +{ + echo "$0: skipping test: $@" 1>&2 + (exit 77); exit 77 +} + +require_acl_() +{ + getfacl --version < /dev/null > /dev/null 2>&1 \ + && setfacl --version < /dev/null > /dev/null 2>&1 \ + || skip_test_ "This test requires getfacl and setfacl." + + id -u bin > /dev/null 2>&1 \ + || skip_test_ "This test requires a local user named bin." +} + +require_ulimit_() +{ + ulimit_works=yes + # Expect to be able to exec a program in 10MB of virtual memory, + # but not in 20KB. I chose "date". It must not be a shell built-in + # function, so you can't use echo, printf, true, etc. + # Of course, in coreutils, I could use $top_builddir/src/true, + # but this should be able to work for other projects, too. + ( ulimit -v 10000; date ) > /dev/null 2>&1 || ulimit_works=no + ( ulimit -v 20; date ) > /dev/null 2>&1 && ulimit_works=no + + test $ulimit_works = no \ + && skip_test_ "this shell lacks ulimit support" +} + +require_readable_root_() +{ + test -r / || skip_test_ "/ is not readable" +} + +# Skip the current test if strace is not available or doesn't work. +require_strace_() +{ + strace -V < /dev/null > /dev/null 2>&1 || + skip_test_ 'no strace program' + + strace -qe unlink echo > /dev/null 2>&1 || + skip_test_ 'strace does not work' +} + +require_built_() +{ + skip_=no + for i in "$@"; do + case " $built_programs " in + *" $i "*) ;; + *) echo "$i: not built" 1>&2; skip_=yes ;; + esac + done + + test $skip_ = yes && skip_test_ "required program(s) not built" +} + +uid_is_privileged_() +{ + # Make sure id -u succeeds. + my_uid=$(id -u) \ + || { echo "$0: cannot run \`id -u'" 1>&2; return 1; } + + # Make sure it gives valid output. + case $my_uid in + 0) ;; + *[!0-9]*) + echo "$0: invalid output (\`$my_uid') from \`id -u'" 1>&2 + return 1 ;; + *) return 1 ;; + esac +} + +skip_if_() +{ + case $1 in + root) skip_test_ must be run as root ;; + non-root) skip_test_ must be run as non-root ;; + *) ;; # FIXME? + esac +} + +require_selinux_() +{ + case `ls -Zd .` in + '? .'|'unlabeled .') + skip_test_ "this system (or maybe just" \ + "the current file system) lacks SELinux support" + ;; + esac +} + +very_expensive_() +{ + if test "$RUN_VERY_EXPENSIVE_TESTS" != yes; then + skip_test_ ' +This test is very expensive, so it is disabled by default. +To run it anyway, rerun make check with the RUN_VERY_EXPENSIVE_TESTS +environment variable set to yes. E.g., + + env RUN_VERY_EXPENSIVE_TESTS=yes make check +' + fi +} + +require_root_() { uid_is_privileged_ || skip_test_ "must be run as root"; } +skip_if_root_() { uid_is_privileged_ && skip_test_ "must be run as non-root"; } +error_() { echo "$0: $@" 1>&2; (exit 1); exit 1; } +framework_failure() { error_ 'failure in testing framework'; } + +test_dir_=$(pwd) + +this_test_() { echo "./$0" | sed 's,.*/,,'; } +this_test=$(this_test_) + +# This is a stub function that is run upon trap (upon regular exit and +# interrupt). Override it with a per-test function, e.g., to unmount +# a partition, or to undo any other global state changes. +cleanup_() { :; } + +mktempd="$abs_top_srcdir/build-aux/mktempd" +t_=$("$SHELL" "$mktempd" "$test_dir_" lv-$this_test.XXXXXXXXXX) \ + || error_ "failed to create temporary directory in $test_dir_" + +# Run each test from within a temporary sub-directory named after the +# test itself, and arrange to remove it upon exception or normal exit. +trap 'st=$?; cleanup_; d='"$t_"'; + cd '"$test_dir_"' && chmod -R u+rwx "$d" && rm -rf "$d" && exit $st' 0 +trap '(exit $?); exit $?' 1 2 13 15 + +cd "$t_" || error_ "failed to cd to $t_" + +if ( diff --version < /dev/null 2>&1 | grep GNU ) 2>&1 > /dev/null; then + compare() { diff -u "$@"; } +elif ( cmp --version < /dev/null 2>&1 | grep GNU ) 2>&1 > /dev/null; then + compare() { cmp -s "$@"; } +else + compare() { cmp "$@"; } +fi + +# Local Variables: +# indent-tabs-mode: nil +# End: diff --git a/tests/vcpupin b/tests/vcpupin new file mode 100755 index 0000000..b56c7f2 --- /dev/null +++ b/tests/vcpupin @@ -0,0 +1,37 @@ +#!/bin/sh +# ensure that an invalid CPU spec elicits a diagnostic + +# Copyright (C) 2008 Free Software Foundation, Inc. + +# This program is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. + +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. + +# You should have received a copy of the GNU General Public License +# along with this program. If not, see <http://www.gnu.org/licenses/>. + +if test "$VERBOSE" = yes; then + set -x + virsh --version +fi + +. $srcdir/test-lib.sh + +fail=0 +virsh --connect test:///default vcpupin test a 0,1 > out 2>&1 +test $? = 1 || fail=1 + +cat <<\EOF > exp || fail=1 +error: vcpupin: Invalid or missing vCPU number. + +EOF + +compare out exp || fail=1 + +(exit $fail); exit $fail -- 1.5.4.4.482.g16f99

On Wed, Mar 19, 2008 at 03:27:30PM +0100, Jim Meyering wrote:
A week or so ago, I read a bug report on this list with a simple reproducer + fix that was just begging for a test suite addition.
This change adds framework to make that easy, and adds the first test. I uncovered problems in tests/Makefile.am along the way and just posted a patch for that.
This adds most of a portable (bourne-shell-based) test framework that also provides for convenient isolation of individual tests (so we can easily parallelize them without worrying one will tromp on files of another). In a couple years when running "make check" runs hundreds of tests, you'll definitely want them to run in parallel. The mktempd script is not new. I first used it in coreutils and parted, but removed it from coreutils after that package acquired its own C-based implementation.
test-lib.sh is from coreutils, but was inspired by the file by the same name in git.git's own t/ (tests) directory.
All seems reasonable to me. Dan. -- |: Red Hat, Engineering, Boston -o- http://people.redhat.com/berrange/ :| |: http://libvirt.org -o- http://virt-manager.org -o- http://ovirt.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: GnuPG: 7D3B9505 -o- F3C9 553F A1DA 4AC2 5648 23C1 B3DF F742 7D3B 9505 :|

On Thu, Mar 20, 2008 at 12:50:31PM +0000, Daniel P. Berrange wrote:
On Wed, Mar 19, 2008 at 03:27:30PM +0100, Jim Meyering wrote:
A week or so ago, I read a bug report on this list with a simple reproducer + fix that was just begging for a test suite addition.
This change adds framework to make that easy, and adds the first test. I uncovered problems in tests/Makefile.am along the way and just posted a patch for that.
This adds most of a portable (bourne-shell-based) test framework that also provides for convenient isolation of individual tests (so we can easily parallelize them without worrying one will tromp on files of another). In a couple years when running "make check" runs hundreds of tests, you'll definitely want them to run in parallel. The mktempd script is not new. I first used it in coreutils and parted, but removed it from coreutils after that package acquired its own C-based implementation.
test-lib.sh is from coreutils, but was inspired by the file by the same name in git.git's own t/ (tests) directory.
All seems reasonable to me.
For libvirt yes. For libxml2 on the opposite it proved useful to be able to aggregate regression tests as a few runnable binaries, first this made it like 10 times faster, but also allowed to run the regression tests on plaforms (MSC, VMS ...) where we could not rely on make and unix shell. For libvirt it's unclear the second point will ever apply, for the first point the main elapsed time seems taken by make check on DNS checks. But I don't have strong opinions one way or another honnestly, as long as make check doesn't take 10mn and the output FAIL/PASS is made obvious that's just fine by me. Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/
participants (3)
-
Daniel P. Berrange
-
Daniel Veillard
-
Jim Meyering