Re: [libvirt] [test-API] RFC: Stabilization of libvirt-test-API
On 04/04/2012 11:09 PM, Laine Stump wrote:
On 03/29/2012 08:14 AM, Martin Kletzander wrote:
> So here are the things I would like to do definitely (the optional
> things follow later on):
> - fix hard-coded options into real options (e.g. commit 65449e)
Forgive me if I'm suggesting things that already exist - I haven't had
time to go through all of the test-API code, but want to make sure this
gets brought up sooner rather than later.
Two things that I think are very important for a general purpose test
setup are:
1) The ability to have multiple physical machines, network devices of
various types (bridges, standard NICs, sr-iov NICs) in the test harness
available via standard names (implying, of course, that the
address/capabilities (and even the existence/non-existence) of at least
one other machine be configurable in a global config file referenced by
every test).
The env.cfg is the global config file that lists the
default data needed by testcases.
we could expand it for various requirements. The testsuit
neither manage these devices
or machines nor check if it is spelling-correct or present
on testing machine right now.
because it depends on testing environment , the tester
should check it before writing it in
env.cfg.
<snip>
##########################
#
# PCI device
# a PIC device to use for attach/detach/reset tests
# for example testpic = 00:19.0
testpic =
</snip>
2) The ability to mark some tests as requiring certain standard objects
from the global configuration (e.g. the second machine, a bridge
interface, sr-iov NICs) and to either skip the test when some required
object is missing, or fail the test (the test itself would be marked
either OPTIONAL or REQUIRED).
Currently, there is a file named BUGSKIP in the root of test-API
If certain testcase is written in it for a bug related
reason, the testcase
will be skipped during running time, we could improve it for
your purpose.
There are 70~80 testcases. Any number of this testcases
could be combined
in different order according to various testing
requirements. So there is not one
shaped running model to go through all of testcases(we could
have a basic one).
there are some testcase file that are mostly used in "cases"
folder in the root directory.
This would allow us to have, for example, full testing of networking
capabilities present in the test suite, but without raising an entry
barrier for "random Joe" who only has a single machine, one NIC, etc.,
but wants to run the test suite. When Joe ran the tests, each test that
required multiple hosts (or sr-iov NICs or whatever esoteric piece of
hardware/driver) and was designated as an "OPTIONAL" test, would be
semi-silently skipped, but someone with a full complement of hardware
who demanded a thorough and complete test could set a switch and
guarantee that every single test would be run (or a failure saying, e.g.
"object REMOTE_HOST required by this test is missing in config", or
something like that).
The current data list in env.cfg is default and
satisfied for
all of testcases. "random Joe" have to know which testcases he
want to
run, if he need the testcase which rely on two NICs , he need to
write MAC address of NICs to env.cfg before testing.
Is there currently an allowance for these two items?
--
libvir-list mailing list
libvir-list(a)redhat.com
https://www.redhat.com/mailman/listinfo/libvir-list