Re: [Libvirt-cim] KVM test report on Fedora 9
Kaitlin Rupert wrote:
> AllocationCapabilities - 02_alloccap_gi_errs.py: FAIL
> "Requested Object could not be found." vs. "Instance not found."
error
> string issue. The return code is correct.
> [Known Issue]
Dan has submitted a patch that sets the changeset number and revision
number of the providers as attributes in the VSMS class.
Could you add a function that gets these values, and then sets them as
globals for the suite? This will allow the individual test cases to use
the changeset/revision number as a way to check for different behavior
depending on when a particular patch went in.
Ok, I'll add that today.
Also, there was some discussion on the mailing list about modifying the
negative test cases so that they only check the provider return codes. I
think it'll be awhile before we can add implementation specific return
codes to the providers. Since the CIM return codes aren't specific
enough to indicate exactly what kind of error occurred, I'm inclined to
continue checking the return messages in the test cases for now.
Thoughts?
I agree with you on checking both the return codes and the messages. But
I thought branching the test cases for different changeset of providers
is a little risky. I am in the mood that we're going to maintain massive
if-else branches on this if the provider message strings change too fast.
An optimistic view would be that even though we need to maintain a
little bit too many of branches at first. But as the providers get more
stable, these frequent changes are less likely to happen.
Ok, my third view is a little unrealistic. We can develop a fifth test
case return code, named 'conditional pass', specifically for the rc
matches, string doesn't match issue. :=)
--
- Zhengang