On Thu, Apr 10, 2008 at 08:59:27AM -0700, Dan Smith wrote:

GY> The processor id changed to domanin_name/proc, then does the GY> libvirt-cim rpm for Fedora 9 will reflect this recent changes? If GY> so, when is it time to replace? Also the same issues in terms of GY> MB versus KB for memory. I'm not sure if libvirt-cim rpm freezed, GY> then how cimtest are represent? Do we have to branch cimtest into GY> two parts, one for libvirt-cim rpm, the other for recent provider GY> changes?

This is a good point. The Fedora 9 RPM has been unchanged for a while now, so none of the changes that you mention are included. I think

Basically the Fedora-9 tree is frozen, but you should be able to push update for it after it has been released

that the test development should proceed to track the libvirt-cim development tree, but perhaps there is a way that we can map FAIL to

Well you could push dated release to rawhide but it can get tiring quickly Daniel -- Red Hat Virtualization group http://redhat.com/virtualization/ Daniel Veillard | virtualization library http://libvirt.org/ veillard@redhat.com | libxml GNOME XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Dan Smith wrote:

DV> Basically the Fedora-9 tree is frozen, but you should be able to DV> push update for it after it has been released

Right, which I definitely plan to do. I'd like the ability to look at a test report and see "X failures that have been fixed in the tree but are not fixed on the current platform". That will help indicate when a refresh to Fedora or a new release is a good idea. Might be a bit of a lofty goal, though :)

For future releases, would it be a good idea to archive the current state of the cimtest tree at the same time a release is snapped of libvirt-cim? This would allow us to preserve the state of the tests and tie them to a given release.

This is a good point. The Fedora 9 RPM has been unchanged for a while now, so none of the changes that you mention are included. I think that the test development should proceed to track the libvirt-cim development tree, but perhaps there is a way that we can map FAIL to XFAIL for older versions of the provider.

Does anyone have a suggestion for a reasonable way we could do this?

Have the test suite check for the libvirt-cim rpm. If it exists, grab version. Otherwise, assume the providers are the most recent version upstream. If a test case needs to be updated due to a provider behavior change, use an if condition based on the rpm version (or lack there of) to determine the pass/fail/xfail behavior. It's not a very elegant idea, but something along these lines might work. And of course, such an idea wouldn't work on environments that don't support rpm / that we currently don't build rpms for. -- Kaitlin Rupert IBM Linux Technology Center kaitlin@linux.vnet.ibm.com

Dan Smith wrote:

KR> For future releases, would it be a good idea to archive the KR> current state of the cimtest tree at the same time a release is KR> snapped of libvirt-cim? This would allow us to preserve the state KR> of the tests and tie them to a given release.

We can, and we could just use a tag for this. However, unless we get closer to an all PASS/XFAIL situation, I'm not sure I see the point :)

That's a good point.

KR> Have the test suite check for the libvirt-cim rpm. If it exists, KR> grab version. Otherwise, assume the providers are the most recent KR> version upstream.

Yeah, I dunno about that. We could definitely embed a version string into one of the providers somewhere (i.e. VSMS) and key off of that. We'd have to assume some base version if it doesn't exist (like for what is in F9 now) of course.

When would you update the version string? Anytime a change goes into libvirt-cim, there's a possibility that the change in behavior will affect the test case. If you want to update a test so that it behaves one way for libvirt releases older than x.x.4 and a different way for releases x.x.4 and higher, then you'd need to update the embedded string every time you check in a changeset. Unless I'm missing what you mean here. Ideally, it'd be nice to use the hg changeset number, but I'm not sure how to make that happen. -- Kaitlin Rupert IBM Linux Technology Center kaitlin@linux.vnet.ibm.com