[xslt-conformance] Re: Test Suite Issues

[David Marston/Cambridge/IBM <david_marston@us.ibm.com>]

>So, we have no "test harness", and we are not likely to produce one
>ourselves.

We discussed "what we could do" back in the early days of the TC, and
chose not to produce the tool that runs all the tests. Our guide for
the test lab needs to say what we provide and what they must get from
elsewhere. I'm sorry that the plan has not been well communicated to
the newer members.

I think it's wrong to say we have no test harness, since we provide a
way to filter the test cases. We are also providing significant parts
of the comparitor phase, though I can't say that I have personally
seen all the stuff that should be there. But it's true that we don't
have the tool that runs the set of tests, or the one that invokes
the canonicalizer/InfoSetizer over the set of tests.

>What do we have? We have tests.  We have validated the [xsl:number]
>tests we have, and we plan to validate all the other tests before we
>deliver them as a package.  This validation is clearly valuable.

Also, flagging discretionary items and gray areas is valuable.

>We also, reportedly, have a test catalog, although I haven't found it
>in the files I have.  This catalog would also be useful.

This catalog is in fact necessary, so that the test lab can generate
batch files or shell scripts (or whatever) that run the set of tests
in their environment. We also have the technology to validate XML
catalog files against the DTD and other constraints.

Regarding the Way Forward: I suggest that we be cautious about new
deliverables, especially those involving programming other than XSLT
stylesheets. Do we have enough talent available in that area? If we
continue with the original plan, we need more documentation talent to
present the usefulness of what we have. Either way, we don't want to
divert resources away from reviewing test cases.
.................David Marston