[xslt-conformance] OASIS XSLT/XPath Conformance Committee -- TestSuite Issues

[Jeff Kenton <jkenton@datapower.com>]

Greetings,

We have issues with our test suite that I didn't expect.

Since I volunteered to "package up" the prototype test suite, I assigned one of my people here at DataPower to take the test harness, add the Number-formatting tests to it, and see what he had.

He discovered that the "test harness" contained only pseudo-code, vague instructions on how to build your own test harness, and a collection of xslt stylesheets that might be helpful.  He said that if we wanted a test harness, we would need to write our own from scratch.  I am honestly flabbergasted.  I had been led to believe that the test harness was 99% complete, and only needed to have tests added to it.

So, we have no "test harness", and we are not likely to produce one ourselves.  What do we have? We have tests.  We have validated the Number-formatting tests we have, and we plan to validate all the other tests before we deliver them as a package.  This validation is clearly valuable.  We also, reportedly, have a test catalog, although I haven't found it in the files I have.  This catalog would also be useful.

The major question now is, is a package of tests alone valuable to the XSLT/XPath community? Can we, in good faith, deliver a test suite without a test harness to go with it?

There are several ways to answer this question.  First, this committee represents various constituencies -- what do people really need? Second, users might be completely confortable taking the tests alone and using their own tools for testing.  I did that when we were testing the Number-formatting tests, and was perfectly happy doing it.  Third, there is a publicly available test harness from DataPower.  Dave Marston (and maybe others) doesn't like the XML normalizer that comes with it, but we could easily plug in a different one if we want to.

So, we have a problem, and we have several ways we could solve it.  Comments and suggestions are welcome.

--jeff

-- 
--------------------------
Jeff Kenton
DataPower Technology, Inc.