Running tests (was: infoset based comparison)

[David_Marston@lotus.com]

Kirill wrote:
>class testXSLT {
>     //iterates through test-suite
>     string getNextTestSuite(string testSuite)
>     string getNextTestID(string testSuite, string prevID);

The above doesn't seem to be dealing with the fully-merged and
filtered test suite. We are getting near the point where we
should design it. Wouldn't the rendered version be a single
<test-catalog> element encompassing the <test-case> elements
from all contributed suites?

>     //returns test xsmlt and input...
>     bool getInput(...)

Within each test-case, we must iterate over all the <input-file>
elements, at least to verify their presence. If there are no such
elements, construct the default inputs from the <Source> element
by applying some name-construction rules.

Next, you set up to run the test case and capture output per the
"operation" and "compare" attributes. This is the place where the
Test Lab has to take at least some responsibility for adjusting
the task to their platform.

>          //verifies that testOutput is correct and returns it's
>    infoset and expected infoset -
>     //testing person can verify whether it's parser's bug...
>     bool test(string id, binaryBase64 testOutput, string
>    iTestOutput, string iTestExpected);
>    }

If "compare" says capture and compare console output, then it's a
grepping operation rather than comparing infosets. Also keep in mind
that there might be several output files to be compared.
.................David Marston