Re: Summary of where we stand on catalog data

[David_Marston@lotus.com]

In the preceding message, I wrote:
>C. script to check that all the input files needed by a test case
>   exist in the correct directories (notable details to come)

The notable detail I had in mind is the "cd issue" that I promised
to write about. This arises because the principal stylesheet can
refer to files in other roles....
supplemental-data: document('filename')
supplemental-stylesheet: xsl:import or xsl:include files
These files may have relative paths, not just names. Thus, the test
lab needs to know that the current directory should be the one
containing the stylesheet, and we have to convey that to test case
submitters. The supplemental (formerly "secondary") inputs can be
tested for existence relative to that directory.

The test lab uses catalog data to generate a script with all the
necessary "cd" (or equivalent) commands intermixed among the
commands to invoke the tests. For example, they could produce a
batch file containing (sorry if you see line wraps):
...
cd tests\Lotus\numbering
xslt -in numb01.xml -xsl numb01.xsl -out
results\Lotus\numbering\numbering01.xml
xslt -in ..\util\u1.xml -xsl numb02.xsl -out
results\Lotus\numbering\numbering02.xml
xslt -in numb03.xml -xsl numb03.xsl -out
results\Lotus\numbering\numbering03.html
xslt -in numb04.xml -xsl numb04.xsl -out
results\Lotus\numbering\numbering04.html
...
xslt -in ..\util\u1.xml -xsl numb91.xsl -out
results\Lotus\numbering\numbering91.xml
cd tests\Microsoft\num
xslt -in ..\stdata\books.xml -xsl XSLT03005.xsl -out results\Microsoft\num\
77489.xml
...
where "xslt" is a subsidiary batch file that has the front part of
the command-line invocation of the processor under test, and "results"
may be a temporary repository of results of the test run or it may be
a longer path that identifies the processor and date of testing. NOTICE
that the XML data, even though it is a "principal" input, may be pathed
to refer to a standard data file shared among many tests in many groups.
In such instances, the input-file element for principal-data contains
a relative path rather than just the name.

Tests for the other operational scenarios need different command lines,
but a similar principle applies.

In the above batch file example, I'm suggesting that the destination
(-out path) was generated from catalog data, including the submitter
and file-path (always concatenated) and the name designated in the
<output-file role="principal" ...> element. It is prepended with
whatever "results" string the lab wants, though they will be coached
to keep results out of the read-only file tree.

Speaking of trees, our deliverable is shaping up like this:
OASIS-XSLT-test/    [our top directory, name to be determined]
    README, etc. files
    tests/          [all the test cases]
        Lotus/
        Microsoft/
        NIST/
        .../
    ref-output-raw/        [all the raw output as submitted]
        Lotus/
        Microsoft/
        NIST/
        .../
    ref-output-infoset/    [InfoSetized reference output]
        Lotus/
        Microsoft/
        NIST/
        .../
and one of the first things the test lab would do with our suite is
run the canonicalizer to populate a ref-output-canonical directory
tree parallel to the others. From that point on, they can treat the
whole OASIS-XSLT-test directory tree as read-only by arranging to put
their test-run outputs and logs elsewhere.
.................David Marston