Re: Requirements

[David_Marston@lotus.com]

Here are the requirements that I see as being strictly for human use:
1) Which specifications is this test about, XSLT or XPATH?
2) Which section or area of the specification this test is addressing.
3) The category of this test (if any).
7.5) Description of the tests.
9) Any known problems with any of the tests.
11) How do I report problems with any of the tests?
12)  Are there any holes in the test suite? (any areas that
           are not addressed by any test).
14) Who wrote this test?
17) Is there a test harness available?
Of these, I'd say 17, 12, 11, and maybe 9 apply at the level of the
whole suite, and the rest apply case-by-case. The Iron Man catalog
has all the above case-by-case data.

Here are the requirements that I see as having some likelihood of
being data subject to machine processing:
    1) Which specifications is this test about, XSLT or XPATH?
    2) Which section or area of the specification this test is addressing.
The above are only if we go ahead with our plans to produce copies of
the W3C Recommendations enhanced with pointers to test cases. That's
looking sorta dubious at this time.
    4) The test name.
    5) The nature of this test (positive, negative, implementation-defined,
etc) .
    6) The expected output to this test (if any).
    7) How do I run the tests?
The above is the scenario.
    8) Any secondary files needed by the tests.
    10) How do I interpret the results? (are they self-explanatory?).
The above is limited to setting up the compare against the reference
output. If it doesn't match, further interpretation is by a human.
    13) When was this test written?
The above is only used in confirming version and errata levels, and when
installing an updated test suite.
    15) Any test that depends on another test.
    16) Any test output that may be an input to another test.
If we use only atomic tests, the above two won't be necessary.
    18) Which information do I need to provide to any of the tests?
The above could be said to describe the setting of external parameters.
    19) Any optional tests?, non-applicable tests?
No tests should be optional. (We *are* talking about a conformance
suite, after all.) There is data by which a lab can filter out tests
that use a scenario/operation they haven't implemented, but we should
encourage complete runs. The non-applicable test filtering is
summarized at the bottom of the Iron Man document.

Additional requirement: We already have data to specify which version
of the spec a test is testing, but we also need to provide some
version data for the test suite itself, so that test labs can report
which version they're using, and also to facilitate updates.
.................David Marston