Re: Revised Submission Policy

[David_Marston@lotus.com]

>1. Submitters' tests should test only a single citable requirement in
>the Specification....

I think this is still too limiting. Here's a way to turn it around:
1. Submitters' tests should test only a single atomic requirement in the
Specification. In a comprehensive test suite, each testable assertion in
the Specification should be tested independently of each other
assertion, to the extent possible. Our experience shows that these
assertions are often the result of taking one or two key sentences from
the spec, but considering effects and conditions described at various
other places in the spec. Citations have to point to portions of text
that may not be as precise as testable assertions, while a "purpose"
statement for the test can say exactly what is being singled out in
that test case....
If a test follows the first guideline above, one failure points out a
singular instance where the processor does not conform to the
Specifications (called "non-conformance" in this document). Failure of
several cases may point out a single failure if one can readily identify
the common element of all the failing cases. One non-conformance may
cause failure of dozens of tests that involve various
invocations of the non-conforming situation....

>2....If submitted, the Committee may not, run or include tests
>involving multiple assertions of the Specification
>composite issues, parser issues or errata of the Specification.

Actually, we may only choose to exclude, under *this* provision, tests
that expose problems in pre- or post-transformation stages. If there are
errata pertaining to XSLT, we'll include the tests but ensure that they
have the necessary errata/gray-area entries.

>4...The suite will differentiate test cases based on choices made by
>the Submitter.  The Reviewers need to know if a test corresponds to a
>particular choice made available to the processor....

Better to say: This package includes a document called the Catalog of
Discretion [filename here], which lists all identified allowances for
differing behavior. These are situations where more than one outcome is
defined as conformant by the specs. Any test case that depends on one
particular choice in a discretionary area, or one choice in each of
several areas, should have the discretionary choices cataloged. Use the
exact words prescribed by the catalog....

>5. Later versions of the test suite may allow a wider range of tests.

Interesting policy point: will we hold onto every case in a submission,
so that those earning "postponed" status will automatically come back
for review later, without an update from the Submitter? I say that we
should do that, but nevertheless give each submitter contact person a
courtesy notice (by email) that a subset of their submission is coming
back for review, and ask if they wish to send a revised set.
.................David Marston