RE: Revised Submission Policy

[Crisman Cooley <ccooley@overdomain.com>]

>> Revised Submission Policy
> David



> -----Original Message-----
> From: David_Marston@lotus.com [mailto:David_Marston@lotus.com]
> Sent: Monday, September 17, 2001 12:27 PM
> To: xslt-conformance@lists.oasis-open.org
> Subject: Re: Revised Submission Policy
>
>
> >1. Submitters' tests should test only a single citable requirement in
> >the Specification....
>
> I think this is still too limiting. Here's a way to turn it around:
> 1. Submitters' tests should test only a single atomic requirement in the
> Specification. In a comprehensive test suite, each testable assertion in
> the Specification should be tested independently of each other
> assertion, to the extent possible. Our experience shows that these
> assertions are often the result of taking one or two key sentences from
> the spec, but considering effects and conditions described at various
> other places in the spec. Citations have to point to portions of text
> that may not be as precise as testable assertions, while a "purpose"
> statement for the test can say exactly what is being singled out in
> that test case....
> If a test follows the first guideline above, one failure points out a
> singular instance where the processor does not conform to the
> Specifications (called "non-conformance" in this document). Failure of
> several cases may point out a single failure if one can readily identify
> the common element of all the failing cases. One non-conformance may
> cause failure of dozens of tests that involve various
> invocations of the non-conforming situation....

Sounds good, although I've been trying to avoid "atomic" and "molecular".
Can we use "simple" and "compound" instead?

>
> >2....If submitted, the Committee may not, run or include tests
> >involving multiple assertions of the Specification
> >composite issues, parser issues or errata of the Specification.
>
> Actually, we may only choose to exclude, under *this* provision, tests
> that expose problems in pre- or post-transformation stages. If there are
> errata pertaining to XSLT, we'll include the tests but ensure that they
> have the necessary errata/gray-area entries.
>

Okay.

> >4...The suite will differentiate test cases based on choices made by
> >the Submitter.  The Reviewers need to know if a test corresponds to a
> >particular choice made available to the processor....
>
> Better to say: This package includes a document called the Catalog of
> Discretion [filename here], which lists all identified allowances for
> differing behavior. These are situations where more than one outcome is
> defined as conformant by the specs. Any test case that depends on one
> particular choice in a discretionary area, or one choice in each of
> several areas, should have the discretionary choices cataloged. Use the
> exact words prescribed by the catalog....
>

Okay.

> >5. Later versions of the test suite may allow a wider range of tests.
>
> Interesting policy point: will we hold onto every case in a submission,
> so that those earning "postponed" status will automatically come back
> for review later, without an update from the Submitter? I say that we
> should do that, but nevertheless give each submitter contact person a
> courtesy notice (by email) that a subset of their submission is coming
> back for review, and ask if they wish to send a revised set.
> .................David Marston

I think I'll leave it as is for now, since this sounds like we need a group
consensus poll.

>
>
> ----------------------------------------------------------------
> To subscribe or unsubscribe from this elist use the subscription
> manager: <http://lists.oasis-open.org/ob/adm.pl>