RE: Submission & Review Policies

[David_Marston@lotus.com]

>>>Recommendation citations are in the form of XPath expressions to
>>>testable statements in the [W3C specs]

Note that one test case will (or a least could) cite several testable
statements. A single testable statement is an atom but, as Cris has
already noted, we are testing some more akin to an assertion about
behavior in a particular case. As programmers know, sometimes you
must branch down several ifs to get to one case.

>I think maybe we should drop the metaphor of "atomic" vs. "molecular"

Sure. We don't need to take the side trip of defining all this for the
submitters.

>We are trying to encourage Submitters to give us tests that cover a
>single requirement, whether that requirement is simple or complex.

You could say that. Maybe "compound" rather than "complex" in the sense
that a testable assertion is compounded from several sentences.

>What we are trying to avoid, for Version 1.0, is a single test testing
>multiple requirements. Is this true?

That gets into murky territory. To maintain positive verbiage rather
than negative, think of this: we expect a "purpose" statement with each
case that says what is being tested in pithy form. Each test should
have one succinct purpose. It may "require" that other parts work, just
to set up the conditions, but the test itself is focused on one point.
There are potential dependencies on other testable assertions, but
there should be other tests whose point is one of the dependencies.

Example: A param that is set to the null string is still considered to
be set. To test, do a call-template and pass such a param, and then
use the select attribute on the xsl:param inside the called template to
set a default of a different value. Then, inside that template, check
whether the param is the null string (success) or the supplied default
(failure, um, non-conformance). Clearly, there are dependencies on
xsl:param, xsl:call-template, xsl:with-param, etc. but those are the
point of other tests and only dependencies in this test.

>> No, (3) needs to be revised to make clear that it refers to the
>> whole parser-XSLT-serializer chain, environment where run, etc.

The "must" provisions of the XSLT spec only define the behavior of a
processor that is given two parsed trees as input (data & stylesheet)
and produces a similar tree as output. Tests whose point is to reveal
mistakes on parsing the input or on serializing the output should be
excluded. OTOH, XSLT normatively incorporates XPath, so tests whose
point is to exercise some aspect of XPath are germane.

Re: "must" vs. "should" assertions
We want tests to flow in. Ideally, the submitter will make some effort
to filter out tests that they know we'll exclude, but the main goal is
that we get the submission. We want tests of...
mandatory provisions (must do X)
dilineated discretionary provisions (must do either A or B)
but not "should" provisions (should do Y, but doesn't prescribe anything
if you don't do Y). Once we get rolling, we can examine language/locale
issues and see what's feasible to include in later versions. If the
submitter doesn't understand the exactitude of the discretionary items,
we could encode that in the test-case catalog at review time.
...............David Marston