RE: Submission & Review Policies

[Cris Cooley <ccooley@overdomain.com>]

> > > = Cris
> > = Ken
> = David
  = Cris

**This email contains a concatenation of several messages from David & Ken**

> > >Guidelines for submission
> > >
> > >1. Submitters' tests should test only a single
> > "citable"
> > >requirement in the
> > >Specification.
> > >
> > >The Conformance Test Suite version 1.0 is designed so
> > >that failure of a single test identifies non-conformance to
> > >a single requirement in the Specification.
> >
> > "Recommendation citations are in the form of XPath expressions to
testable
> > statements in the XML working group source documents producing the HTML
W3C
> > documents."
> >
> > >Many of the guidelines below are designed to enforce this principle.

Is this principle accurately stated?  (See comments below under #2 that may
clarify...)

> > >
> > >2. The tests submitted for version 1.0 of the test suite
> > >should be "atomic" instead of "molecular".
> > >
> > >An atomic test covers a single specific issue.  Ideally, it
> > >should check just one assertion of the Specification. In a
> > >comprehensive test suite, each testable assertion in the
> > >Specification should be tested independently of each other
> > >assertion, to the extent possible.  If more than one
> > >assertion is tested at a time (a "molecular" test), the
> > >cause of a failure of the test may not easily indicate the
> > >nature of what is specifically wrong.  Failure in an atomic
> > >test will be much easier to identify and to resolve.
> > >
> > >Essentially in an atomic test one failure points out one
> > >bug.
>
> I think the draft of the Submission Policy gives way too negative a
> characterization to molecular tests. What we don't want are tests that
> have an uncoordinated assemblage of XSLT instructions. A true molecular
> test exercises the interaction of more than one testable item, when the
> spec makes it clear that those items interact. If we exclude these
> "good" molecular tests, passing our suite will prove little because so
> many bugs would be allowed through.
>
> Here's a good (and classic) example. Consider these two separate
> testable items from the XPath spec:
> [21]OrExpr ::= AndExpr | OrExpr 'or' AndExpr
> [22]AndExpr ::= EqualityExpr | AndExpr 'and' EqualityExpr
> The molecular test where you evaluate
> bool1 or bool2 and bool3
> given bool1 and bool2 true but bool3 false is a good conformance test.
> I say we want tests like that in our suite! (Correct answer: true)
>
> The good molecular tests can be said to cover a single issue, but the
> issue is a compound or contingent one: "What happens when you number by
> counting a given pattern AND the pattern has a union, as patterns are
> allowed to have?" One issue, but more than one assertion in the specs.
> Is that okay with the rest of the Committee?

I think maybe we should drop the metaphor of "atomic" vs. "molecular",
because it seems to me to confuse the issue (maybe because my feeble grasp
of quantum physics and chemistry does not allow me to apply the metaphor
accurately).  It seems to me from what you are saying David that
requirements in the Specification can be simple or complex.  We are trying
to encourage Submitters to give us tests that cover a single requirement,
whether that requirement is simple or complex.  What we are trying to avoid,
for Version 1.0, is a single test testing multiple requirements.  Is this
true?

> > >The converse is not true: one bug may cause failure of
> > >dozens of tests that involve various invocations of the bug
> > >situation. (Example: XT doesn't support xsl:key, so XT will
> > >be judged non-conformant in that respect. Other processors
> > >may implement keys quite well, but have a certain problem
> > >exposed in one case. That case may have to be "molecular"
> > >if several "atoms" must interact in a certain way to expose
> > >the bug.)

> > >3. The tests should target specific language issues, not
> > >"composite issues".
> > >
> > >The tests should be aimed at the language features and
> > >versions that are in scope for the Specification.
> > >"Composite issues" are those that would cause parser errors
> > >or that involve other W3C specifications that are out of
> > >scope for the current test suite.
> >
> > Is this perhaps a bit redundant?  All of 1, 2 and 3 seem to be
> repeating
> > the same principle.

> No, (3) needs to be revised to make clear that it refers to the
> whole parser-XSLT-serializer chain, environment where run, etc.
>

I do not know what you mean by "the whole parser-XSLT-serializer chain".
What we need is a precise definition of "composite issues"; or perhaps we
should throw out the term and replace it with what it is meant to define.
If "composite issues" are different than "molecular issues" (i.e., in my
redefinition above, multiple requirements in the Specification) then we need
to say what the difference is.  Can you clarify?

> >>4. The tests should target "must" provisions of the
> >>Specification, not "should" provisions....
> >I think we should encourage as many tests as possible and then
> >ascertain their applicability through our review policies.
>
> I agree with Ken.

Try out the language I suggest in my response to Ken.  I will also keep this
in mind as I do the next draft.

> > >13. The final test ID will be concatenation of Submitter
> > >and test IDs.
>
> >I don't think this is necessary from the point of view of a
> >submission policy document.
>
> We may want to inform the submitters that we ultimately need to
> produce a unique fully-qualified-name for every test case.

Okay, will do.