Re: [tag] Test Assertion Modeling - comments, etc

[david_marston@us.ibm.com]

Stephen Green writes:
>This kind of illustrates that not only must the document be valid
>according to the spec in its structure but maybe with regard to its
>semantics...
>So the app has to test user input according to test assertions, say,
>and a test suite might test the app according to the same or other
>test assertions perhaps and the test suites can be tested using test
>assertions too....
>This means test assertions of all kinds and aimed at all sorts of
>targets will be good as a basis for quality assurance of the
>standard spec but could lead to information overload and some shaky
>assumptions too among the standard architects perhaps. So moderation
>has to be exercised, one never knows all the uses in advance I guess.

In thinking about the Variability in a specification, the uppermost
(i.e., most independent) dimension is the Class of Product (CoP).
The WG or TC writing the spec should, as part of the Conformance
clause, identify the product or products they are trying to specify.
For HTML, the CoP could be documents, user agents, or both. Only the
CoP being specified would be subject to conformance testing. For
another example, the XSLT spec specifies conformant behavior for the
XSLT processor but not the stylesheets. Thus, only XSLT processors
are subject to conformance testing. Any testing of the validity of
the inputs (such as stylesheets) is conducted from the point of view
that a processor is required to behave in a certain way when given
an invalid input.

This could mean, by extension, that the TAs are assertions about the
characteristics of the CoP that is subject to testing. If so, then
the set of targets for the TAs is well constrained.

>...the test suites can be tested using test assertions too....

Probably. What is much more interesting is the possibility (someday)
of *generating* test materials from the TAs. TAs should be designed
now in a way that will allow such activities in the future.
.................David Marston