Test Case Markup, Straw Man edition

[David_Marston@lotus.com]

In this document, I describe information that should be embedded in the
primary file of a test case for (1) identification, (2) description and
mapping to spec provisions, (3) filtering (choosing whether or not to
execute with a given processor) and discretionary choices, and finally (4)
some operational parameters. I believe that it's fair to say that each test
case is represented by a stylesheet file, then use the operational
parameters to set up all inputs for the particular test case.

At this stage, I am not attempting to answer the question of how these can
be stored in the XSL file. Lotus has chosen (so far, anyway) to embed each
item in a special comment, because the comments do the least to perturb the
test harness. With Xalan, it is possible to retrieve the values from these
comments and perform transformations on the stylesheets to obtain data
about the tests. The other approach that I could see is for OASIS to
designate a namespace URI for this information, and the values to be stored
in top-level elements associated with that namespace. Any XSLT processor
under test would have to be very conformant about handling of "foreign"
top-level elements to be able to run the tests at all.

I will discuss the informational items as "parameters" that take on
"values", which are strings unless noted otherwise. In some cases, the
strings are limited to a set of keywords, and the committee would have to
prescribe the set of keywords. All parameter names are shown below with
spaces for readability, but actual names may be collapsed to non-spaced
strings with interior caps to signify separate words if we so decide.

(1) IDENTIFICATION
Following on discussions at our previous meetings, each submitter of a
group of tests should choose a globally-unique "Suite name", which string
should also be valid as a directory name in all prominent operating
systems. Thus, Lotus would submit a test suite called "Lotus" and the OASIS
procedures would load it into a "Lotus" directory. It could have arbitrary
directory structure under that top-level directory, which could be captured
in the "File path" parameter if needed, with forward slashes as the
directory delimiters. The actual name of the particular file (and test
case) would be in the "Test case name" parameter, which should be a valid
file name in all prominent operating systems. Question: should the file
path, if used, contain the file name? Also, note that the test suite may
contain directories that have no test cases, only utility or subsidiary
files.

OASIS may bless a particular hierarchical organization of test cases. If we
do, then a separate parameter called "Category" should be used to track
where the test fits in OASIS' scheme of categories. That way, OASIS
categories will not dictate the directory structure nor the case names.

Submitters should be encouraged to use the "Author" parameter to name
contributors at the individual-person level. They may also wish to use a
field called "Submission date" to record, as yyyymmdd, the date stamp on
the test case. That will allow the submitter to match cases with their own
source code management systems, and will likely aid in future updates,
either due to submitter enhancements or W3C changes.

(2) DESCRIPTION AND MAPPING TO SPEC PROVISIONS
Submitters should have a "Purpose" parameter whose value describes the
point of the test. Does anyone think that this string should be limited in
length? If so, we might also want an "Elaboration" parameter whose length
is unlimited. Only the "Purpose" string would be machine-extracted into
other places. Nothing in this document should be construed as discouraging
the use of comments elsewhere in the stylesheet to clarify it.

There should be one or more "Spec citation" parameters to point at
provisions of the spec that are being tested. The pointing mechanism is the
subject of a separate discussion. The more exact it is, the less need there
is for an "Elaboration" string, and also the better inversion from the spec
to the test cases. We may want to have a naming scheme that designates one
of the spec citations as primary (i.e., exactly one "Primary citation" and
zero or more "Spec citation" parameters), but I anticipate that such a
requirement will be more burdensome on test case writers than it's worth.

(3) FILTERING AND DISCRETIONARY CHOICES
The XSLT 1.1 effort is underway, so we need to anticipate it in our test
case organization, even if we're only trying to cover version 1.0 right
now. In addition to being tied to the XSLT spec, the cases rely on a
particular version of XPath and will soon also involve XBase. XML
Internationalization or XInclude may also affect the test suites. Each
pertinent standard should be cited by version number, but also flagged as
to its errata status, if relevant. The "Spec Version" parameters, one per
standard, are numeric so that inequality tests may be applied. The XSLT
spec version should always be present, and should be set to 1.0 if the test
is really about XPath or some other associated spec. In other words, any
test that is essentially pure XPath (or XBase, etc.) should try to rely on
XSLT 1.0 for its XSLT portion if at all possible. The naming scheme is
probably "SpecVersion-XSLT", "SpecVersion-XPath", etc. Errata are
independent of newer spec versions, and multiple errata could be issued per
version. One possible syntax is to have a parameter named "SpecErrata-XSLT"
that takes on a value like "1.0+0" (base 1.0 document), "1.0+1" (first
errata issued on XSLT 1.0), "1.0+2", etc. Spec errata parameters need only
be specified where the test applies to a specific erratum, or the base
document only. Do we need separate values for minimum and maximum errata
levels? This would allow a test case to be marked as being relevant for
errata that later get further clarified.

We have begun to catalog discretionary choices available to the processor
developer, and these choices have names. These choices should be encoded in
parameters which act as excluders when a test suite is assembled. By
serving as excluders, we eleiminate the need to specify all 39 or so in
every test case; if a discretionary item is not mentioned, the test case
doesn't care about that item and should be included for any choice made on
that item. I hope that in most cases, the value can be expressed as a
keyword from a set of keywords designated by the committee. For example,
the "Discretionary signal-comment-non-text-content" parameter takes on a
value of either "error" or "ignore" to show that the case should be
excluded when the processor under test made the other choice on this item.
Depending on the choice, there could be parallel tests, with distinct
parallel "correct output" files, for different values of the choice, and
only one would be selected in any assembly of a test suite. I have
considered the possibility that the value of the parameter specifies the
"correct output" file, but I think that it adds complication that I hope is
unnecessary.

Vague areas in the spec would be handled in the same manner as the
discretionary items above, with the word "Vague" (or gentler equivalent)
substituting for the word "Discretionary" and the abbreviated names chosen
from The Catalog of Vague. This is where the errata level is likely to come
in to play, since errata should clear up some vague areas. Once again, the
tester has to ask the developer to answer questions about their design
decisions, and the answers should be encoded using keywords which can then
be matched to the "Vague..." parameters.

(4) OPERATIONAL PARAMETERS
At Lotus, we have thought a lot about how comments in the test file can
describe the scenario under which the test is run, though we have not yet
implemented most of the ideas. These parameters describe inputs and
outputs, and a "Scenario" parameter could describe the whole situation
through its value, which is a keyword. In the "Standard" scenario, one XML
file whose name matches the XSL stylesheet file is used as the input
document, and output is expected in a file that could then be
binary-compared to the "correct output" file. One or more "Input file" and
"Output file" parameters could be used to specify other files needed or
created, and the values of these parameters should permit relative paths. A
single "Input file" parameter could be used to specify that one of the
heavily-used standard input files should be retrieved instead of a
test-specific XML file. (Lotus has hundreds of tests where the XML input is
just a document-node-only trigger, and we would benefit from keeping one
such file in a Utility directory.) The implication of the latter rule is
that if there exists even one "Input file" parameter, no inputs are assumed
and all must be specified.

If the "Scenario" keyword says "ExtParam", then the processor should be
launched with parameters being set via whatever mechanism the processor
supports. We may want to push responsibility to the processor developer to
provide a script/batch mechanism to take values in a standardized way and
map them to the specific syntax of their processor. We would still need to
define a method, probably involving an extra input file but possibly using
more parameters in the test case, where the test case can store the
parameter names and values.

If the "Scenario" keyword says "XMLEmbed", then the XSL stylesheet wasn't
really wanted, and the test should run as if the XML file sufficed. The
stylesheet file should probably do nothing and contain only comments.
Nevertheless, we may again want the processor developer to supply a
mechanism to set this up, since the way in which the stylesheet is marked
inapplicable will vary.

We also want to be able to test that a message was issued (as in
xsl:message) and that an error was issued. I propose that parameters
"Console standard output" and "Console error output" be used to designate
strings that must be present in the respective outputs. The "Scenario"
keyword "MatchStandardOutput" or "MatchErrorOutput" would be instruction
that says: when running this test, capture the standard/error output into a
file, and ignore the normal transformation output. The test of correctness
is to grep for the designated string in the captured output file. If a
tester wished, they could get actual error message strings from the
processor developer and refine the test harness to search for those exact
messages in error output.

Additional "Scenario" keywords can be devised as necessary, but OASIS
should control the naming. We might want to allow names beginning with a
specific letter to be local to particular test labs.