Test Case Markup & Cataloging, Iron Man edition

[David_Marston@lotus.com]

This document is at the "Iron Man" stage, which resembles a Candidate
Recommendation of the W3C. If this is approved at the next Committee
meeting, we will build a prototype of our system using this design, then
use the lessons learned to revise this into Steel Man. Upon approval, this
document should be published on the Committee's website, and Tin Man should
be relegated to the historical section.

Notable changes (from Tin Man):
1. This incorporates answers to the design questions we discussed earlier
this year, though some just reflect our current best understanding rather
than full resolution.
2. Contains data from the essentially-final Catalog of Discretion.
3. Contains data from a very tentative version of The Catalog of Vague, for
illustrative purposes.
4. Contains new verbiage concerning the Title, which names a contributed
tree of files.
5. OASIS can apply a date to submitted tests, if the submitter doesn't.
6. The <elaboration> can contain HTML markup. Nothing else can.
7. Errata dates rule, but errata numbers can also be in the catalog.
8. Operational parameters are divided into how-to-run and how-to-compare
elements.

Notable features, not necessarily changes:
1. I didn't name the element that encloses all the <test-case> elements. Is
<catalog> acceptable to everyone?
2. The <Source> element doesn't carry a filetype suffix, but <input-file>
and <output-file> may do so, unless someone knows a problem that would
occur.
.................David Marston

In this document, I describe information that should be associated with a
test case for (1) identification, (2) description and mapping to spec
provisions, (3) filtering (choosing whether or not to execute with a given
processor) and discretionary choices, and finally (4) some operational
parameters. I believe that it's fair to say that each test case is
represented by a stylesheet file, then use the operational parameters to
set up all inputs for the particular test case. The data described below
can be accumulated into a catalog of test cases, in XML of course, with one
<test-case> element for each case. However, good code management practices
would probably dictate that the creators of these cases retain the
definitive data in the primary stylesheet file. A catalog file can be
generated from the stylesheets. The catalog file would be the definitive
version as far as the OASIS package is concerned. That is, we expect the
submitter to provide a catalog and a file tree of test cases (including
allegedly-correct results), and to coordinate with OASIS on a "Title" for
the submission.

(I am not attempting to answer the question of how the data can be stored
in the XSL file, if the provider has chosen to do so. Lotus has chosen (so
far, anyway) to embed each item in a special comment, because the comments
do the least to perturb the test harness. With Xalan, it is possible to
retrieve the values from these comments and perform transformations on the
stylesheets to obtain data about the tests.)

Within the catalog, each test is represented as a <test-case> element with
numerous sub-elements. Most parameters would be interpreted as strings.
Values can be interpreted numerically, specifically in inequality
relations, when they refer to versions, dates, and the like. Excerpts of a
potential DTD are shown.

(1) IDENTIFICATION
Each group of tests is marked with a globally-unique "Title", which string
should also be valid as a directory name in all prominent operating
systems. The title can be suggested by the submitter, but must be approved
by OASIS. Thus, Lotus would submit a test suite called "Lotus" and the
OASIS procedures would load it into a "Lotus" directory (assuming that the
name "Lotus" is acceptable to the OASIS committee). The catalog as
submitted need not have the <Title>Lotus</Title> element in every entry
(thus it's shown as optional below), but it will be necessary in the merged
catalog that results from the "assembly" process. A submitted suite can
have arbitrary directory structure under its top-level directory, captured
in the "Identifier" element for each case, with forward slashes as the
directory delimiters. The actual name of the particular file (and test
case) would be in the "Source" element, which should be a valid file name
in all prominent operating systems. The Identifier contains the Source
string at the end, but not the Title at the beginning. Note that the test
suite may contain directories that have no test cases, only utility or
subsidiary files.

<!ELEMENT test-case ( Title? , Source , Identifier , Creator* , Date? ,
  purpose , elaboration? , spec-citation+ , discretionary? , gray-area? ,
  scenario ) >
<!-- Dublin Core ("DC") used for convenience/standardization where possible
for meta-data level of this DTD, here we replace FilePath with Identifier,
per http://purl.org/DC/documents/rec-dces-19990702.htm, "example formal
identification systems include the Uniform Resource Identifier (URI)
(including the Uniform Resource Locator (URL))."  Hereafter, quotes within
comments are from the URI above. -->

<!-- DC Title used in place of SuiteName, per "name by which the resource
is
  formally known". This must also meet filename constraints: letter first,
  no spaces, "reasonable" length -->
<!ELEMENT Title ( #PCDATA ) >
<!-- DC Source, per "best practice is to reference the resource by means of
a
  string or number conforming to a formal identification system," but must
  meet filename constraints and have no internal periods. This names a
  single test case. -->
<!ELEMENT Source ( #PCDATA ) >
<!-- Identifier uses forward slashes as separators, begins with the name of
a
  directory that is directly within the top directory named per Title, and
  ends with the name-part in Source. -->
<!ELEMENT Identifier ( #PCDATA ) >

OASIS may bless a particular hierarchical organization of test cases. If we
do, then an attribute called "category" should be used to track where the
test fits in OASIS' scheme of categories. That way, OASIS categories will
not dictate the directory structure nor the case names. The goal is that no
case should be marked as belonging to more than one category. We may need a
category named "Mixed" if we don't have a clean partitioning. The DTD
excerpt below shows a slight adaptation of Carmelo's proposed categories.

<!ATTLIST test-case
  category ( XSLT-Structure | XSLT-Data-Model | XSLT-Template
  | XSLT-Result-Tree | XSLT-Data-Manipulation | XSLT-Extendability
  | XSLT-Output | XPath-Location-Path | XPath-Expression
  | XPath-Core-Function | XPath-Data-Model | Mixed ) #IMPLIED >

Submitters should be encouraged to use the "Creator" element(s) to name
contributors at the individual-person level. They may also wish to use an
element called "Date" to record, as yyyy-mm-dd, the date stamp on the test
case. That will allow the submitter to match cases with their own source
code management systems, and will likely aid in future updates, either due
to submitter enhancements or W3C changes. OASIS reserves the right to
insert this element, containing the date received, if no value was supplied
by the submitter.

<!-- Dublin Core Creator instead of Author -->
<!ELEMENT Creator ( #PCDATA ) >
<!-- DC/ISO-8601 Date for the date of submission (from creator's POV) -->
<!ELEMENT Date ( #PCDATA ) >

(2) DESCRIPTION AND MAPPING TO SPEC PROVISIONS
Submitters must have a "purpose" element whose value describes the point of
the test. This string should be limited in length so that the document
generated by the OASIS tools doesn't ramble too extensively. (The new
length limit of 255 came from a 1-1 discussion with Carmelo and could be
subject to reduction later.) There would also be an optional "elaboration"
element whose length is unlimited and which may contain some HTML tags.
Nothing in this document should be construed as discouraging the use of
comments elsewhere in the stylesheet to clarify it.

<!ELEMENT purpose ( #PCDATA ) ><!-- Max 255 characters, no new-lines -->
<!ELEMENT elaboration ANY >

There must be one or more "spec-citation" elements to point at provisions
of the spec that are being tested. The pointing mechanism is the subject of
a separate discussion. The more exact it is, the less need there is for an
"elaboration" string, and also the better inversion from the spec to the
test cases. The spec-citation element contains a "Rec" attribute to say
which recommendation (XSLT, XPath, etc.), a "Version" sub-element to say
which version thereof, and some form of text pointer. To encourage
submissions before the pointer scheme is final, we may need to accept
alternative sub-elements of different names: <section> for a plain section
number, <doc-frag> for use of fragment identifiers that are already
available, and <OASISptr1> for the first OASIS pointer scheme, available
from the customized renditions of the specs on the Committee's "Resources"
page. OASIS pointers of types 2 and up may be necessary in the future,
hence the extendable design.

<!-- There must always be an XSLT spec-citation element, and optionally
other spec-citation elements can be added as appropriate -->
<!ELEMENT spec-citation ( place , Version , version-drop? , errata-add? ,
errata-drop? ) >
<!ATTLIST spec-citation
  Rec ( XSLT | XPath | XML-Stylesheet | XBase ) #REQUIRED >
<!-- Anticipate XML, XLink, Schema, Namespace, Internationalization, XSLFO,
XInclude in the future -->
<!ELEMENT place ( #PCDATA ) ><!-- syntax of content depends on Type -->
<!-- Type is a Dublin Core keyword -->
<!ATTLIST place Type ( section | doc-frag | OASISptr1 ) #REQUIRED >

(3) FILTERING AND DISCRETIONARY CHOICES
XSLT 2.0 is coming, so we need to anticipate it in our test case
organization, even if we're only trying to cover version 1.0 right now. In
addition to being tied to the XSLT spec, the cases rely on a particular
version of XPath and will soon also involve XBase. A few cases test
"Associating Style Sheets with XML Documents", or "XML-Stylesheet" as it is
popularly known. XML Internationalization or XInclude may also affect the
test suites. Each pertinent standard should be cited by version number, but
also flagged as to its errata status, if relevant. The version elements
mentioned above are numeric so that inequality tests may be applied. The
XSLT spec version should always be present, and should be set to 1.0 if the
test is really about XPath or some other associated spec. In other words,
any test that is essentially pure XPath should try to rely on XSLT 1.0 for
its XSLT portion if at all possible. All tests should use XML 1.0 if at all
possible, but again, we have noted the potential to specify a
higher-numbered version. Any test that is essentially about a newer spec,
such as XBase, should specify the lowest practical level of XSLT, which may
have to be higher than 1.0 if XSLT modifications are necessary for the
newer facility to work at all.

<!-- Version is another Dublin Core element; must be numeric -->
<!ELEMENT Version EMPTY >
<!ATTLIST Version number CDATA #REQUIRED >
<!-- version-drop, if specified, must be strictly greater (later) than
Version -->
<!ELEMENT version-drop EMPTY >
<!ATTLIST version-drop number CDATA #REQUIRED >

Errata are independent of newer spec versions, and multiple errata could be
issued per version. The flexible approach is to have a spec-citation
sub-element named "errata-add" that contains a numeric value (0 for the
base document) like the E-number in the XSLT errata; "errata-drop" is
numerically larger and indicates that the test case is no longer pertinent
as of that errata version. However, not all Working Groups are numbering
their errata, so there is some safety in using dates. Date is a required
attribute and should be in ISO-8601 format, which will sort numerically.
The add and drop levels would allow a test case to be marked as being
relevant for errata that later get further clarified. The errata-drop must
always be numerically greater than errata-add. Spec errata parameters need
only be specified where the test applies to a specific erratum, or the base
document only, because they are used for filtering.

<!-- errata-add and errata-drop should be rendered as dates with best
practice
  of ISO 8601, yyyy-mm-dd, W3CDTF. Errata numbers should be put in element
  content, if used at all. -->
<!ELEMENT errata-add ( #PCDATA ) >
<!ATTLIST errata-add Date CDATA #REQUIRED >
<!-- errata-drop, if specified, must be strictly greater (later) than
  errata-add -->
<!ELEMENT errata-drop ( #PCDATA ) >
<!ATTLIST errata-drop Date CDATA #REQUIRED >

We have cataloged discretionary choices available to the processor
developer, and these choices have names. These choices should be encoded in
elements which act as excluders when a test suite is assembled. By serving
as excluders, we eliminate the need to specify all of them in every test
case; if a discretionary item is not mentioned, the test case doesn't care
about that item and should be included for any choice made on that item.
The value can be expressed as a keyword from a set of keywords designated
by the Committee. For example, the <discretionary>
<attribute-name-not-QName> element contains a behavior attribute of either
"raise-error" or "ignore" to show that the case should be excluded when the
processor under test made the other choice on this item. Depending on the
choice, there could be parallel tests (differently named), with distinct
parallel "correct output" files, for different values of the choice, and
only one would be selected in any assembly of a test suite. The
questionnaire to developers about discretionary choices may allow "moot" as
a response in some situations, but one cannot use "moot" as a behavior
value in the test case catalog because, as stated above, moot items are
just omitted from the "discretionary" element.

<!ELEMENT discretionary ( unresolved-strip-preserve-conflict |
unresolved-template-rule-conflict |
  multiple-alias-same-precedence | element-name-not-QName |
attribute-name-not-QName | add-attribute-after-children |
  add-attribute-to-non-element | attribute-non-text-content |
two-attribute-set-same-attribute |
  PI-name-not-NCName-PItarget | PI-non-text-content |
PI-content-contains-delimiter | comment-non-text-content |
  comment-content-contains-delimiter | RTF-root-attribute-namespace-child |
add-namespace-to-non-element |
  add-namespace-after-children | number-not-positive |
unretrievable-resource | unprocessable-fragment-identifier |
  document-unresolvable-URI | two-output-same-attribute |
unsupported-encoding-error | unsupported-encoding-UTF-8 |
  unsupported-encoding-UTF-16 | default-encoding-UTF-8 |
default-encoding-UTF-16 | support-disable-output-escaping |
  unsupported-disabled-output-escaping | non-text-disabled-output-escaping
|
  unrepresented-character-disabled-output-escaping |
public-identifier-generate-URI |
  convert-number-English | sort-English | stylesheet-parameter |
result-tree-as-bytes | obey-xsl-output |
  converted-RTF-disabled-output-escaping )* >
<!ELEMENT unresolved-strip-preserve-conflict EMPTY >
<!ATTLIST unresolved-strip-preserve-conflict behavior ( choose-last |
raise-error ) #REQUIRED >
<!ELEMENT unresolved-template-rule-conflict EMPTY >
<!ATTLIST unresolved-template-rule-conflict behavior ( choose-last |
raise-error ) #REQUIRED >
<!ELEMENT multiple-alias-same-precedence EMPTY >
<!ATTLIST multiple-alias-same-precedence behavior ( choose-last |
raise-error ) #REQUIRED >
<!ELEMENT element-name-not-QName EMPTY >
<!ATTLIST element-name-not-QName behavior ( pass-through | raise-error )
#REQUIRED >
<!ELEMENT attribute-name-not-QName EMPTY >
<!ATTLIST attribute-name-not-QName behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT add-attribute-after-children EMPTY >
<!ATTLIST add-attribute-after-children behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT add-attribute-to-non-element EMPTY >
<!ATTLIST add-attribute-to-non-element behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT attribute-non-text-content EMPTY >
<!ATTLIST attribute-non-text-content behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT two-attribute-set-same-attribute EMPTY >
<!ATTLIST two-attribute-set-same-attribute behavior ( choose-last |
raise-error ) #REQUIRED >
<!ELEMENT PI-name-not-NCName-PItarget EMPTY >
<!ATTLIST PI-name-not-NCName-PItarget behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT PI-non-text-content EMPTY >
<!ATTLIST PI-non-text-content behavior ( ignore | raise-error ) #REQUIRED >
<!ELEMENT PI-content-contains-delimiter EMPTY >
<!ATTLIST PI-content-contains-delimiter behavior ( add-space | raise-error
) #REQUIRED >
<!ELEMENT comment-non-text-content EMPTY >
<!ATTLIST comment-non-text-content behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT comment-content-contains-delimiter EMPTY >
<!ATTLIST comment-content-contains-delimiter behavior ( add-space |
raise-error ) #REQUIRED >
<!ELEMENT RTF-root-attribute-namespace-child EMPTY >
<!ATTLIST RTF-root-attribute-namespace-child behavior ( ignore |
raise-error ) #REQUIRED >
<!ELEMENT add-namespace-to-non-element EMPTY >
<!ATTLIST add-namespace-to-non-element behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT add-namespace-after-children EMPTY >
<!ATTLIST add-namespace-after-children behavior ( ignore | raise-error )
#REQUIRED >
<!ELEMENT number-not-positive EMPTY >
<!ATTLIST number-not-positive behavior ( pass-through | raise-error )
#REQUIRED >
<!ELEMENT unretrievable-resource EMPTY >
<!ATTLIST unretrievable-resource behavior ( return-empty | raise-error )
#REQUIRED >
<!ELEMENT unprocessable-fragment-identifier EMPTY >
<!ATTLIST unprocessable-fragment-identifier behavior ( return-empty |
raise-error ) #REQUIRED >
<!ELEMENT document-unresolvable-URI EMPTY >
<!ATTLIST document-unresolvable-URI behavior ( return-empty | raise-error )
#REQUIRED >
<!ELEMENT two-output-same-attribute EMPTY >
<!ATTLIST two-output-same-attribute behavior ( choose-last | raise-error )
#REQUIRED >
<!ELEMENT unsupported-encoding-error EMPTY >
<!ATTLIST unsupported-encoding-error behavior ( yes | no ) #REQUIRED >
<!ELEMENT unsupported-encoding-UTF-8 EMPTY >
<!ATTLIST unsupported-encoding-UTF-8 behavior ( yes | no ) #REQUIRED >
<!ELEMENT unsupported-encoding-UTF-16 EMPTY >
<!ATTLIST unsupported-encoding-UTF-16 behavior ( yes | no ) #REQUIRED >
<!ELEMENT default-encoding-UTF-8 EMPTY >
<!ATTLIST default-encoding-UTF-8 behavior ( yes | no ) #REQUIRED >
<!ELEMENT default-encoding-UTF-16 EMPTY >
<!ATTLIST default-encoding-UTF-16 behavior ( yes | no ) #REQUIRED >
<!ELEMENT support-disable-output-escaping EMPTY >
<!ATTLIST support-disable-output-escaping behavior ( yes | no ) #REQUIRED >
<!ELEMENT unsupported-disabled-output-escaping EMPTY >
<!ATTLIST unsupported-disabled-output-escaping behavior ( ignore |
raise-error ) #REQUIRED >
<!ELEMENT non-text-disabled-output-escaping EMPTY >
<!ATTLIST non-text-disabled-output-escaping behavior ( ignore | raise-error
) #REQUIRED >
<!ELEMENT unrepresented-character-disabled-output-escaping EMPTY >
<!ATTLIST unrepresented-character-disabled-output-escaping behavior (
ignore | raise-error ) #REQUIRED >
<!ELEMENT public-identifier-generate-URI EMPTY >
<!ATTLIST public-identifier-generate-URI behavior ( yes | no ) #REQUIRED >
<!ELEMENT convert-number-English EMPTY >
<!ATTLIST convert-number-English behavior ( yes | no ) #REQUIRED >
<!ELEMENT sort-English EMPTY >
<!ATTLIST sort-English behavior ( yes | no ) #REQUIRED >
<!ELEMENT stylesheet-parameter EMPTY >
<!ATTLIST stylesheet-parameter behavior ( yes | no ) #REQUIRED >
<!ELEMENT result-tree-as-bytes EMPTY >
<!ATTLIST result-tree-as-bytes behavior ( yes | no ) #REQUIRED >
<!ELEMENT obey-xsl-output EMPTY >
<!ATTLIST obey-xsl-output behavior ( yes | no ) #REQUIRED >
<!ELEMENT converted-RTF-disabled-output-escaping EMPTY >
<!ATTLIST converted-RTF-disabled-output-escaping behavior ( ignore |
raise-error ) #REQUIRED >

Vague areas in the spec would be handled in the same manner as the
discretionary items above, with <gray-area> substituting for the
<discretionary> and the abbreviated names chosen from The Catalog of Vague.
This is where the errata level is likely to come in to play, since errata
should clear up some vague areas. Once again, the tester has to ask the
developer to answer questions about their design decisions, and the answers
should be encoded using keywords which can then be matched to the
<gray-area> elements. One test case could serve as both a gray-area for one
choice and as the lone case for errata-add, when that gray-area choice is
the one that the errata later chose.

<!ELEMENT gray-area ( RTF-disable-escaping | DTD-requires-XML |
element-available-no-namespace | format-number-without-format-declaration |
document-empty-nodeset | duplicate-passed-params |
duplicate-top-level-variables | sort-descending-case-order |
sort-number-NaN-position | number-not-positive | copy-namespace-node |
import-use-attribute-sets | negative-zero-multiply-div | starts-with-empty
| contains-empty | substring-before-empty | substring-after-empty |
ceiling-NaN | floor-NaN | attrib-set-not-exist | fallback-top-level |
key-unique-name | namespace-alias-intermediate | decimal-format-zero-digit
| decimal-format-bugs | message-disable-output-escaping |
element-prefixed-but-null-namespace )* >

<!ELEMENT copy-RTF-disable-escaping EMPTY >
<!ATTLIST copy-RTF-disable-escaping behavior ( disabled | enabled )
#REQUIRED >
<!ELEMENT doctype-system-requires-XML EMPTY >
<!ATTLIST doctype-system-requires-XML behavior ( yes | no ) #REQUIRED >
<!ELEMENT standalone-requires-XML EMPTY >
<!ATTLIST standalone-requires-XML behavior ( yes | no ) #REQUIRED >
<!ELEMENT element-available-no-namespace EMPTY >
<!ATTLIST element-available-no-namespace behavior ( default | none )
#REQUIRED >
<!ELEMENT format-number-without-format-declaration EMPTY >
<!ATTLIST format-number-without-format-declaration behavior ( default |
error ) #REQUIRED >
<!ELEMENT document-empty-nodeset EMPTY >
<!ATTLIST document-empty-nodeset behavior ( error | empty ) #REQUIRED >
<!ELEMENT duplicate-passed-params EMPTY >
<!ATTLIST duplicate-passed-params behavior ( error | choose-last )
#REQUIRED >
<!ELEMENT duplicate-top-level-variables EMPTY >
<!ATTLIST duplicate-top-level-variables behavior ( import-lesser |
import-equal ) #REQUIRED >
<!ELEMENT sort-descending-case-order EMPTY >
<!ATTLIST sort-descending-case-order behavior ( first | last ) #REQUIRED >
<!ELEMENT sort-number-NaN-position EMPTY >
<!ATTLIST sort-number-NaN-position behavior ( first | last | interspersed )
#REQUIRED >
<!ELEMENT number-not-positive EMPTY >
<!ATTLIST number-not-positive behavior ( error | string-conversion |
special-format ) #REQUIRED >
<!ELEMENT copy-namespace-node EMPTY >
<!ATTLIST copy-namespace-node behavior ( yes | no ) #REQUIRED >
<!ELEMENT import-use-attribute-sets EMPTY >
<!ATTLIST import-use-attribute-sets behavior ( expand-first | merge-first )
#REQUIRED >
<!ELEMENT negative-zero-multiply-div EMPTY >
<!ATTLIST negative-zero-multiply-div behavior ( negative | positive |
sign-rules ) #REQUIRED >
<!ELEMENT starts-with-empty EMPTY >
<!ATTLIST starts-with-empty behavior ( true | false ) #REQUIRED >
<!ELEMENT contains-empty EMPTY >
<!ATTLIST contains-empty behavior ( true | false |
true-except-when-both-empty ) #REQUIRED >
<!ELEMENT substring-before-empty EMPTY >
<!ATTLIST substring-before-empty behavior ( empty | entire-first )
#REQUIRED >
<!ELEMENT substring-after-empty EMPTY >
<!ATTLIST substring-after-empty behavior ( empty | entire-first ) #REQUIRED
>
<!ELEMENT ceiling-NaN EMPTY >
<!ATTLIST ceiling-NaN behavior ( NaN | other ) #REQUIRED >
<!ELEMENT ceiling-negative-fraction EMPTY >
<!ATTLIST ceiling-negative-fraction behavior ( negative | positive )
#REQUIRED >
<!ELEMENT floor-NaN EMPTY >
<!ATTLIST floor-NaN behavior ( NaN | other ) #REQUIRED >
<!ELEMENT attrib-set-not-exist EMPTY >
<!ATTLIST attrib-set-not-exist behavior ( error | ignore | discretionary )
#REQUIRED >
<!ELEMENT fallback-top-level EMPTY >
<!ATTLIST fallback-top-level behavior ( no | same-as-instruction |
not-same-as-instruction ) #REQUIRED >
<!ELEMENT key-unique-name EMPTY >
<!ATTLIST key-unique-name behavior ( error | merge ) #REQUIRED >
<!ELEMENT namespace-alias-intermediate EMPTY >
<!ATTLIST namespace-alias-intermediate behavior ( first | last ) #REQUIRED
>
<!ELEMENT decimal-format-zero-digit EMPTY >
<!ATTLIST decimal-format-zero-digit behavior ( zero | all ) #REQUIRED >
<!ELEMENT decimal-format-bugs EMPTY >
<!ATTLIST decimal-format-bugs behavior ( same | fix ) #REQUIRED >
<!ELEMENT message-disable-output-escaping EMPTY >
<!ATTLIST message-disable-output-escaping behavior ( yes | no ) #REQUIRED >
<!ELEMENT element-prefixed-but-null-namespace EMPTY >
<!ATTLIST element-prefixed-but-null-namespace behavior ( error |
local-part-only ) #REQUIRED >

(4) OPERATIONAL PARAMETERS
At Lotus, we have thought a lot about how comments in the test file can
describe the scenario under which the test is run, though we have not yet
implemented most of the ideas. These parameters describe inputs and
outputs, and a <scenario> element could describe the whole situation
through its "operation" and "compare" attributes. The "operation" value
describes how to run the test, while "compare" describes how to evaluate
the outcome. In the "standard" Operation scenarios, one XML file whose name
matches the XSL stylesheet file ("Source" element) is used as the input
document, and output is expected in a file that could then be suitably
compared to the "correct output" file. "Compare" options include "XML",
"HTML", and "Text", corresponding to the three methods of xsl:output and
the possible three methods of comparison. One or more <input-file> and
<output-file> elements could be used to specify other files needed or
created, and the values of these elements should permit relative paths. A
single input-file element could be used to specify that one of the
heavily-used standard input files should be retrieved instead of a
test-specific XML file. (Lotus has hundreds of tests where the XML input is
just a document-node-only trigger, and we would benefit from keeping one
such file in a Utility directory.) The implication of the latter rule is
that if there exists even one input-file element, no inputs are assumed and
all must be specified.

<!ELEMENT scenario ( input-file* , output-file* , param-set? , console ) >
<!ATTLIST scenario
  operation ( standard | embedded | external-param ) #REQUIRED
  compare ( XML | HTML | Text | message | message-XML | message-HTML |
message-Text |
  manual ) #REQUIRED >
<!ELEMENT input-file ( #PCDATA ) >
<!ELEMENT output-file ( #PCDATA ) >

If the operation keyword says "embedded", then the XSL stylesheet wasn't
really wanted, and the test should run as if the XML file sufficed. The
stylesheet file should probably do nothing and contain only comments.
Nevertheless, we may again want the processor developer to supply a
mechanism to set this up, since the way in which the stylesheet is marked
inapplicable will vary.

If the operation keyword says "external-param", then the processor should
be launched with parameters being set via whatever mechanism the processor
supports. We will probably push responsibility to the processor developer
to provide a script/batch mechanism to take values in a standardized way
and map them to the specific syntax of their processor. We would still need
to define a method, probably involving an extra input file (i.e.,
<Source>.ini) but possibly using more parameters in the test case, where
the test case can store the parameter names and values. RDF is probably
overkill for this usage. The most likely formats are:
(1) (type) name=value [new-line delimits?]
(2) a simple XML element with name and type attributes

<!-- This needs further design. Assume it designates an input file. -->
<!-- This value is only relevant when operation="external-param" -->
<!ELEMENT param-set ANY >

We also want to be able to test that a message was issued (as in
xsl:message) and that an error was issued. The "console" element will be
used to designate strings that must be present in either the standard
output or standard error stream. (The test lab would be responsible for
setting up capture of the console output.) The compare keyword "message"
can designate that, when running this test, capture the standard/error
output into a file, and ignore the normal transformation output. I suspect
that we need compare keywords like "message-HTML" to say that both the
console output and an HTML file must be compared. (For console output, the
test of correctness is to grep for the designated string in the captured
output stream.) If a tester wished, they could get actual error message
strings from the processor developer and refine the test harness to search
for those exact messages in error output. In that case, the string in the
console element is used as an indirect reference to the actual string.

<!-- should contain actual error report output string,
  or could be pointer to another file containing such strings.
  Less desirable: description of the problem. -->
<!ELEMENT console ( #PCDATA ) >

A compare value of "manual" would be used sparingly, for generate-id() and
system-property() output. Additional "scenario" keywords can be devised as
necessary, but OASIS should control the naming. We might want to allow
names beginning with a specific letter to be local to particular test labs.
For example, we would reserve all names beginning with "O-" and instruct
the test labs that they should put their name as the next field, then
another hyphen, then their local scenario keywords (e. g., O-NIST-whatever)
that allow them to set up local conditions (e.g., use of APIs) as needed.

HOW IT WORKS
When rendering a specific instance of the test suite, a test case can be
excluded on any one of the following bases:
A discretionary item of a given name is set to a different value.
A gray-area item of a given name is set to a different value.
The spec-citation/Version value on the test case is numerically larger than
what the processor implements. (This could be for any spec named, not just
XSLT.)
There is a spec-citation for a spec (e.g., XBase) that the processor claims
not to implement.
The test lab wishes to test against an errata level that is numerically
lower than the errata-add or higher than the errata-drop for a spec. In the
former case, there should be a cross-check against gray-area items.
Thus, it is the "user" (test lab) who renders a test suite by deciding
which spec version and errata level they wish to test, and by specifying
the settings of the discretionary and gray-area items they know. Before
running the specific rendition, they must ascertain how they will handle
those tests that run in external-param and possibly other scenarios, taking
into account the operating system where the tests will run and
processor-specific input and output design.

Note that the test suite itself is not filtered by scenario values. The
test lab may wish to devise a harness that can be configured to exclude
certain scenarios from some runs, but I think we want to encourage testing
and reporting against the full range of scenarios.

When a test case is included, it is run according to the values in the
<scenario> element. If inputs are specified, they are marshalled as
necessary. If no inputs are specified, a single file named <Source>.xml is
assumed to be the input and <Source>.xsl is assumed to be the stylesheet.
In some scenarios, special steps must be taken to capture the output. In
the standard scenarios, if no outputs are designated the name of the
intended output file is generated from the Source and from the
scenario/@compare value. (File names <Source>.xml, <Source>.html, and
<Source>.txt, and <Source>.msg are suggested.)