OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

xslt-conformance message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Subject: Test Case Markup & Cataloging, Tin Man edition

In this document, I describe information that should be associated with a
test case for (1) identification, (2) description and mapping to spec
provisions, (3) filtering (choosing whether or not to execute with a given
processor) and discretionary choices, and finally (4) some operational
parameters. I believe that it's fair to say that each test case is
represented by a stylesheet file, then use the operational parameters to
set up all inputs for the particular test case. The data described below
can be accumulated into a catalog of test cases, in XML of course, with one
<TestCase> element for each case. However, good code management practices
would probably dictate that the creators of these cases retain the
definitive data in the primary stylesheet file. A catalog file can be
generated from the stylesheets. The catalog file would be the definitive
version as far as the OASIS package is concerned.

DESIGN DECISIONS--Be prepared to discuss these at our next meeting:
1. How to represent element names: "TestCase" vs. "Test-Case" vs.
"test-case" (likewise for keywords)
2. How to represent the file hierarchy in Title, Identifier, Source
3. Whether to have categories, and if so, what the categories are
4. What limits to impose on "Purpose" to make it a one-liner
5. Is it okay to refer to versions by number but errata by date? (Note:
dates are yyyymmdd)
6. Is there any operational problem with having all discretionary items
lumped under one <Discretionary> element?
7. Shall we define every choice for every discretionary item separately in
the DTD or pool the choices?
8. What status do we attach to GrayArea items?
9. Do we want to collapse standard output and standard error together?
10. Extent to which we specify how to set external parameters, when
11. How do we accommodate operational scenarios that are local to the test

(I am not attempting to answer the question of how the data can be stored
in the XSL file, if the provider has chosen to do so. Lotus has chosen (so
far, anyway) to embed each item in a special comment, because the comments
do the least to perturb the test harness. With Xalan, it is possible to
retrieve the values from these comments and perform transformations on the
stylesheets to obtain data about the tests. The other approach that I could
see is for OASIS to designate a namespace URI for this information, and the
values to be stored in top-level elements associated with that namespace.
Any XSLT processor under test would have to be very conformant about
handling of "foreign" top-level elements to be able to run the tests at
all. At our 9/6/2000 meeting, we agreed to research whether OASIS has a
policy on assigning namespaces under their domain name.)

Within the catalog, each test is represented as a <TestCase> element with
numerous sub-elements. Most parameters would be interpreted as strings.
Values can be interpreted numerically, specifically in inequality
relations, when they refer to versions, dates, and the like. JR and I
discussed a DTD rendition, and you will see excerpts interleaved below.

Each submitter of a group of tests should choose a globally-unique "Title"
(formerly "SuiteName"), which string should also be valid as a directory
name in all prominent operating systems. Thus, Lotus would submit a test
suite called "Lotus" and the OASIS procedures would load it into a "Lotus"
directory. A submitted suite can have arbitrary directory structure under
that top-level directory, captured in the "Identifier (formerly "FilePath")
element for each case, with forward slashes as the directory delimiters.
The actual name of the particular file (and test case) would be in the
"source" (formerly "CaseName") element, which should be a valid file name
in all prominent operating systems. The Identifier probably contains the
full path between the Title and Source, inclusive. Note that the test suite
may contain directories that have no test cases, only utility or subsidiary

<!ELEMENT TestCase ( Identifier , Title? , Source , Creator* , Date? ,
Purpose ,
  Elaboration? , SpecCitation+ , Discretionary? , GrayArea? , Scenario ) >
<!-- Dublin Core used for convenience/standardization where possible for
meta-data level of this DTD, here we replace FilePath with Identifier, per
http://purl.org/DC/documents/rec-dces-19990702.htm, "xample formal
identification systems include the Uniform Resource Identifier (URI)
(including the Uniform Resource Locator (URL)."  Hereafter, quotes within
comments are from the URI above. -->

<!ELEMENT Identifier ( #PCDATA ) >
<!-- DC Title used in place of SuiteName, per "name by which the resource
is formally known" -->
<!ELEMENT Title ( #PCDATA ) >
<!-- was CaseName, now DC Source, per "best practice is to reference the
resource by means of a string or number conforming to a formal
identification system," but must meet filename constraints -->
<!ELEMENT Source ( #PCDATA ) >

OASIS may bless a particular hierarchical organization of test cases. If we
do, then an attribute called "Category" should be used to track where the
test fits in OASIS' scheme of categories. That way, OASIS categories will
not dictate the directory structure nor the case names. At the 9//6/2000
meeting, a question was raised about whether a case might occupy more than
one category. If the purpose is clear, probably not, but we need to assess
the uses to which the categories will be put. We may need a category named
"Mixed" if we don't have a clean partitioning. The DTD excerpt below shows
a slight adaptation of Carmelo's proposed categories.

<!ATTLIST TestCase
  Category ( XSLT-Structure | XSLT-Data-Model | XSLT-Template |
XSLT-Result-Tree | XSLT-Data-Manipulation
  | XSLT-Extendability | XSLT-Output | XPath-Location-Path |
XPath-Expression | XPath-Core-Function
  | XPath-Data-Model | Mixed ) #REQUIRED >

Submitters should be encouraged to use the "Creator" (formerly "Author")
element to name contributors at the individual-person level. They may also
wish to use an element called "Date" to record, as yyyymmdd, the date stamp
on the test case. That will allow the submitter to match cases with their
own source code management systems, and will likely aid in future updates,
either due to submitter enhancements or W3C changes.

<!-- Dublin Core Creator instead of Author -->
<!ELEMENT Creator ( #PCDATA ) >
<!-- DC Date for what we apply here as the date of submission (from
creator's POV) -->
<!ELEMENT Date ( #PCDATA ) >

Submitters should have a "Purpose" element whose value describes the point
of the test. This string should be limited in length so that the document
generated by the OASIS tools doesn't ramble too extensively. There would
also be an optional "Elaboration" element whose length is unlimited.
Nothing in this document should be construed as discouraging the use of
comments elsewhere in the stylesheet to clarify it.

<!ELEMENT Purpose ( #PCDATA ) ><!-- Max 72 characters, no new-lines -->
<!ELEMENT Elaboration ( #PCDATA ) >

There should be one or more "SpecCitation" elements to point at provisions
of the spec that are being tested. The pointing mechanism is the subject of
a separate discussion. The more exact it is, the less need there is for an
"Elaboration" string, and also the better inversion from the spec to the
test cases. The SpecCitation element contains a "Rec" attribute to say
which recommendation (XSLT, XPath, etc.), a "Version" sub-element to say
which version thereof, and some form of text pointer. To encourage
submissions before the pointer scheme is final, we may need to accept
alternative sub-elements of different names: <Section> for a plain section
number, <DocFrag> for use of fragment identifiers that are already
available, and <OASISptr1> for the first pointer scheme we choose. OASIS
pointers of types 2 and up may be necessary in the future, hence the
extendable design.

<!-- There must always be an XSLT SpecCitation element, and optionally
other SpecCitation elements can be added as appropriate -->
<!ELEMENT SpecCitation ( Place , Version , VersionDrop? , ErrataAdd? ,
ErrataDrop? ) >
<!ATTLIST SpecCitation
  Rec ( XSLT | XPath | XBase | XInclude ) #REQUIRED >
<!-- Anticipate XLink, Schema, Namespace, Internationalization in the
future -->
<!ELEMENT Place ( #PCDATA ) >
<!ATTLIST Place type ( Section | DocFrag | OASISptr1 ) #REQUIRED >

The XSLT 1.1 effort is underway, so we need to anticipate it in our test
case organization, even if we're only trying to cover version 1.0 right
now. In addition to being tied to the XSLT spec, the cases rely on a
particular version of XPath and will soon also involve XBase. XML
Internationalization or XInclude may also affect the test suites. Each
pertinent standard should be cited by version number, but also flagged as
to its errata status, if relevant. The Version elements mentioned above are
numeric so that inequality tests may be applied. The XSLT spec version
should always be present, and should be set to 1.0 if the test is really
about XPath or some other associated spec. In other words, any test that is
essentially pure XPath should try to rely on XSLT 1.0 for its XSLT portion
if at all possible. Any test that is essentially about a newer spec, such
as XBase, should specify the lowest practical level of XSLT, which may have
to be higher than 1.0 if XSLT modifications are necessary for the newer
facility to work at all.

<!-- Version/VersionDrop is to be numbered or dated, suggest best practice
dated with ISO 8601 -->
<!-- VersionDrop, if specified, must be strictly greater (later) than
Version -->
<!ELEMENT Version ( #PCDATA ) >
<!ATTLIST Version number CDATA #REQUIRED >
<!ELEMENT VersionDrop ( #PCDATA ) >
<!ATTLIST VersionDrop number CDATA #REQUIRED >

Errata are independent of newer spec versions, and multiple errata could be
issued per version. The flexible approach is to have a SpecCitation
sub-element named "ErrataAdd" that takes on a numeric value like 0 (base
document), 1 (first errata issued on the indicated version of the indicated
spec), 2, etc. "ErrataDrop" ranges from 1 upward and indicates that the
test case is no longer pertinent as of that errata version. However, W3C is
not numbering their errata, so there is some safety in using dates. If we
use dates, they should be in ISO-8601 format, which will sort numerically.
The Add and Drop levels would allow a test case to be marked as being
relevant for errata that later get further clarified. ErrataDrop must
always be numerically greater than ErrataAdd. Spec errata parameters need
only be specified where the test applies to a specific erratum, or the base
document only, because they are used for filtering.

<!-- ErrataAdd and ErrataDrop should be rendered as dates with best
practice of ISO 8601, yyyy-mm-dd, W3CDTF -->
<!-- ErrataDrop, if specified, must be strictly greater (later) than
ErrataAdd -->
<!ELEMENT ErrataAdd ( #PCDATA ) >
<!ELEMENT ErrataDrop ( #PCDATA ) >

We have begun to catalog discretionary choices available to the processor
developer, and these choices have names. These choices should be encoded in
elements which act as excluders when a test suite is assembled. By serving
as excluders, we eliminate the need to specify all 50+ in every test case;
if a discretionary item is not mentioned, the test case doesn't care about
that item and should be included for any choice made on that item. I hope
that in most cases, the value can be expressed as a keyword from a set of
keywords designated by the committee. For example, the <Discretionary>
<signal-comment-non-text-content> element contains a Behavior attribute of
either "RaiseError" or "Ignore" to show that the case should be excluded
when the processor under test made the other choice on this item. Depending
on the choice, there could be parallel tests (differently named), with
distinct parallel "correct output" files, for different values of the
choice, and only one would be selected in any assembly of a test suite.

<!ELEMENT Discretionary ( DiscretionaryItem+ ) >
<!-- DiscretionaryItem is a placeholder for what will be some 50
discretionary behavior items, e.g.,
signal-unresolved-template-rule-conflict, each with a controlled vocabulary
for the attribute list. -->
<!ELEMENT DiscretionaryItem EMPTY >
<!-- What follows is the complete pool of keywords, with "True" and "False"
added as a precaution. -->
<!ATTLIST DiscretionaryItem Behavior ( True | False | RaiseError |
ChooseLast |
  PassThrough | Ignore | AddSpace | ReturnEmpty | ChooseDefault | UsePublic
| UseSystem | UseGiven |
  FIFO | LIFO | Interleave ) #REQUIRED >

Vague areas in the spec would be handled in the same manner as the
discretionary items above, with <GrayArea> substituting for the
<Discretionary> and the abbreviated names chosen from The Catalog of Vague.
This is where the errata level is likely to come in to play, since errata
should clear up some vague areas. Once again, the tester has to ask the
developer to answer questions about their design decisions, and the answers
should be encoded using keywords which can then be matched to the
<GrayArea> elements. If we're clever, one test case could serve as both a
GrayArea for one choice and as the lone case for ErrataAdd, when that
GrayArea choice is the one that the errata later chose.

<!ELEMENT GrayArea ( GrayItem+ ) >
<!-- Some 35 of these GrayItems, the name is place holder until those 35 or
so are filled in -->
<!ATTLIST GrayItem Behavior ( True | False | RaiseError | Zero | Positive |
Negative |
  Whatever ) #REQUIRED >

At Lotus, we have thought a lot about how comments in the test file can
describe the scenario under which the test is run, though we have not yet
implemented most of the ideas. These parameters describe inputs and
outputs, and a <Scenario> element could describe the whole situation
through its "Operation" and "Compare" attributes. "Operation" describes how
to run the test, while "Compare" describes how to evaluate the outcome. In
the "Standard" Operation scenarios, one XML file whose name matches the XSL
stylesheet file ("Source" element) is used as the input document, and
output is expected in a file that could then be suitably compared to the
"correct output" file. "Compare" options include "XML", "HTML", and "Text",
corresponding to the three methods of xsl:output and the possible three
methods of comparison. One or more <InputFile> and <OutputFile> elements
could be used to specify other files needed or created, and the values of
these elements should permit relative paths. A single InputFile element
could be used to specify that one of the heavily-used standard input files
should be retrieved instead of a test-specific XML file. (Lotus has
hundreds of tests where the XML input is just a document-node-only trigger,
and we would benefit from keeping one such file in a Utility directory.)
The implication of the latter rule is that if there exists even one
InputFile element, no inputs are assumed and all must be specified.

<!ELEMENT Scenario ( InputFile+ , OutputFile+ , ParamSet? ,
ConsoleStandardOutput ,
  ConsoleStandardError ) >
<!ATTLIST Scenario
  operation ( Standard | Embedded | ExternalParam ) #REQUIRED
  compare ( XML | HTML | Text | StandardOutput | StandardError | SE-XML |
SE-HTML | SE-Text |
  Manual ) #REQUIRED >
<!ELEMENT InputFile ( #PCDATA ) >
<!ELEMENT OutputFile ( #PCDATA ) >

If the Scenario keyword says "Embedded", then the XSL stylesheet wasn't
really wanted, and the test should run as if the XML file sufficed. The
stylesheet file should probably do nothing and contain only comments.
Nevertheless, we may again want the processor developer to supply a
mechanism to set this up, since the way in which the stylesheet is marked
inapplicable will vary.

If the Scenario keyword says "ExternalParam", then the processor should be
launched with parameters being set via whatever mechanism the processor
supports. We may want to push responsibility to the processor developer to
provide a script/batch mechanism to take values in a standardized way and
map them to the specific syntax of their processor. We would still need to
define a method, probably involving an extra input file (i.e.,
<Source>.ini) but possibly using more parameters in the test case, where
the test case can store the parameter names and values.

<!-- This needs further design -->
<!ELEMENT ParamSet ( #PCDATA ) >

We also want to be able to test that a message was issued (as in
xsl:message) and that an error was issued. I propose that elements
"ConsoleStandardOutput" and "ConsoleErrorOutput" be used to designate
strings that must be present in the respective outputs. The Compare keyword
can designate that, when running this test, capture the standard/error
output into a file, and ignore the normal transformation output. I suspect
that we need Compare keywords like SE-HTML to say that both the standard
error and an HTML file must be compared. (For console output, the test of
correctness is to grep for the designated string in the captured output
file.) If a tester wished, they could get actual error message strings from
the processor developer and refine the test harness to search for those
exact messages in error output. In that case, the string in the
ConsoleErrorOutput element is used as an indirect reference to the actual
string. A Compare value of Manual would be used sparingly, for generate-id
() and system-property() output.

<!-- should contain actual message string -->
<!ELEMENT ConsoleStandardOutput ( #PCDATA ) >
<!-- should contain actual error report output string, or could be pointer
to another file -->
<!ELEMENT ConsoleStandardError ( #PCDATA ) >

Additional "Scenario" keywords can be devised as necessary, but OASIS
should control the naming. We might want to allow names beginning with a
specific letter to be local to particular test labs. For example, we would
reserve all names beginning with "O-" and instruct the testlabs that they
should put their name as the next field, then another hyphen, then their
local scenario keywords (e. g., O-NIST-whatever) that allow them to set up
local conditions as needed.

When generating a specific instance of the test suite, a test case can be
excluded on any one of the following bases:
A Discretionary item of a given name is set to a different value.
A GrayArea item of a given name is set to a different value.
The SpecCitation/Version value on the test case is numerically larger than
what the processor implements. (This could be for any spec named, not just
There is a SpecCitation for a spec (e.g., XBase) that the processor claims
not to implement.
The test lab wishes to test against an errata level that is numerically
lower than the ErrataAdd or higher than the ErrataDrop for a spec.
Thus, it is the "user" (test lab) who renders a test suite by deciding
which spec version and errata level they wish to test, and by specifying
the settings of the Discretionary and GrayArea items they know. Before
running the specific rendition, they must ascertain how they will handle
those tests that run in ExtParam and possibly other scenarios, taking into
account the operating system where the tests will run and
processor-specific input and output design.

Note that the test suite itself is not filtered by Scenario values. The
test lab may wish to devise a harness that can be configured to exclude
certain scenarios from some runs, but I think we want to encourage testing
and reporting against the full range of scenarios.

When a test case is included, it is run according to the value of the
<Scenario> element. If inputs are specified, they are marshalled as
necessary. If no inputs are specified, a single file named <Source>.xml is
assumed to be the input. In some scenarios, special steps must be taken to
capture the output. In the standard scenarios, if no outputs are designated
the name of the intended output file is generated from the the final part
of the scenario name and from the Source. (Probably <Source>.xml,
<Source>.html, and <Source>.txt unless someone has a better idea.)
.................David Marston

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Powered by eList eXpress LLC