OASIS XSLT/XPath Conformance Committee

Test Lab Guidelines

I.	General Information
A.	This package includes test cases, reference output for comparison, and catalog data that you can use to generate various scripts. The catalog data is in XML, and the scripts can be generated via XSLT. We include a standalone sufficiency test so you can determine whether your XSLT processor can generate the anticipated scripts.
B.	The expected "reference" results from the submitters will be XML, HTML or text format, as will the user's actual results.  These raw results will be converted to an XML document with an indirect representation of the content in InfoSet-style markup, always in UTF-8 encoding.  The XSLT/XPath Conformance Committee will supply the user with the "reference" results and the user will apply a committee-supplied Information Set Analysis mechanism to produce the Users Results Description of the actual test results run on a particular processor.  The XML results can be canonicalized and the user can perform a byte-wise or text comparison.  For example, if the result is an XML document, the user can apply an XSLT stylesheet supplied in the XSLT/XPath Conformance Testing package and use the processor being tested or another processor to produce an XML InfoSet representation of that result.  Processing the information set result using Canonical XML or any consistent serializer with both the committee-supplied expected results and actual results allows for easy comparison.

II.	Unpacking and setting up the test system
A.	This document explains
1.	Instructions for installing the test suite
2.	Instructions for rendering the suite for a particular test run
3.	Instructions for canonicalizing and comparing output
4.	Instructions particular to each "operation" scenario
5.	Guidance on running single test cases that check the setup
6.	Guidance on the uses of individual data items in the catalog
7.	Guidance on the directory structure (e.g., read-only parts)
	Other documents explain
1.	Information about discretionary items and the questionnaire
2.	Information on the W3C specs and errata we deal with
3.	(When the time comes) instructions about updating the suite
4.	How to submit test cases
5.	The OASIS generic testing framework and this package as an instance of it
B.	Planning your file system layout
1.	File tree of test cases (read-only)
2.	Configuration data and docs in top-level directory [name?]
3.	Reference output in two trees, raw and InfoSet (read-only); will need 3rd for canonical
4.	Generate the canonical version of the reference output (one-time process)
a.	Known conformant canonical processor available at <http://www.elcel.com/products/xmlcanon.html>
b.	Checking that it ran correctly
5.	Suggestion: directory trees for each processor being tested
6.	May need extra trees for different invocation options, run dates, etc.
7.	Remarks about relative paths
C.	Planning your test operations
1.	You will need to know how to invoke each processor (see next section)
2.	Operating environment: command shell? JRE? other?
3.	Do you want to create output in its storage place or move it when done?
4.	Master script that runs all test cases, invoking single-test script
5.	Byte-wise file comparison tool
6.	Capturing console output
7.	Lack of generated output, planned vs. unplanned
8.	If multiple processors under test, need to render suite for each
9.	If a processor "compiles" the stylesheet first, plan for storage of compiled versions

III.	Getting an XSLT processor ready for testing
A.	The Test Lab needs to get the following information from the processor developer(s)
1.	Specifications of the necessary operating system (or JRE)
2.	Specifications of other software needed to run the processor
3.	Instructions for installing the processor
4.	Command-line and/or API arguments; invocation sequence
5.	Information about how errors and xsl:message are handled
6.	Their answers to our questionnaire about discretionary items
7.	Information about languages and locales supported
8.	(For later use) How to set parameters "externally"
B.	Ensure that the processor works
1.	Vendor's setup tests
2.	Try a couple tests from this suite
3.	Devise a single-case script, batch file, or invocation program
4.	Try pertinent invocation options
C. Using XSLT as part of the installation
1.	The sufficiency test
2.	Trial generation of a master script for running tests
3.	How you can manipulate fixed strings and name parts to make other scripts

IV.	Running the tests and evaluating a processor
A.	Setup and planning
1.	Script to check that all the input files needed by a test case exist in the correct directories
a.	The "cd issue" arises because the principal stylesheet can refer to files in other roles
1)	supplemental-data: document('filename')
2)	supplemental-stylesheet: xsl:import or xsl:include files
b.	Supplemental files may have relative paths, not just names. 
c.	The current directory should be the one containing the stylesheet.
d.	The supplemental inputs can be tested for existence relative to that directory.
2.	The test lab uses catalog data to generate a script with all the necessary "cd" (or equivalent) commands intermixed among the commands to invoke the tests. 
3.	The following naming parts are supplied to the test lab 
a.	test-catalog/@submitter names the directory for one submission
b.	file-path gives the intermediary directories to locate a case
c.	case-name or "id" is the name of a case for display purposes
d.	input-file and output-file elements contain names of all files used in a test case
e.	Note that (a) and (b) are only separate for Committee purposes, and that all the test lab uses need to use the concatenation of the two (with a / or appropriate separator between).
4.	Using the above name parts plus naming conventions suggested by the testing guide, a test lab should be able to create all the following in an automated way. In these items, "script" means a batch file, shell script, or whatever. These would be generated off the catalog data by an XSLT transformation with text-method output
a.	Script to canonicalize all InfoSetized reference output
b.	Script that wraps the process of rendering the test suite for a given processor and errata level
c.	Script to check that all the input files needed by a test case exist in the correct directories
d.	Line in a script to run a test from the command line, feeding in the name of the principal output file, and ensure that the output is either generated in or moved to the correct directory
e.	Variation of (d) where console output must be captured and moved to an appropriate directory.
f.	Script to run all tests using a mixture of (d) and (e), and also make a log file that names the cases being run (could also display  "purpose" strings or spec-citations if desired)
g.	Programs similar to (d) and (e) that use API calls
h.	Scripts to InfoSetize and then canonicalize the output from the test run, using different directory trees for each
i.	Script to compare all canonicalized test-run outputs against the corresponding reference outputs
j.	Script to delete files no longer needed after a successful run
k.	HREF-clean version of fully-qualified name (change / to _)
5.	Each "operation" has its own details of invocation
a.	standard: provide both XML and stylesheet as arguments
b.	embedded: provide XML as only input
c.	external-param [TBD later]
d.	execution-error: need to capture console, transformation output is a mistake
			e.   result-analysis: human review of results needed
6.	Certain "compare" values require special attention
a.	ignore: presence of this file is a mistake
b.	manual/generate-id: look at generated IDs, ensure that they are in correct format
c.	manual/system-property: look at actual values, ensure they are accurate
B.	 Rendering the test suite for a given processor and errata level
1.	Gray areas and discretionary items filter out cases - need data in XML
2.	Specify the errata level
3.	Running the rendering tool
C.	Running the applicable test cases through the processor
1.	The XSLT/XPath Conformance Committee will supply the test lab with the expected results and the test lab will apply a committee-supplied Information Set Analysis mechanism to produce the Users Results Description of the actual test results run on a particular processor. 
2.	The XML results can be serialized in canonical or other proprietary consistent way and then compared byte wise. 
3.	The raw results from XSLT Stylesheet (Test Case) can be XML, HTML or text format. 
4.	The user's actual results after running the test can be XML, HTML or text format.
D.	Comparing the results of this run against the reference output
a.	The XML results will be converted to an XML document using a subset of the XML InfoSet
b.	Canonical XML or any consistent serialization can be used for byte-wise comparison
2.	Text Output
a.	If the results are text, the test lab or user can be supplied with a form of the text code in an XML InfoSet document.
1)	Encoded in UTF-8
2)	Separate text node for each line
3)	Actual encoding specified as an attribute of the rootb.	An alternate approach is supplying the output as the combination of all text nodes from XML output in document order.
3.	HTML Output
a.	This one is hard due to:
1)	Different treatment for tags and attributes between HTML and XML
a.	HTML allows attributes without values. Ex: <INPUT disabled />
2)	Different treatment of white spaces between HTML and XML
b.	Tools to solve:
1)	W3C Tidy tool to solve 1^st problem
Tool that produce consistent output regarding whitespace (IE libraries has one)
V.   Additional notes about going into "production mode"
	A. Generating scripts dynamically via XSLT
	B. File cleanup after a run
	C. Moving results to different directory trees
	D. Moving and/or cleaning up log files