OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

oiic-formation-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: More about test case metadata

As in my last message, I am using the term "test lab" to refer generically to any organization that wants to perform conformance testing of ODF, and hence would use the deliverables of the ODF I&C TC in a hands-on way. This could include software testers working for an implementer of ODF reader/editor/producer software, but also test-only organizations at buyer sites or working for a market analyst.

One of the probable justifications for creating this TC is to enable results reported by different labs to be compared. We would like the labs to be proud when stating, "We ran the official OASIS ODF test ____ ." The blank would be filled in with "suite" if the TC decides to make actual test cases a deliverable. If the deliverable is only a set of guidelines, maybe "procedure" is a better word for that blank. It could also (by a stretch) be called a "suite" if the TC delivers a catalog of test cases that should be run, but only provides information about the cases rather than the actual materials.

This brings us into the realm of test case metadata. I pointed to some in the last message when I mentioned the work of the XQuery Test Task Force [1], which combined contributed and committee-written test cases. The parts about "catalog entries" are showing the metadata in XML. Here is one of those examples:
<test-case name="modules-simple" FilePath="Modules/ModuleImport/" scenario="standard" Creator="Jane Doe">
 <description>Import simple library module</description>
 <spec-citation spec="XQuery" section-number="4.11" section-title="Module Import" section-pointer="id-module-imports"/>
 <spec-citation spec="XQuery" section-number="4.2" section-title="Module Declaration"

 <query name="modules-simple" date="2005-12-05"/>
 <module namespace="http://www.w3.org/TestModules/test1">test1-lib</module>
 <input-file role="principal-data" variable="input-context">emptydoc</input-file>
 <output-file role="principal" compare="XML">modules-simple.xml</output-file>

This describes one test case. The @scenario gives guidance about how to run the query, and the output-file/@compare gives guidance about how to compare the actual and reference results. For more about what might be in metadata, see [2].

The above example has no explicit filtering information, though in fact the presence of the <module> element means that this test case should only be applied to the XQuery implementations that implement the "module feature", an optional aspect of XQuery. XQuery has 6 "features" which may be present or absent independent of each other, as additions to the core functionality. A test case that applies to the core applies to all processors and should be run in every conformance test. Other test cases apply when the feature flagged in the metadata is present (or combination of features, if they interact). You can also have negative tests for when the feature is absent; an XQuery processor that does not implement the module feature MUST raise a specific error when it encounters a module declaration.

These XQuery "features" are bundles of functionality, and there are only six. What ODF 1.1 seems to have is highly-granular discretionary choice for low-level elements. Nevertheless, the test case metadata could be designed to account for each individual element whose implementation or non-implementation is a distinct choice available to the implementer. Having such flags would allow a test lab to use the test suite to answer, "How well does this ODF tool implement those bits that it claims to implement?" You can have separate flags for the profiles, so that the test suite can be used to determine how well the implementation fulfills a given profile. The metadata should be designed so that a list of several profiles can be provided for each test case.

How do we know what the implementer claims to implement? Through an Implementation Conformance Statement. The answers needed there relate to each individual bit of variability **defined in the spec** as a permissible unit of variation. Continuing the XQuery example, if the implementer says that they implement the "static typing feature" then they will be subject to all the test cases that are flagged as being limited to implementations having that feature. From the viewpoint of this TC, we might want to identify a questionnaire as a deliverable. See [3] for the questionnaire for XSLT 1.0, as written by the OASIS XSLT Conformance Testing TC. (Please note that the questions about English were planned to be supplemented by parallel questions concerning other languages when we obtained cases that depended on those languages.)

Dave Pawson writes:
>I'm really not sure how to address tests a vendor knows will fail
>(e.g. because they haven't implemented para x.y.z. It needs addressing
>but I'm unsure how best to do it. A control file mapping spec clauses to
>implemented features, used to control test run (and impact results) is
>perhaps an approach.)

That's the same idea, I think. "Implemented features" will come from the Implementation Conformance Statement (a.k.a. answers to the questionnaire); and the test case metadata is the control file; before running tests, filter the metadata file to exclude tests that need not be run against this particular implementation, then process the remainder of the metadata into an executable script. If a vendor says they don't implement XYZ, they're risking a negative view in the marketplace, and we don't really need to run the test cases that depend on XYZ to confirm the lack of implementation. If they say they implement XYZ, they are subject to all the XYZ tests, which may reveal that their implementation is incomplete or wrong. It is up to the ODF TC to write the spec so that XYZ is clearly delineated. Again, the vendor cannot decline to implement a feature unless the ODF spec (1) clearly isolates it and (2) states that the implementation SHOULD or MAY implement it, rather than MUST.

Returning to the main theme of test case metadata: the <test-case> elements can be gathered up into a single XML file to facilitate processing by XSLT. In addition to filtering out test cases that do not apply to the Implementation Under Test, there are many other uses for transformed metadata, such as administrative scripts for the test lab. This TC may want to write metadata for cases that do not (yet) have the necessary materials, and give them a status flag that indicates they are "proposed" or "designed" but not yet runnable. Thus, the catalog could describe the cases that are wanted, which may serve as a spur to their creation.

Dave Pawson questions the <output-file> element and the notion that the TC would deliver the reference (correct) result for each test case:
DM>> Reference outputs that show the correct behavior of a test case
DP> -1
DP> Rationale. If we define fixed expected data test cases can be built to pass it.
DP> If we leave the detail up to the implementer then the vendors are left guessing as to what the data might be.

Once again, the test case metadata can help. If a test case can be set up so that one or a few properties vary, the metadata can identify the variable(s) and their dataypes. From that, run-time instructions can be generated with a randomly-chosen value to be assigned (say, the font for a certain bit of text) and the reference output, having been delivered with a parameter flag, would be processed to have that same randomly-chosen value plugged in for the file actually used in the compare portion of the test. (More generally, the metadata can specify any pre-processing needed to set up the test case, including actions like choosing a font at random.)

Test assertions are not in the metadata. If anything, the test case metadata can cite assertions in addition to citing passages in the spec, if there happens to be a collection of test assertions, each with its own unique ID.

There are several other aspects of the metadata, such as the file paths shown in the example above, that I have not explained. I'm just trying to give a general idea of the range of possibilities, to help those following this discussion to decide whether test case metadata ought to be itemized as a deliverable of the TC. I hope it also helps you think about the other potential deliverables as well.
.................David Marston
IBM Research

[1] http://www.w3.org/XML/Query/test-suite/Guidelines%20for%20Test%20Submission.html
[2] http://www.w3.org/QA/2005/08/test-case-metadata.html
[3] open.org/committees/download.php/12370/xsltDevQ.htm

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]