More about test case metadata

[david_marston@us.ibm.com]

As in my last message, I am using the
term "test lab" to refer generically to any organization that
wants to perform conformance testing of ODF, and hence would use the deliverables
of the ODF I&C TC in a hands-on way. This could include software testers
working for an implementer of ODF reader/editor/producer software, but
also test-only organizations at buyer sites or working for a market analyst.

One of the probable justifications for
creating this TC is to enable results reported by different labs to be
compared. We would like the labs to be proud when stating, "We ran
the official OASIS ODF test ____ ." The blank would be filled in with
"suite" if the TC decides to make actual test cases a deliverable.
If the deliverable is only a set of guidelines, maybe "procedure"
is a better word for that blank. It could also (by a stretch) be called
a "suite" if the TC delivers a catalog of test cases that should
be run, but only provides information about the cases rather than the actual
materials.

This brings us into the realm of test
case metadata. I pointed to some in the last message when I mentioned the
work of the XQuery Test Task Force [1], which combined contributed and
committee-written test cases. The parts about "catalog entries"
are showing the metadata in XML. Here is one of those examples:
<test-case name="modules-simple" FilePath="Modules/ModuleImport/"
scenario="standard" Creator="Jane Doe">
  <description>Import simple library module</description>
  <spec-citation spec="XQuery" section-number="4.11"
section-title="Module Import" section-pointer="id-module-imports"/>
  <spec-citation spec="XQuery" section-number="4.2"
section-title="Module Declaration"
    section-pointer="id-module-declaration"/>
  <query name="modules-simple" date="2005-12-05"/>
  <module namespace="http://www.w3.org/TestModules/test1">test1-lib</module>
  <input-file role="principal-data" variable="input-context">emptydoc</input-file>
  <output-file role="principal" compare="XML">modules-simple.xml</output-file>
</test-case> 
This describes one test case. The @scenario
gives guidance about how to run the query, and the output-file/@compare
gives guidance about how to compare the actual and reference results. For
more about what might be in metadata, see [2].

The above example has no explicit filtering
information, though in fact the presence of the <module> element
means that this test case should only be applied to the XQuery implementations
that implement the "module feature", an optional aspect of XQuery.
XQuery has 6 "features" which may be present or absent independent
of each other, as additions to the core functionality. A test case that
applies to the core applies to all processors and should be run in every
conformance test. Other test cases apply when the feature flagged in the
metadata is present (or combination of features, if they interact). You
can also have negative tests for when the feature is absent; an XQuery
processor that does not implement the module feature MUST raise a specific
error when it encounters a module
declaration.

These XQuery "features" are
bundles of functionality, and there are only six. What ODF 1.1 seems to
have is highly-granular discretionary choice for low-level elements. Nevertheless,
the test case metadata could be designed to account for each individual
element whose implementation or non-implementation is a distinct choice
available to the implementer. Having such flags would allow a test lab
to use the test suite to answer, "How well does this ODF tool implement
those bits that it claims to implement?" You can have separate flags
for the profiles, so that the test suite can be used to determine how well
the implementation fulfills a given profile. The metadata should be designed
so that a list of several profiles can be provided for each test case.

How do we know what the implementer
claims to implement? Through an Implementation Conformance Statement. The
answers needed there relate to each individual bit of variability **defined
in the spec** as a permissible unit of variation. Continuing the XQuery
example, if the implementer says that they implement the "static typing
feature" then they will be subject to all the test cases that are
flagged as being limited to implementations having that feature. From the
viewpoint of this TC, we might want to identify a questionnaire as a deliverable.
See [3] for the questionnaire for XSLT 1.0, as written by the OASIS XSLT
Conformance Testing TC. (Please note that the questions about English were
planned to be supplemented by parallel questions concerning other languages
when we obtained cases that depended on those languages.)

Dave Pawson writes:
>I'm really not sure how to address tests a vendor
knows will fail
>(e.g. because they haven't implemented para x.y.z. It needs addressing
>but I'm unsure how best to do it. A control file mapping spec clauses
to
>implemented features, used to control test run (and impact results)
is
>perhaps an approach.)

That's the same idea, I think. "Implemented
features" will come from the Implementation Conformance Statement
(a.k.a. answers to the questionnaire); and the test case metadata is the
control file; before running tests, filter the metadata file to exclude
tests that need not be run against this particular implementation, then
process the remainder of the metadata into an executable script. If a vendor
says they don't implement XYZ, they're risking a negative view in the marketplace,
and we don't really need to run the test cases that depend on XYZ to confirm
the lack of implementation. If they say they implement XYZ, they are subject
to all the XYZ tests, which may reveal that their implementation is incomplete
or wrong. It is up to the ODF TC to write the spec so that XYZ is clearly
delineated. Again, the vendor cannot decline to implement a feature unless
the ODF spec (1) clearly isolates it and (2) states that the implementation
SHOULD or MAY implement it, rather than MUST.

Returning to the main theme of test
case metadata: the <test-case> elements can be gathered up into a
single XML file to facilitate processing by XSLT. In addition to filtering
out test cases that do not apply to the Implementation Under Test, there
are many other uses for transformed metadata, such as administrative scripts
for the test lab. This TC may want to write metadata for cases that do
not (yet) have the necessary materials, and give them a status flag that
indicates they are "proposed" or "designed" but not
yet runnable. Thus, the catalog could describe the cases that are wanted,
which may serve as a spur to their creation.

Dave Pawson questions the <output-file>
element and the notion that the TC would deliver the reference (correct)
result for each test case:
DM>> Reference outputs that show the correct
behavior of a test case
DP> -1
DP> Rationale. If we define fixed expected data test cases can be built
to pass it.
DP> If we leave the detail up to the implementer then the vendors are
left guessing as to what the data might be.
Once again, the test case metadata can
help. If a test case can be set up so that one or a few properties vary,
the metadata can identify the variable(s) and their dataypes. From that,
run-time instructions can be generated with a randomly-chosen value to
be assigned (say, the font for a certain bit of text) and the reference
output, having been delivered with a parameter flag, would be processed
to have that same randomly-chosen value plugged in for the file actually
used in the compare portion of the test. (More generally, the metadata
can specify any pre-processing needed to set up the test case, including
actions like choosing a font at random.)

Test assertions are not in the metadata.
If anything, the test case metadata can cite assertions in addition to
citing passages in the spec, if there happens to be a collection of test
assertions, each with its own unique ID.

There are several other aspects of the
metadata, such as the file paths shown in the example above, that I have
not explained. I'm just trying to give a general idea of the range of possibilities,
to help those following this discussion to decide whether test case metadata
ought to be itemized as a deliverable of the TC. I hope it also helps you
think about the other potential deliverables as well.
.................David Marston
IBM Research 


[1] http://www.w3.org/XML/Query/test-suite/Guidelines%20for%20Test%20Submission.html
[2] http://www.w3.org/QA/2005/08/test-case-metadata.html
[3] open.org/committees/download.php/12370/xsltDevQ.htm