Test case metadata is not in scope (was: Re: Reasons to wait...)

[david_marston@us.ibm.com]

DM>>Test case metadata is on the list
>>of technology enhancements, above TAs in priority,...

DP>Nowhere have I seen any rationale for test metadata, nor suggestions
>as to what it migh contain.

That's because it's not in scope for a TC on TAs. TAs are the
independent variable, metadata the dependent. The existing work is
mostly in the W3C, but building in part on work I did with the old
XSLT Conformance Testing TC, which was in OASIS.

Like the OASIS Conformance TC and the W3C QA Interest Group, which had
a lot of the same people as are on this list, a group defining TAs
should be working for the benefit of W3C, OASIS, and several similar
consortia, plus all the industry-specific consortia that do Web specs.

The W3C has a Wiki page about test case metadata at
http://esw.w3.org/topic/TestCaseMetadata
but the part collecting ideas about specific data items has been organized 

into a Note found at
http://www.w3.org/TR/2005/NOTE-test-metadata-20050914/

DM>>work is underway to put test case metadata into XML.

DP>Which, prior to the formation of the [TA] TC, seems (from an Oasis
>viewpoint) surely wasted effort?

No, because traceability to the specs (via TAs or other citations) is
only part of the value. More significant is the platform agility and the
filtering value that arises from encoding Dimensions of Variability into
the metadata. In other words, we (W3C, OASIS, etc.) needed test case
metadata for conformance testing even before having the luxury of TAs.

DM>>>>If we design TAs to be amenable to machine processing (e.g., by
>>>>expressing them in XML), then there is hope that early efforts at
>>>>TAs will become more valuable in the future as new tools emerge.
>>>
DP>>>A test matrix will do that.
>>
DM>>Intriguing thought! Can you elaborate? To me, the term "test
>>matrix" suggests a display of selected test case metadata.
>
>Simple matrix of test spec para vs test addressing that para.
>A simple means of checking test completeness.

Essentially, you are confirming my thought that this matrix *is* a
display of test case metadata in a special format. It's currently
test cases across the top and spec paragraphs down the side. If TAs
become defined, metadata will use them, so the matrix will become
test cases across the top and TAs down the side. Completeness
checking is done by a human.

I will comment on other parts of Dave Pawson's missive on the main
thread about whether we are ready to draw up a charter for a TC.
.................David Marston