Re: Reasons to wait on convening a TC

[david_marston@us.ibm.com]

No insult intended to Dave Pawson! I
have contributed material to his XSLT FAQ in the past. I was specifically
calling attention to how XML metadata and XSLT transformations are central
to **conformance testing** of the XSLT/XPath/XQuery family of specs.

Back on the main line of skepticism
about TAs in XML, I think a blunt statement might be in order:
The Spec Guidelines tell editors to
"write testable sentences" and to a great extent, spec editors
already do that. If the TA TC is formed and all they do is tell spec editors
how to write more-refined forms of testable sentences, it wasn't worth
it.

Right now, test planners and writers
of test cases can do a reasonably good job of using prose in the specs
to design a set of test cases by the "manual" methods of understanding
the sentences and joining multiple constraints in their minds. If the TA
TC is formed and they issue a standard for machine-processable TAs, then
we have a significant advance in conformance testing, and the work of the
TC was worth it.

DP>It would appear you have an end to end solution
mapped out.

Well, yes and no. I think the conformance
testing community is aware that some aspects of their individual solutions
are hard to quantify in planning and might be due for enhanced technology.
Test case metadata is on the list of technology enhancements, above TAs
in priority, and we know that work is underway to put test case metadata
into XML. Because it's XML, it can be reformatted to different XML if future
developments change the details. If you get the people on this list together
face-to-face and have them talk about their respective needs and pains
in their own end-to-end solutions, TAs will come up as a frequent issue.
In my involvement with XSLT testing (since early 1998), planning always
has to assume that a hard-to-quantify but large amount of time is spent
in just reading the specs very carefully, building up implications from
testable sentences that are far apart in the documents, and figuring out
the correct behavior and how to test it. Machine processable TAs would
act as a catalyst for tooling to help in this process. It is appropriate
to convene a TC to fill a hole identified by many, without presuming that
the hole exists in just one end-to-end process.

Not mentioned so far, but also worth
considering: machine-processable TAs might also help developers write correct
code. TAs might be usable by debuggers and tracing/logging tools, or even
by code-correctness checkers.

DM>>If we design TAs to be amenable to machine
processing (e.g., by
>>expressing them in XML), then there is hope that early efforts
at
>>TAs will become more valuable in the future as new tools emerge.

DP>A test matrix will do that.

Intriguing thought! Can you elaborate?
To me, the term "test matrix" suggests a display of selected
test case metadata. How does a test matrix involve TAs and allow them to
be compounded? Perhaps there should be a TC standardizing test matrices
instead of TAs?

DP>What of a WG that doesn't have any XML expertise?
Wouldn't
>[defining the TA format as an XML vocabulary]
>be making it unecessarily hard for them?

It's possible that TAs will only apply
to a subset of the type of specs issued by OASIS, W3C, OMG, and the like.
(BTW, XML knowledge is effectively mandatory for spec editors at the W3C.)
This gets back to the "testable sentences" remark I made earlier.
I'm pretty sure that the hard part about composing well-formed TAs is formulating
the contingencies and conclusion (or whatever the various parts will be
named) to be suitably rigorous. On the other side, there may already be
vocabularies for "rules" that have already solved all aspects
of the TA problem other than TA-specific markup.
.................David Marston