Re: [tag] Proposals for updating the "Anatomy of a TA" section (V0.6):

[david_marston@us.ibm.com]

The latest proposal of "qualifiers" and "indicators" is getting closer to 
a good resolution. Let me remind you of two important goals.
1. TAs must be usable for more than conformance testing.
2. There should be some way that a qualifier (formerly precondition, 
antecedent, etc.) of one TA can be tied to the outcome of another TA, so 
you know whether you will be getting a usable result.

Also, I want to remind you of a statement I made in my email of 3 October 
2007 ("On non-passage and non-failure"):
3. The idea of "unable to evaluate" pass/fail as a third outcome state 
cannot be truly eliminated, it can only be pushed into one corner or 
another.
And indeed, the latest proposal chooses a corner (corner #3, for those 
going back to the earlier message).

When looking at conformance testing, we also need to keep in mind this 
observation that was brought up a few months ago:
4. No conformance test suite can ever "prove" conformance. If an IUT fails 
a test case, that can prove NON-conformance, but no amount of passing 
cases (even if it's 100% of available cases) shows that the IUT always 
does the right thing in every usage outside the test suite.
I think this principle is in jeopardy under the latest proposal, so I want 
to ask you to go to the next level of rigor when discussing the new 
anatomy at your F2F.

I also assume that the latest proposal talks about qualifier (singular) to 
stand in for the logical AND of a set of qualifiers that apply to the TA. 
This concept has arisen before and does not appear controversial, but it 
will affect the structure of a TA, at least in XML form.

With all the above as background, I propose the following step-by-step 
process for evaluating the new anatomy proposal. Each step ends in an 
answer to an important question, and I would expect that the answers are 
needed by anyone who would attempt to use TAs.
FIRST, you must decide how the TAs work in a conformance context. That is, 
what set of indicators must be TRUE to indicate conformance (to the extent 
tested)? Or by applying (4) above, will it really become: what set of 
indicators would indicate non-conformance if any one of them is FALSE? 
Would it be better to structure the indicators to focus on true being good 
or false being bad? Under the proposed design, you can't get both decisive 
good and bad in the same set of TAs, which one of JD's notes admits. Do 
you want to allow the use of either structure? If so, you will have to go 
through the rest of the decisions enumerated here twice.
SECOND, after having established what a positive test assertion indicator 
signifies, you want a negative test indicator to work the same way. This 
may be mostly a burden on the test case writers, but the specs need to 
make clear how a TA should be written so that positive and negative will 
blend together consistently.
THIRD, re-check the design for test situations other than conformance. In 
a real world situation, we are often deciding whether to ship a product 
based on test results that show (A) delivery of a certain level of correct 
functionality, (B) absence of any really bad failures, and (C) overall 
"good enough" numbers of tests being passed. Will the TAs be able to be 
divided along similar lines? (Groups A and B are represented by test cases 
that must pass, while C is a percentage of all tests for which the 
qualifier is true. In this scenario, the decision to not implement a 
certain feature acts as a qualifier.)

After answering the above, you can then move on to fill in detailed charts 
of how the qualifier(s) cause a test to be reduced from decisive to 
non-decisive. I think this involves a matrix of qualifier true/false 
against test outcome true/false/undetermined, and each cell inside is the 
indicator value, either true or false. (I have a suspicion that a test 
could result in "undetermined" for reasons other than failing to satisfy 
the qualifier. Maybe that could be a side discussion.)

Finally, the above decisions MAY lead to the need to specify an 
abstraction of a particular bit of metadata for each test case. The datum 
would specify whether pass or fail is the definitive outcome for the test 
case, other outcomes being mere suggestions. The reason you would want to 
define this is so that you can explain the application of outcomes to 
qualifiers of other tests. This is a consequence of JD's suggestion that "
The qualifier may refer to one or more other TA, with the semantics that 
the test target must then evaluate to "true" by the Indicator of these 
referred TAs, in order to qualify." You don't know whether the indicator 
of the referred TA is true until you run the test cases for that TA, but 
the test metadata may be saying that failure is the decisive outcome.
.................David Marston