RE: [tag] Proposals for updating the "Anatomy of a TA" section (V0.6):

["Durand, Jacques R." <JDurand@us.fujitsu.com>]

I believe indeed some of the questions from David must be answered in
order to write a TA for a requirement such as:
 
"...an Error message caused by the reception of a message that is not
well-formed MUST refer to the faulty message using its MessageID."

 

where it does not seem that a single indicator expression can do the job
of indicating either conformance/adherence if "true", and non adherence
if "false".
Indeed, if we want a TA to  do more than just repeat the specification
requirement, it must include something about the nature of the test
operation:
E.g. 
(An Error message is send out by MSH in response to receiving an
ill-formed message 'M') AND (The ReferenceId field of the Error message
has same value as the MessageId of M).
 
While "true"  clearly "indicates" conformance, "false" does not indicate
non-conformance because the reason for failure may come from the first
member of the boolean conjunction. Now, we may consider various ways to
rewrite this, as one could argue that the above expression mixes two
different things: (a) test preparation + (b) the actual test expression.
For example, put the first member in a "test flow" part instead, or make
it a pre-requisite, or both. Or rewrite as "when (faulty message is sent
and error geenrated) then (test expression)".
 
But there are other examples better than this one where a single boolean
expression will indicate conformance if true, but NOT non-conformance if
false - and conversely.
 
Jacques
 
 

________________________________

From: david_marston@us.ibm.com [mailto:david_marston@us.ibm.com] 
Sent: Wednesday, December 05, 2007 8:42 AM
To: tag-comment@lists.oasis-open.org
Cc: Durand, Jacques R.
Subject: Re: [tag] Proposals for updating the "Anatomy of a TA" section
(V0.6):



The latest proposal of "qualifiers" and "indicators" is getting closer
to a good resolution. Let me remind you of two important goals. 
1. TAs must be usable for more than conformance testing. 
2. There should be some way that a qualifier (formerly precondition,
antecedent, etc.) of one TA can be tied to the outcome of another TA, so
you know whether you will be getting a usable result. 

Also, I want to remind you of a statement I made in my email of 3
October 2007 ("On non-passage and non-failure"): 
3. The idea of "unable to evaluate" pass/fail as a third outcome state
cannot be truly eliminated, it can only be pushed into one corner or
another. 
And indeed, the latest proposal chooses a corner (corner #3, for those
going back to the earlier message). 

When looking at conformance testing, we also need to keep in mind this
observation that was brought up a few months ago: 
4. No conformance test suite can ever "prove" conformance. If an IUT
fails a test case, that can prove NON-conformance, but no amount of
passing cases (even if it's 100% of available cases) shows that the IUT
always does the right thing in every usage outside the test suite. 
I think this principle is in jeopardy under the latest proposal, so I
want to ask you to go to the next level of rigor when discussing the new
anatomy at your F2F. 

I also assume that the latest proposal talks about qualifier (singular)
to stand in for the logical AND of a set of qualifiers that apply to the
TA. This concept has arisen before and does not appear controversial,
but it will affect the structure of a TA, at least in XML form. 

With all the above as background, I propose the following step-by-step
process for evaluating the new anatomy proposal. Each step ends in an
answer to an important question, and I would expect that the answers are
needed by anyone who would attempt to use TAs. 
FIRST, you must decide how the TAs work in a conformance context. That
is, what set of indicators must be TRUE to indicate conformance (to the
extent tested)? Or by applying (4) above, will it really become: what
set of indicators would indicate non-conformance if any one of them is
FALSE? Would it be better to structure the indicators to focus on true
being good or false being bad? Under the proposed design, you can't get
both decisive good and bad in the same set of TAs, which one of JD's
notes admits. Do you want to allow the use of either structure? If so,
you will have to go through the rest of the decisions enumerated here
twice. 
SECOND, after having established what a positive test assertion
indicator signifies, you want a negative test indicator to work the same
way. This may be mostly a burden on the test case writers, but the specs
need to make clear how a TA should be written so that positive and
negative will blend together consistently. 
THIRD, re-check the design for test situations other than conformance.
In a real world situation, we are often deciding whether to ship a
product based on test results that show (A) delivery of a certain level
of correct functionality, (B) absence of any really bad failures, and
(C) overall "good enough" numbers of tests being passed. Will the TAs be
able to be divided along similar lines? (Groups A and B are represented
by test cases that must pass, while C is a percentage of all tests for
which the qualifier is true. In this scenario, the decision to not
implement a certain feature acts as a qualifier.) 

After answering the above, you can then move on to fill in detailed
charts of how the qualifier(s) cause a test to be reduced from decisive
to non-decisive. I think this involves a matrix of qualifier true/false
against test outcome true/false/undetermined, and each cell inside is
the indicator value, either true or false. (I have a suspicion that a
test could result in "undetermined" for reasons other than failing to
satisfy the qualifier. Maybe that could be a side discussion.) 

Finally, the above decisions MAY lead to the need to specify an
abstraction of a particular bit of metadata for each test case. The
datum would specify whether pass or fail is the definitive outcome for
the test case, other outcomes being mere suggestions. The reason you
would want to define this is so that you can explain the application of
outcomes to qualifiers of other tests. This is a consequence of JD's
suggestion that "The qualifier may refer to one or more other TA, with
the semantics that the test target must then evaluate to "true" by the
Indicator of these referred TAs, in order to qualify." You don't know
whether the indicator of the referred TA is true until you run the test
cases for that TA, but the test metadata may be saying that failure is
the decisive outcome. 
.................David Marston