RE: [tag] Groups - TA Anatomy V0.5 (AnatomyTA-v05.doc) uploaded

["Durand, Jacques R." <JDurand@us.fujitsu.com>]

Thank you all for a quick feedback! (nothing such as floating a bait
around, to get a few bites ;-) 

1. The "laughters" would come, as S.G. pointed out , from the fact that
a 3rd outcome besides pass/fail - call it "inconclusive" - has not been
explicitly stated. No I'm not logically-challenged... 

2. At the root of all this, a fundamental question on pass/fail
semantics. Do we want:
(a) pass/fail that has a test semantics only, i.e. nothing more than
true/false if the test is presented as a logical condition. Thus, a
"pass"  in a "negative TA" could be interpreted later as the IUT being
NOT consistent with spec requirement.
(b) pass/fail that has a conformance semantics (i.e. an outcome = "fail"
means the IUT is inconsistent with the addressed spec requirement, and
"pass" means consistency within scope of this test.) 
I am leaning in favor of (b) as I think it is more intuitive. Certainly
many people - including me - have written their TAs that way without
trouble. But (b) must make room for a third outcome:
pass/fail/inconclusive.  

3. There seems to be a consensus (including me) that at the end of the
day, a conformance clause or a test suite will make sense of the results
of all the test cases that exercise these Tas, w/r to conformance. So
indeed, a "pass" or a "fail" can be interpreted as needed based on what
kind of conformance we shoot for.  
But if we go with (a) in (2) above, I would argue that a TA writer MUST
provide guidance as how this test result is to be interpreted, i.e. what
it says about adherence to the spec requirement. Otherwise, talk about
confusion for test case writers, as well as for conf clause writers. 
I still think it's nice to have a TA say something about IUT consistency
with the spec requirement it addresses !! (a) above would give
absolutely no indication as how to interpret the result of this test.

4. About "fail" != not("pass") and vice-versa: agree needs be
illustrated better. But note that the example starting page #1, is
worded as such a case - if condition for "pass" is false, does not mean
that the IUT is incompatible with spec requirement ("fail")... The test
condition could be "false" for other reasons than spec requirement
issues (unless we craft adequate pre-requisites/pre-conditions?).
Depends on how it is worded. But more common is negative testing
practice, all about demonstrating "fail" or not, but saying nothing
about "pass".  

Regards,
Jacques


-----Original Message-----
From: stephen.green@systml.co.uk [mailto:stephen.green@systml.co.uk] 
Sent: Wednesday, October 03, 2007 1:02 PM
To: tag@lists.oasis-open.org
Subject: Re: [tag] Groups - TA Anatomy V0.5 (AnatomyTA-v05.doc) uploaded


<quote>
Pass: the condition for adherence is satisfied (or is "true") ...
Notes:
- A test assertion may specify condition(s) for pass alone, or for fail
alone, or for both. </quote>


Is there an example of a TA for pass alone?
I imagine most concerned with pass will also cover fail as the opposite
of pass won't they?


<quote>
-  If the adherence condition associated with a pass outcome is not
satisfied for an IUT (i.e. it evaluates to "false") this does not cause
a fail outcome.
</quote>

I guess this is the statement which Dave points out could be laughed at

a bit. Fair enough if it is not thought about too much but, still, if
the TA is for pass alone then this statement is necessarily true.
I think there's a problem believing there are TAs for pass alone which
is why it seems a bit laughable.

Perhaps the Note should also cover the easier to understand type of
adherence condition which is associated with both a pass outcome and a
fail outcome. How about adding it at the beginning (with a bit of
rewording)?

"Notes:

...

The adherence condition may simply be associated with a pass outcome and
also cover the fail outcome such that if the condition is not satisfied
for the IUT (i.e. it evaluates to "false") this will cause a fail
outcome, as expected. However, if the adherence condition is associated
with just a pass outcome and not the fail and is not satisfied for an
IUT (i.e. it evaluates to "false") this does NOT cause a fail outcome.
Conversely, if the incompatibility condition associated with just a fail
outcome is not satisfied (i.e. it evaluates to "false") this does not
cause a pass outcome.
..."

I think examples would be clearer though than just stating these things
as (in some cases apparently unlikely) facts in the Notes.
We'd especially need an example of a condition for a pass alone but one
for a fail alone seems more likely to be typical.

I think I agree now with both Dave and Lynne if I understand it rightly
that most existing TA examples will be for a pass but will also include
(perhaps implicitly) a fail and I think the fail might tend to be
associated with a false condition in most existing TAs. Maybe there is
some change management needed...

Regards


Steve