Thread: TA Outcome (formerly: RE: [tag] some thoughts and comments from reading a month worth ofmessages)

["Durand, Jacques R." <JDurand@us.fujitsu.com>]

My 2 cents on the outcome of a TA :


Agree that only the overall result of a TA needs be described. But I would give this result a clear meaning w/r to the specification requirement addressed by the TA:

I see only two "conclusive" outcomes (w/r to this I agee with Dave):

"pass": when the conditions and/or behavior leading to "pass" outcome are met by a Test Case derived from this TA for a given IUT (or UUT), then the IUT conforms to the specification requirement addressed by this TA.

"fail": when the conditions and/or behavior leading to "fail" outcome are met by a Test Case derived from this TA, for a given IUT (or UUT), then the IUT does NOT conform to the specification requirement addressed by this TA.

But I feel we also need an unconclusive outcome at TA level:

"non-applicable": when the conditions and/or behavior of the IUT (or of the system IUT+TestEnvironment?) leading to the initiation of the actual test are NOT met, then this TA does not apply to this IUT. No conclusion can be made w/r to the addressed specification requirement. Note that this "non-application"  is different from a failure of the test environment itself: here it only means the IUT is not qualified, while a test env failure means that some unexpected contingencies made the testing impossible - an unexpected test case problem, not a definition of the TA application scope.

NOTES:

-  Every other "corner case" outcome or outcome variant (test error due to failure of the test procedure itself, various error messages associated with different types of failure) IMO are relevant to test cases and do not need be described in the TA.

- "warnings" and all kinds of severity levels do not need be described in a TA : either they are "fail"  cases that are interpreted as non-critical for conformance - so this is a conformance assessment out of scope for us  (e.g. when the spec reqt was a SHOULD, a "fail"  TA outcome will translate into a "warning" in the final test report.) Or, I could imagine a warning being produced by the test case execution (orthogonal to the TA outcome), expressing for example some reduced testability due to some unfavorable input material or condition.

-  I wouldn't try to go "fine-grain" in defining a TA as a whole workflow of sub-steps  with step-level outcomes, etc. But I can imagine that there may be several ways to fail or to pass a TA, that need be listed. So the TA model should allow to do that.

- unless explicitly stated by the TA, a lack of "pass" outcome does not mean "fail" ! (and vice versa). A TA may describe how to achieve both "pass" and "fail " outcomes, or just one of them.

- Also, successful production of the "test effect" or "test result" described in a TA could be associated with a "fail" outcome (negative testing) instead of a "pass". That is up to the TA to define the behavior/condition that defines either outcome ("pass", or "fail").

-Jacques

 

 

-----Original Message-----
From: Dave Pawson [mailto:dave.pawson@gmail.com]
Sent: Saturday, September 22, 2007 5:51 AM
To: tag@lists.oasis-open.org
Subject: Re: [tag] some thoughts and comments from reading a month worth ofmessages

On 22/09/2007, stephen.green@systml.co.uk <stephen.green@systml.co.uk> wrote:
> I agree that a test error is out of scope. If the test has an error
> that is hardly the concern of this generalized TA. It is something for
> the test itself to cover.
>
> I'm not sure that applies to a test being inapplicable though - the
> outcome of a condition not being met. So if conditions are not met in
> a flow the flow might stop or take a particular branch.

All that is internal to the test, hence the domain of the test developer.
At the TA level it is all part of the test.
A condition being met or not is just a test carried out in order to determine the test outcome.

If the
> conditions are not met in an overall TA the TA is inconclusive.

I disagree with that. If some interim test, internal to the defined TA/TEST fails, then it is likely that the entire TEST fails.
There are generally only two options. Pass and other. Anything other than a full pass is by default a TEST fail.

I'm using UC for the TEST as an entity. It may be implemented as 480 lines of code, but together it implements a TEST as agreed with the design authority.

I'm using a definition of TA as this level of TEST.



> So maybe we can define how a whole TA

Not until we agree at what level of abstraction the TA relates to a test (or TEST as I've used it above :-)


reaches results of pass, fail
> or inconclusive. Then we can say for a step in a flow the step value
> can be pass, fail or something else

You might. I never would. I'd consider that ineffective testing.
Pass or other|fail. No other choices, else go back and re-define the TEST!



> Can a TA overall evaluate to true/false or is it always going to be
> pass/fail/inconclusive (inconclusive being an outcome of conditions
> not being met on certain steps)?

Just pass and fail. Why has it failed? See the test results.



regards




--
Dave Pawson
XSLT XSL-FO FAQ.
http://www.dpawson.co.uk