[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Thread: TA Outcome (formerly: RE: [tag] some thoughts and comments from reading a month worth ofmessages)
My 2 cents on the outcome of a TA :
Agree
that only the overall result of a TA needs be described. But I would give this
result a clear meaning w/r to the specification requirement addressed by the
TA:
I see only two "conclusive" outcomes (w/r to this I agee with
Dave):
"pass": when the conditions and/or behavior
leading to "pass" outcome are met by a Test Case derived from this TA for a
given IUT (or UUT), then the IUT conforms to the specification requirement
addressed by this TA.
"fail": when the conditions and/or
behavior leading to "fail" outcome are met by a Test Case derived from this TA,
for a given IUT (or UUT), then the IUT does NOT conform to the specification
requirement addressed by this TA.
But I feel we also need an unconclusive
outcome at TA level:
"non-applicable": when the
conditions and/or behavior of the IUT (or of the system IUT+TestEnvironment?)
leading to the initiation of the actual test are NOT met, then this TA does not
apply to this IUT. No conclusion can be made w/r to the addressed specification
requirement. Note that this "non-application" is different from a failure
of the test environment itself: here it only means the IUT is not qualified,
while a test env failure means that some unexpected contingencies made the
testing impossible - an unexpected test case problem, not a definition of
the TA application scope.
NOTES:
- Every other "corner case" outcome or outcome variant (test error due to failure of the test procedure itself, various error messages associated with different types of failure) IMO are relevant to test cases and do not need be described in the TA.
- "warnings" and all kinds of severity levels do not need be described in a TA : either they are "fail" cases that are interpreted as non-critical for conformance - so this is a conformance assessment out of scope for us (e.g. when the spec reqt was a SHOULD, a "fail" TA outcome will translate into a "warning" in the final test report.) Or, I could imagine a warning being produced by the test case execution (orthogonal to the TA outcome), expressing for example some reduced testability due to some unfavorable input material or condition.
- I wouldn't try to go "fine-grain" in defining a TA
as a whole workflow of sub-steps with step-level outcomes, etc. But I
can imagine that there may be several ways to fail or to pass a TA, that need be
listed. So the TA model should allow to do that.
- unless explicitly
stated by the TA, a lack of "pass" outcome does not mean "fail" ! (and vice
versa). A TA may describe how to achieve both "pass" and "fail " outcomes, or
just one of them.
- Also, successful production of the "test effect" or "test result" described in a TA could be associated with a "fail" outcome (negative testing) instead of a "pass". That is up to the TA to define the behavior/condition that defines either outcome ("pass", or "fail").
-Jacques
-----Original Message-----
From: Dave Pawson [mailto:dave.pawson@gmail.com]
Sent:
Saturday, September 22, 2007 5:51 AM
To: tag@lists.oasis-open.org
Subject:
Re: [tag] some thoughts and comments from reading a month worth
ofmessages
On 22/09/2007, stephen.green@systml.co.uk
<stephen.green@systml.co.uk> wrote:
> I agree that a test error is
out of scope. If the test has an error
> that is hardly the concern of
this generalized TA. It is something for
> the test itself to
cover.
>
> I'm not sure that applies to a test being inapplicable
though - the
> outcome of a condition not being met. So if conditions are
not met in
> a flow the flow might stop or take a particular
branch.
All that is internal to the test, hence the domain of the test
developer.
At the TA level it is all part of the test.
A condition being
met or not is just a test carried out in order to determine the test
outcome.
If the
> conditions are not met in an overall TA the TA is
inconclusive.
I disagree with that. If some interim test, internal to the
defined TA/TEST fails, then it is likely that the entire TEST fails.
There
are generally only two options. Pass and other. Anything other than a full pass
is by default a TEST fail.
I'm using UC for the TEST as an entity. It may
be implemented as 480 lines of code, but together it implements a TEST as agreed
with the design authority.
I'm using a definition of TA as this level of
TEST.
> So maybe we can define how a whole TA
Not until
we agree at what level of abstraction the TA relates to a test (or TEST as I've
used it above :-)
reaches results of pass, fail
> or
inconclusive. Then we can say for a step in a flow the step value
> can be
pass, fail or something else
You might. I never would. I'd consider that
ineffective testing.
Pass or other|fail. No other choices, else go back and
re-define the TEST!
> Can a TA overall evaluate to true/false
or is it always going to be
> pass/fail/inconclusive (inconclusive being
an outcome of conditions
> not being met on certain steps)?
Just
pass and fail. Why has it failed? See the test
results.
regards
--
Dave Pawson
XSLT
XSL-FO FAQ.
http://www.dpawson.co.uk
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]