test case outcomes: semantics

[Jacques Durand <JDurand@us.fujitsu.com>]

Title: test case outcomes: semantics

Miekand all:

here is a proposed insert on the meaning of test case outcomes:
For review
(note: there may be a blurred distinction between som cases of "Undetermined" and "system exception")

Jacques

Pass:
-----

Meaning:
The material under test (e.g. messages from the implementation under test) is satisfying
the test requirement associated with the test case.

Fail:
-----

Meaning:
The material under test (e.g. messages from the implementation under test)
does not satisfy the test requirement associated with the test case.

Action:
The expected remedial actions include fixing the implementation under test,
as this is a failure to conform to the specification as being verified by this test suite.

Undetermined:
-------------

Meaning:
The test case execution could not conclude by either a Fail or a Pass,
because of the irrelevance of the material under test, or because the right context for this test case could not
be created. These reasons could be:
(a)- the material under test is simply not relevant to the test requirement being tested, and a preliminary test assertion (acting as a pre-requisite or pre-condition) filtered this material out. (e.g. test case on conformance to "simple SOAP", will not be relevant to messages generated as SWA, and the test case may not be able to "force" the implementation under test to generate the former instgead of the latter.)

(b)- The environment of the test case other than the testbed itself (e.g. expected events from other parties assisting the test case, expected resources external to the testbed such as an error log site) cannot be obtained, or failed during execution, so that the outcome would not be meaningful.

Action:
The expected remedial actions include, for (a) possibly replay the test case (in case there are some chances the proper material can be obtained), and for (b) fixing the test case environment, possibly the test case itself, so that the test case execution can proceed.

System Exception:
----------------

Meaning:
The test case execution failed due to the inability of the testbed to complete its execution
because of problems related to the testbed itself. E.g., a connection between testbed components failed.
Or, material being manipulated (e.g. message data) is too large to be handled by this version of the tsetbed.

Action:
The expected remedial actions includes either upgrading / fixing the testbed, or modifying the testcases so that the material being handled can be processed by the testbed.