On non-passage and non-failure

[david_marston@us.ibm.com]

First of all, I want to remind you that test assertions (TAs) will be
very useful for testing situations other than conformance, even if they
are written with conformance in mind. So the notion of "pass" equauting
to "conforms" should not be deeply baked into the structure or other
definitive aspects of TAs.

The idea of "unable to evaluate" pass/fail as a third outcome state
cannot be truly eliminated, it can only be pushed into one corner or
another. As I see it, there are three scenarios in contention:

1. Reducing the test result to an "outcome" can produce at least the
three outcomes pass, fail, or unable to evaluate, if not more.

2. Attempt to have only pass and fail as outcomes, and avoid running
any test where you can tell ahead of time that you would not get a
clean pass/fail result. One situation in which such avoidance would
take place is when the preconditions (antecedents) cannot be set up.

3. Run whatever tests you wish, but then refuse to assess the outcome
of any test where you determine that the antecedents were actually
not obtained. Presumably, every antecedent would itself be the result
of some other test that is closer to atomic. If the test of the
antecedent fails, then all cases that need that antecedent are
eliminated from pass/fail evaluation.

Note that scenarios 2 and 3, when applied to conformance, require that
you have a provably correct list of "all the tests that should have
passed" because you want to ensure that you don't say the IUT conforms
when many tests were eliminated.

I think scenario 2 has many operational difficulties, especially
involving that issue of sequencing test cases. The precondition may
be satisfied in more than one way, so you don't want to say that a
particular test is a precondition, but rather any means of obtaining
the precondition will do.

EXAMPLE:
Suppose that the spec has a normative statement that says, "The plus
operator, when applied to an integer and a float, yields a float as
a result." You devise the test case
2 + 2.0 = 4.0
where the result is the type-test on the sum (4.0), which must obtain
"float" as the type.

There are also some other normative statements that get put to use:
Every integer can be converted to an equal float value (no integers
are unconvertible). A string of digits without a decimal point is
type-annotated as integer. A string of digits with one decimal point
is type-annotated as float. Float+float yields float, etc.

Now suppose that a particular implementation has the flaw that the
sub-expression "2" gets type-annotated as float rather than integer.
None of the other normative statements is violated. When run, the
test case yields 4.0, type-annotated as float. But the pass/fail
outcome should not be evaluated, because the implementation did not
really perform integer+float, which was the point of the test.

If we follow scenario 3, then there should be another test case that
just does a type-test on the expression "2" and this implementation
will fail that case outright. If the original test marked passage of
this simpler test as one of its preconditions, we would know to not
evaluate the result.

Scenario 2 could also be adjusted to fit, but consider that another
way to obtain the integer 2 might be available:
string-length("ab")
and another case, if you remembered to write it, could test
type-test(string-length("ab")+2.0)
expecting "float" as the result. Pretty soon, you get a complex mesh
of test sequencing, when it would be much easier to just run the whole
set of tests and post-process to not evaluate certain tests, or go
back to scenario 1 and designate their outcome as unable to evaluate.
.................David Marston