Fwd: [tag-comment] Comment: include a means to reverse logic of a test assertion

[Stephen D Green <stephengreenubl@gmail.com>]

Jacques,

 

Just to clarify a typical use case I had in mind for negating a test assertion.

 

Looking at the WS-I Basic Profile 2.0 and its test assertions I noticed

that the Glossary definition includes for 'Reporting' the following

 

"NOTES: the predicate of the TA may be worded in a negative way so that @false='passed' although that is not recommended." 

 

(e.g. http://ws-i.org/Profiles/BasicProfile-2.0-2010-11-09.html#glossaryBP1904 )

 

I have come across in my own work cases where such negation seems desirable,

where there is scope to stretch the use of test assertions to this extent, although

such cases did occur in isolated systems when it wasn't necessary to conform

to the test assertion semantics. When I saw the WS-I glossary it seemed there may

be more widespread desire to stretch the semantics of the test assertion model,

albeit accepting it is not recommended.

Best regards

----

Stephen D Green

On 28 August 2012 00:05, Jacques Durand <JDurand@us.fujitsu.com> wrote:

Stephen:

 

We acknowledge the comment you made below.

 

Here is my personal take on your comment:

 

There are indeed cases where it is easier to demonstrate a “failure” than to demonstrate conformance.

 

> This would allow for assertions which are more easily stated by their negative.

You do not provide examples of this, so let me provide some in my view and start from there:

Let us consider the normative statement “all widgets must be painted blue.” Let us also assume that as far as testability goes, it is common knowledge that test equipment for widgets can only test for red and yellow colors, i.e. for “failure to be blue”, but no conclusive test exists for being blue.

Your request makes sense in situations where (a) there are known limits to testability of some test target for some requirement, that make it only possible to detect a subset of failure cases, but not to assert conformance to the requirement, (b) one wants to design test assertions that reflect this reality (instead of merely reflecting the property required in the normative statement, and let the test suite designer deal with the test implementation issue.)

I must say that in our methodology so far, we made the choice to have the TA predicate faithfully reflect the normative statement, so (b) is not really an option we favor.

However, nothing prevents us to use a combination of more than one TA to address a normative statement, while “interpreting” the normative source to indicate what aspect of the normative statement each TA is addressing, i.e. here:

TA”notRed”:

-          normSource (with “interpretation”): “an aspect of being “blue” is to NOT being red”

-          predicate: “the widget is NOT red” ,

TA”notYellow”:

-          normSource (with “interpretation”): “an aspect of being “blue” is to NOT being yellow”

-          predicate: “the widget is NOT yellow”

Note that these are not “reverted” TAs. They follow the expected logic: if predicate is false then the normative source interpretation is adhered with, otherwise is violated.

Then we can write a TA fully addressing the original normative statement , yet one we know will only be partially “testable”:

TA”Blue”:

-          normSource (with equivalent “interpretation”): “widget being “blue” is same as NOT being of any other known color.”

-          Predicate:  (TA”notRed” = ‘pass’) AND  (TA”notYellow” = ‘pass’) AND (widget is not of any other color)

Here the job of the TA writer is done: s/he has faithfully matched the normative requirements, yet in a way that will help the test suite writer. The test suite writer knows that in the current state of affair, TA”Blue” cannot be fully implemented YET can be partially implemented, thanks to the testability of “red” and “yellow”: its predicate will definitely fail the test if widget is red or yellow, yet cannot be evaluated by known test technology in other cases.

So the test suite writer , recognizing his/her inability to fully implement the TA”Blue”, should write for it a test case that will either return:

-          Fail (if either TA”notRed” or TA”notYellow” fail)

-          Inconclusive otherwise.

In other words: I take your suggestion Stephen as relevant to best practices (TA guidelines) and indeed a case that is worth addressing in the “guidelines” but not in the model so far. The current model appears to be able to address the “partial testability” concern as I understand it at least. I am afraid the “reverse logic” feature would open a can of worms that affect more deeply the model fundamentals…  If we need to go that route I think we need first a convincing set of use cases and if we can’t address it with TA model 1.0 then roll our sleeves for 2.0.

 

The TAG TC will nevertheless examine your comment and ways to address it.

 

Best regards,

 

Jacques Durand

TAG TC chair

 

From: Stephen D Green [mailto:stephengreenubl@gmail.com]
Sent: Thursday, August 02, 2012 7:38 AM
To: tag-comment@lists.oasis-open.org
Subject: [tag-comment] Comment: include a means to reverse logic of a test assertion

 

A comment on Test Assertions Model V1.0:

 

May I suggest (perhaps for a future minor version) that there be a

means in the model to indicate a reversal of the logic such that

a test assertion predicate can be expressed negatively where a

test outcome based on such a reversed test assertion is positive

when the test fails. This would allow for assertions which are

more easily stated by their negative. I suggest one way to do this

might be to extend the list of prescription levels to include negated

values (MandatoryNOT, PreferredNOT) but this does not work if

the prescription level is optional, so another means to indicate

negation of the semantics of a test assertion would be preferred.

 

Best regards,

----

Stephen D Green