Re: [ebxml-iic] call tomorrow Wed 8th, 3pm PT

Jacques,

I modified section 7.1 based on comments from last conference call (changed structure/design of logic flow HTML table. Plus I modified the “pass”, “fail”, “undetermined” description at the end. Please see attached document (turn on "tracking" to see changes).

It seems that the last sticking point is the “default” behavior of a Test Driver when encountering a “failed” TestAssertion. Please see my comments below:

[Durand]

It is annoying that the interpretation of an assertion failure (what to do if fails) , is context-dependent (presence of a step behind or not…) Shouldn’t we say that an assertion failure just aborts the thread, in the absence of any other explicit exit statement?

[MIKE] – Why can’t there be a “default” behavior for a failed TestAssertion. The fundamental meaning of TestAssertion is just that.. testing an assertion. Generally, if such an assertion is not met, this means the implementation failed. However, we have created a more flexible meaning..saying that one can now “continue” from a failed TestAssertion..or conceivably even “pass” after failing a TestAsssertion operation. I believe that the meaning has been “watered down” to the point where it is now nothing more than a logic flow operator. I believe that the default behavior should be “fail” for a “failed” TestAssertion. One can still “override” it using the <Return> instruction ( or <Abort>… although I think that you suggested that we not use <Abort>). I am in favor of trying to keep close to the meaning of <TestAssertion> as we originally defined it.

[Durand]

This thread may be or-joined later and the entire test case may still succeed thanks to other concurrent threads that have successfully completed.

[MIKE] – I’m OK with that. I just do not think that <Abort> should be the default Test Driver behavior when a TestAssertion fails. In our 216 ebMS Test Cases, that would result in 216 “undetermined” Test Case results.

Can’t we say that an aborted test case wil be by default interpreted as “Undetermined”?

[MIKE] – An “aborted” Test Case should result in an “undetermined” result. But I do not believe that a “false” TestAssertion result should not (by default) <Abort> a Test Case.

Below are the rules added to the end of section 7.1 for your review

A “default” Test Case state of "continue" exists after a TestAssertion operation returns a boolean result of "true" , or a “Return” instruction is encountered by the Test Driver, indicating that the current Thread MUST be terminated, but that logic flow will continue within the “calling” Thread.

A final Test Case state of "pass" occurs when the Test Driver encounters an explicit "exit/pass" instruction from within a TestAssertion, OR logical testing execution proceeds from beginning to end without:

a) An explicit "exit/fail" or "exit/undetermined"

b) A system "exception" condition

c) A failed Join that precludes further testing execution beyond the Join

A final Test Case state of "fail" is given to a TestCase where :

a) A TestAssertion boolean operation returns a result of "false" (default behavior) (i.e. it is assumed that a TestAssertion is a meaningful measurement of conformance/interoperability that MUST pass.. unless explcitly overriden by the test writer in such cases where a TestAssertion verifies a "precondition" to further testing, in which case the test writer may wish to "exit" with a state of "undetermined". Additionally, the test writer may with to "continue" if the failed result of the TestAssertion is used to alter the flow of Test Case execution.

b) The Test Driver encounters an explicit "exit/fail" instruction within a TestAssertion operation.

A final Test Case state of "undetermined" occurs if:

1) An explicit "exit/undetermined" instruction occurs from a TestAssertion operation

2) Testing logic flow cannot proceed beyond a Join due to failed andJoin or orJoin condition

Comments?

Mike

ebxml-iic message