RE: [ebxml-iic] scripting of test cases: latest advances

[MIKE2] - I agree with you. Using an attribute to express this is not clear. So in the revised schema, I used the <Split> syntax from a <TestAssertion> to maintain that clarity.
[Jacques Durand 2] ideally, we should be able to use as value for these attributes, the same control statements we use at thread level: split(), exit(). Question: will we also be able to just "exit(pass)" or "exit(fail)" as standalone statements in a test case script, without an Assertion, e.g. we could imagine some test case where reaching the last step is good enough to exit(pass) or exit(fail), because the decision to pass or fail has been made in an earlier step before, yet we need to do some additional "clean-up" step, e.g. a PutMessage op for complying with a protocol expected on the other side, before we terminate. So the exit statement is not combined with any Assertion in that case.

- A thread, when invoked (whether within or outside a conditional statement), will inherit visibility of parameters from the invoking thread.

[MIKE] - This is true. Additionally, if the Thread were invoked from within a Test Step, then it would be passed the parameters of the Test Step. I think that it would be useful to have a "scope" attribute for any parameter defined via <SetParameter>. For example, we could have a <SetParameter scope="self"> (meaning scope is limited to the TestStep or Thread in which it is defined, and is undefined/unavailable to any ancestor or descendant Threads and Test Steps) or <SetParameter scope="selfAndDescendants", meaning the parameter's value can be passed to any descendant Thread and its Test Steps. Doing this would avoid inadvertent side-effects by passing parameters whose only purpose is for the current Test Step to descendent Threads, where its value may be inadvertently substituted as message content to be sent or as a comparison parameter used to verify received message content.

[Jacques Durand] do we have to worry about Test Step scope? So far we only spawn threads from a split op, or now from Assertion op.

[MIKE2] - Test Step scooping appears to be important, if we agree that a Test Step is a series of message sends/receives for a common type of transaction (i.e. Purchase Order). Setting a $CPAId or $ConversationId value may only be appropriate for that Purchase Order transaction, but may be inappropriate for subsequent Test Steps in a Thread. If we treat all parameter definitions as "global" to the Thread in which they are defined, then collisions between sibling TestSteps can occur.

[MIKE2] - Thinking about this more, I am wondering if we should just "do away" with <TestStep> entirely, and treat everything as a <Thread> (i.e. merge the operations of a <TestStep> in to <Thread>). While this removes the meaning of a <TesStep> as a container for common message request/receive operations and test assertion verification, it would simplify the testing model even more. Plus it would permit a simpler syntax for recursion:
[Jacques Durand 2] If a Test Step is a container that can be as complex as a thread, then lets do away with it ! I thought it was jsut for one message op at a time, either GetMessage or PutMessage, but no more than one. I think it is still useful to have a notion of container even for atomic operations, because we can attach some properties to it that can then be common to any contained op, like timeout, etc. So I'd keep the notion of TestStep, but only containing one main op statement at most, plus some variable setting statements if needed?

<IfTrue><ThreadRef>Thread01</ThreaRef>

</TestAssertion>

</Thread>

rather than:

<IfTrue><ThreadRef>Thread01</ThreaRef></IfTrue>

</TestAssertion>

</TestStep>

</Thread>

[MIKE2] - This avoids one level (Test Step) of indirection to do recursion. Comments?

[Jacques Durand 2] If our Assertion becomes a 1st class control flow op like split(), then it shold anyway be outside a Test Step. That is, I thought the statement:

<TestAssertion whenTrue="ThreadRef01">....</TestAssertion>
would be at same level in a thread as a split().

- Because we did not want to spawn a thread from within a test step, and for other reasons, we came to the conclusion that it is better to extract the "Assertion" test outside of the GetMessage step.

[MIKE] - Removing TestAssertion from within a GetMessage "operation" makes sense, since we may Test Assertions without having to evaluate message content. This does not put TestAssertion outside of a <TestStep> element however.. and there may be a need for it to be available even at the <Thread> level (see "looping" comments below)

[Jacques Durand] Wouldn't TestAssertion be considered as just another (fancy) workflow operator? At same level as split()? (We have to clarify what TestStep means)

[MIKE2] - Yes, that is the intention.

<Filter> .. some XPath expression </Filter>

</GetMessage>

<VerifyContent> .. some XPath expression </VerifyContent>

</TestAssertion>

</TestStep>
[Jacques Durand] Again, we need to justify the logic of what is a Test Step. I'd say it is an atomic unit of execution, but it is NOT a control flow operator like split, or now Assertion (e.g. cannot branch somewhere else). Should it include sleep()?

[MIKE2] - Under our new definition, <TestStep> is no longer an "atomic" level of execution.. since we can now <Split> Threads from within a TestStep. This blurs the distinction between <Thread> and <TestStep>, and raises the question if we should not simply have <Threads> alone, capable of doing all the operations of a TestStep.
[Jacques Durand 2] so again if we take Assertion (or TestAssertion) out of a tset step, as a standalone control operator for branching or exiting, then we would not split() in any way from inside a test Step, and Test Step could remain an atomic unit of execution.

] ...

[MIKE] - So do we need to provide a "whenTrue" and "whenFalse" attribute for every TestAssertion? Shouldn't we have an "implied" Test Driver behavior if one or both of these attributes aren't' present? I would think the default attribute values should be whenTrue="continue" whenFalse="fail".

[Jacques Durand] that can be the default. (though you note that a test case would never "pass" with these defaults, unless we say that if it terminates gracefully, it passes.) Any outcome that we need to be different from these, needs the attribute reset explicitly.

[MIKE2] - Actually , I should have said: be whenTrue="continue" whenFalse="exitFail". This would seem to eliminate any ambiguity in Test Case result, as well as provide an implicit/unambiguous Test Driver behavior, since in our new schema, we can either "continue", "split" or "exit" based upon a false result.

[Jacques Durand 2] Right.

Additionally, I believe we need to address the <TestPreCondition> operation. It is semantically different than <TestAssertion>. A precondition is something beyond the control of the Test Driver and Test material. It is testable, but we cannot fail a Test Case if that precondition does not exist. Therefore, I believe we need to keep the <TestAssertion> operation, and reduce its possible enumerated types for "onTrue" and "onFalse" to "undetermined", "split"... since you would never "pass" a Test Case based upon the existence of a precondition..., nor would you "fail" a Test Case based upon a precondition not being met.

[Jacques Durand] Couldn't we just consider the preCond (when we need one) like being just another Assertion op that preceeds the "normal" one, and with the right outcome (when_false="undetermined" when_true="continue"), now that we have this degree of control? (I feel that it can be confusing to introduce a "preCOnd" to users: what special semantics can we give to it that jsutifies a special operation, now that we can express it with another Assertion op?)

[MIKE2] - Mike. We'll break backward compatibility with V1.0 Test Suites if we eliminate <TestPreCondition>.. but perhaps we should do this to simplify more.

[Jacques Durand 2] I think backward com is probably broken anyway? Plus if it were a real concern, test case scripts could be converted using some XSLT from 1.0 to 1.1.

I would suggest the "default" values for onTrue and onFalse for <TestPreCondition> be "continue" and "undetermined".

Likewise, a <TestAssertion> operation is semantically different from a TestPrecondition operation. And I believe the enumerated list of values for "onTrue" and "onFalse" should be reduced. One would not set a final state of a Test Case to "undetermined" based upon the result of a <TestAssertion>. There are not ambiguities, and no unsatisified preconditions at this point, so a Test Case either "passes" or "fails" based upon that assertion. So I suggest the possible values for "onTrue" and "onFalse" be "pass", "fail" , "continue" and "split".

Of course, using 2 optional attributes in our XML can create some interesting paradoxes... i.e. onTrue="pass" and onFalse="pass".. or onTrue="continue" onFalse="continue" .... again this means the TestAssertion is meaningless. But perhaps we can live with this.. it's the semantic meaning of <TestPreCondition> and <TestAssertion> that I am concerned with. I definitely feel that we should limit the enumerated values of those 2 test objects to meaningful values.

Lastly, as I understand it, if the Test Case proceeds and finishes execution to the end of its logic flow, and the last <TestAssertion> evaluated to "true", then the Test Case passes. This also assumes all Threads run to completion. If any Thread has not completed execution, then I would think that the Test Case result would be "undetermined".

[Jacques Durand] This kind of implicit semantics becomes complex... As best practice, I believe every final outcome should result from an explicit exit statement, even the last one.

[MIKE2] - But we can't be "ambiguous" with testing results, so our Test Driver behavior must be deterministic. I believe that it will be with regard to TestAssertion results with a default behavior of onTrue="continue", onFalse="exitFail". As far as "unfinished" Thread results, I would think that all threads MUST complete to make a final test case pass/fail call.
[Jacques Durand 2] Right, as far as no thread has explicitly exited, then ALL threads must complete to claim theh test case is over. (Timeouts may help this). But in case ANY thread , at any time, exits on pass or fail, then that will enough to decide of the test case outcome, even if there are still pending threads out there for this tset case. This is how "exception threads" will work: if they catch an error message, they exit(fail), even though the regular main thread is still waiting for a regular message...

ebxml-iic message