RE: [ebxml-iic] scripting of test cases: latest advances

Jacques and all,

Here are my comments

Regards,

Mike

-----Original Message-----
From: Jacques Durand [mailto:JDurand@us.fujitsu.com]
Sent: Tuesday, May 25, 2004 3:11 PM
To: ebXML IIC - main list (E-mail) (E-mail)
Cc: 'tsakach@certivo.net'
Subject: [ebxml-iic] scripting of test cases: latest advances

We had a productive call yesterday, and also discussion after the call.
I think we outlined a scripting solution that can handle every use cases properly, BPSS reqs,
while keeping the scripting simple.
Here is my summary of the consensus we reached, of course this needs to be validated
after a detailed review and use case re-coding. Mike please correct if needed:

Based on review of the three use cases and previous scripting:

- When branching is required based on received message: either header data or payload data can be used. But only one GetMessage op execution should be needed (not one per condition outcome).

[MIKE] – This is the case. Branching would occur based upon the <TestAssertion> result, which bases its assertion on the message content retrieved by the <FilterResult> obtained from the single <GetMessage> (or possibly its child <GetPayload> operation). TestAssertions using XPath can only be executed on XML payloads.

- Branching (like needed in Use Case #3) will be done by invoking threads by name, within a split() op.

[MIKE] – We may just be able to code:

<TestAssertion whenTrue=”ThreadRef01”>….</TestAssertion> … (and if we wish to do more complicated choreography (like split a series of Threads), place that choreography inside of Thread01.. comments? That way we keep the scripting syntax simple.

- A thread, when invoked (whether within or outside a conditional statement), will inherit visibility of parameters from the invoking thread.

[MIKE] – This is true. Additionally, if the Thread were invoked from within a Test Step, then it would be passed the parameters of the Test Step. I think that it would be useful to have a “scope” attribute for any parameter defined via <SetParameter>. For example, we could have a <SetParameter scope=”self”> (meaning scope is limited to the TestStep or Thread in which it is defined, and is undefined/unavailable to any ancestor or descendant Threads and Test Steps) or <SetParameter scope=”selfAndDescendants”, meaning the parameter’s value can be passed to any descendant Thread and its Test Steps. Doing this would avoid inadvertent side-effects by passing parameters whose only purpose is for the current Test Step to descendent Threads, where its value may be inadvertently substituted as message content to be sent or as a comparison parameter used to verify received message content.

- Because we did not want to spawn a thread from within a test step, and for other reasons, we came to the conclusion that it is better to extract the "Assertion" test outside of the GetMessage step.

[MIKE] – Removing TestAssertion from within a GetMessage “operation” makes sense, since we may Test Assertions without having to evaluate message content. This does not put TestAssertion outside of a <TestStep> element however.. and there may be a need for it to be available even at the <Thread> level (see “looping” comments below)

<Filter> .. some XPath expression </Filter>

</GetMessage>

<VerifyContent> .. some XPath expression </VerifyContent>

</TestAssertion>

</TestStep>

[MIKE] - Making <TestAssertion> available at the <Thread> level wouldn’t make sense from the logical standpoint that <Threads> choreograph <Threads> and <TestStep>s… and <TestStep>s is where the message sending, receiving and assertion evaluation occur.

- The Assertion becomes a control flow operator: it may test FilterOutput material from several GetMessage inputs (these need be named), or may test value of variables (no message data).

[MIKE] – Yes the assertion becomes the key control flow operator now.. since we no longer evaluate the boolean result of a <TestStep>.. or its <Thread> container… It tests FilterResult from a “SINGLE” GetMessage <Filter> operation (the current <Filter>.. although many <Filter> operations are possible within a <TestStep>).

- The outcome of the assertion test will be stated explicitly (e.g. might be indicated via two attributes: "when_true", "when_false" that are just a representation for if..then...else, without the composability.)

[MIKE] – I think that a reference to a single Thread using “whenTrue” and/or “whenFalse” may be the way to do this

- These statements effectively decouple the boolean result of an assertion, with the test case outcome.

[MIKE] – I’m not sure what you mean there, since the boolean result of a <TestAssertion> is fundamental to the test case outcome in your proposal, when coupled with the enumerated attribute values you describe below.

This was needed to handle "error-catching threads" that must fail the case when assertion is true.
- The Assertion outcome (e.g. specified as value of above attributes) may be:
. exit(...) with arg = fail/pass/undetermined. This terminates the test case.
. continue (default). This let the test case continue its flow (next test step, thread join, etc.)
. split(threadA [,...]). This spawns a thread concurrently to current thread.

[MIKE] – So do we need to provide a “whenTrue” and “whenFalse” attribute for every TestAssertion? Shouldn’t we have an “implied” Test Driver behavior if one or both of these attributes aren’t’ present? I would think the default attribute values should be whenTrue=”continue” whenFalse=”fail”.

Additionally, I believe we need to address the <TestPreCondition> operation. It is semantically different than <TestAssertion>. A precondition is something beyond the control of the Test Driver and Test material. It is testable, but we cannot fail a Test Case if that precondition does not exist. Therefore, I believe we need to keep the <TestAssertion> operation, and reduce its possible enumerated types for “onTrue” and “onFalse” to “undetermined”, “split”… since you would never “pass” a Test Case based upon the existence of a precondition…, nor would you “fail” a Test Case based upon a precondition not being met.

I would suggest the “default” values for onTrue and onFalse for <TestPreCondition> be “continue” and “undetermined”.

Likewise, a <TestAssertion> operation is semantically different from a TestPrecondition operation. And I believe the enumerated list of values for “onTrue” and “onFalse” should be reduced. One would not set a final state of a Test Case to “undetermined” based upon the result of a <TestAssertion>. There are not ambiguities, and no unsatisified preconditions at this point, so a Test Case either “passes” or “fails” based upon that assertion. So I suggest the possible values for “onTrue” and “onFalse” be “pass”, “fail” , “continue” and “split”.

Of course, using 2 optional attributes in our XML can create some interesting paradoxes… i.e. onTrue=”pass” and onFalse=”pass”.. or onTrue=”continue” onFalse=”continue” … again this means the TestAssertion is meaningless. But perhaps we can live with this.. it’s the semantic meaning of <TestPreCondition> and <TestAssertion> that I am concerned with. I definitely feel that we should limit the enumerated values of those 2 test objects to meaningful values.

Lastly, as I understand it, if the Test Case proceeds and finishes execution to the end of its logic flow, and the last <TestAssertion> evaluated to “true”, then the Test Case passes. This also assumes all Threads run to completion. If any Thread has not completed execution, then I would think that the Test Case result would be “undetermined”.

- Timeout may be simply handled by: (1) spawning a sleeping thread (sleep maxtime), (2) setting a flag variable when the last step is complete, (3) sleeping thread checks the status of the flag after sleeping, and makes decision (e.g. exit) if failure. This allows for any pattern of time checks (intertwined, etc.)

[MIKE] – This is definitely useful.

- Looping ("while...do") can be handled by a thread invoking itself recursively, from an Assertion op.

[MIKE] – So a construct such as:

….. (some assertion) ….

</TestAssertion>

</Thread>

should work. That means we need to make <TestAssertion> available at the <Thread> level (currently it is defined as a child of the <TestStep> element only. I don’t think that that will be a problem.

Jacques

ebxml-iic message