OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

ebxml-iic message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: RE: [ebxml-iic] scripting of test cases: latest advances


Title: scripting of test cases: latest advances
Mike:
 
sounds good. inline
-----Original Message-----
From: Michael Kass [mailto:michael.kass@nist.gov]
Sent: Wednesday, May 26, 2004 3:57 PM
To: Jacques Durand; ebXML IIC - main list (E-mail) (E-mail)
Cc: tsakach@certivo.net
Subject: RE: [ebxml-iic] scripting of test cases: latest advances

Jacques and all,

 

 

    Here are my comments

 

Regards,

Mike

- Branching (like needed in Use Case #3) will be done by invoking threads by name, within a split() op.

 

[MIKE] - We may just be able to code:

<TestAssertion whenTrue="ThreadRef01">....</TestAssertion>   ... (and if we wish to do more complicated choreography (like split a series of Threads), place that choreography inside of Thread01.. comments?  That way we keep the scripting syntax simple.


[Jacques Durand]  I thought doing that too, but then we have to be clear of the semantics of starting a thread that way: since it can be done any place within a thread (not just the end), it actually should have the semantics of: split(ThreadRef01) (i.e. the main thread keeps going concurrently, unless the next op after this is a join(ThreadRef01). So we may keep teh above notation, but it has to be clear its a short for split(ThreadRef01).


- A thread, when invoked (whether within or outside a conditional statement), will inherit visibility of parameters from the invoking thread.

 

[MIKE] - This is true. Additionally, if the Thread were invoked from within a Test Step, then it would be passed the parameters of the Test Step.  I think that it would be useful to have a "scope" attribute for any parameter defined via <SetParameter>.  For example, we could have a <SetParameter scope="self"> (meaning scope is limited to the TestStep or Thread in which it is defined, and is undefined/unavailable to any ancestor or descendant Threads and Test Steps) or <SetParameter scope="selfAndDescendants", meaning the parameter's value can be passed to any descendant Thread and its Test Steps.  Doing this would avoid inadvertent side-effects by passing parameters whose only purpose is for the current Test Step to descendent Threads, where its value may be inadvertently substituted as message content to be sent or as a comparison parameter used to verify received message content.
 
[Jacques Durand] do we have to worry about Test Step scope? So far we only spawn threads from a split op, or now from Assertion op. 

-        Because we did not want to spawn a thread from within a test step, and for other reasons, we came to the conclusion that it is better to extract the "Assertion" test outside of the GetMessage step.

 

[MIKE] - Removing TestAssertion from within a GetMessage "operation" makes sense, since we may Test Assertions without having to evaluate message content.  This does not put TestAssertion outside of a <TestStep> element however.. and there may be a need for it to be available even at the <Thread> level (see "looping" comments below)

[Jacques Durand]  Wouldn't TestAssertion be considered as just another (fancy) workflow operator? At same level as split()? (We have to clarify what TestStep means)

 

<TestStep>

<GetMessage>

                        <Filter> .. some XPath expression </Filter>

            </GetMessage>

<TestAssertion>

            <VerifyContent> .. some XPath expression </VerifyContent>

</TestAssertion>

</TestStep>
[Jacques Durand] Again, we need to justify the logic of what is a Test Step. I'd say it is an atomic unit of execution, but it is NOT a control flow operator like split, or now Assertion (e.g. cannot branch somewhere else). Should it include sleep()?  

[MIKE] - Making <TestAssertion> available at the <Thread> level wouldn't make sense from the logical standpoint that <Threads> choreograph <Threads> and <TestStep>s... and <TestStep>s   is where the message sending, receiving and assertion evaluation occur.

-        The Assertion becomes a control flow operator: it may test FilterOutput material from several GetMessage inputs (these need be named), or may test value of variables (no message data).

 

[MIKE] - Yes the assertion becomes the key control flow operator now.. since we no longer evaluate the boolean result of a <TestStep>.. or its <Thread> container...  It tests FilterResult from a "SINGLE" GetMessage <Filter> operation (the current <Filter>.. although many <Filter> operations are possible within a <TestStep>).

-        The outcome of the assertion test will be stated explicitly (e.g. might be indicated via two attributes: "when_true", "when_false" that are just a representation for if..then...else, without the composability.)

[MIKE] - I think that a reference to a single Thread using "whenTrue" and/or "whenFalse" may be the way to do this

-       These statements effectively decouple the boolean result of an assertion, with the test case outcome.

[MIKE] - I'm not sure what you mean there, since the boolean result of a <TestAssertion> is fundamental to the test case outcome in your proposal, when coupled with the enumerated attribute values you describe below.  

[Jacques Durand]  Yes, I only meant that a boolean result "false" does not automatically mean "fail". These new attributes can map a true/false result, to any of these test case outcomes: fail/pass/undetermined/continue.

This was needed to handle "error-catching threads" that must fail the case when assertion is true.
- The Assertion outcome (e.g. specified as value of above attributes) may be:
. exit(...) with arg = fail/pass/undetermined. This terminates the test case.
. continue (default). This let the test case continue its flow (next test step, thread join, etc.)
. split(threadA [,...]). This spawns a thread concurrently to current thread.

[MIKE] - So do we need to provide a "whenTrue" and "whenFalse" attribute for every TestAssertion?  Shouldn't we have an "implied" Test Driver behavior if one or both of these attributes aren't' present? I would think the default attribute values should be whenTrue="continue"  whenFalse="fail".   

[Jacques Durand]  that can be the default. (though you note that a test case would never "pass" with these defaults, unless we say that if it terminates gracefully, it passes.) Any outcome that we need to be different from these, needs the attribute reset explicitly.

Additionally, I believe we need to address the <TestPreCondition> operation. It is semantically different than <TestAssertion>.  A precondition is something beyond the control of the Test Driver and Test material.  It is testable, but we cannot fail a Test Case if that precondition does not exist.  Therefore, I believe we need to keep the <TestAssertion> operation, and reduce its possible enumerated types for "onTrue" and "onFalse" to "undetermined", "split"... since you would never "pass" a Test Case based upon the existence of a precondition..., nor would you "fail" a Test Case based upon a precondition not being met.

[Jacques Durand] Couldn't  we just consider the preCond (when we need one) like being just another Assertion op that preceeds the "normal" one, and with the right outcome (when_false="undetermined" when_true="continue"), now that we have this degree of control? (I feel that it can be confusing to introduce a "preCOnd" to users: what special semantics can we give to it that jsutifies a special operation, now that we can express it with another Assertion op?)


 I would suggest the "default" values for onTrue and onFalse for <TestPreCondition> be "continue" and "undetermined".

Likewise, a <TestAssertion> operation is semantically different from a TestPrecondition operation. And I believe the enumerated list of values for "onTrue" and "onFalse" should be reduced.  One would not set a final state of a Test Case to "undetermined" based upon the result of a <TestAssertion>.  There are not ambiguities, and no unsatisified preconditions at this point, so a Test Case either "passes" or "fails" based upon that assertion.  So I suggest the possible values for "onTrue" and "onFalse" be "pass", "fail" , "continue" and "split".

Of course, using 2 optional attributes in our XML can create some interesting paradoxes... i.e.  onTrue="pass" and onFalse="pass".. or onTrue="continue" onFalse="continue" ... again this means the TestAssertion is meaningless.  But perhaps we can live with this.. it's the semantic meaning of <TestPreCondition> and <TestAssertion> that I am concerned with.  I definitely feel that we should limit the enumerated values of those 2 test objects to meaningful values.

Lastly, as I understand it, if the Test Case proceeds and finishes execution to the end of its logic flow, and the last <TestAssertion> evaluated to "true", then the Test Case passes.  This also assumes all Threads run to completion.  If any Thread has not completed execution, then I would think that the Test Case result would be "undetermined".

[Jacques Durand] This kind of implicit semantics becomes complex... As best practice, I believe every final  outcome should result from an explicit exit statement, even the last one. At best, the kind of implicit outcome we could keep is: when the entire test case is over (all its threads are over) meaning that no exit has been made on either pass/fail/undetermined, then it is a pass.


- Timeout may be simply handled by: (1) spawning a sleeping thread (sleep maxtime), (2) setting a flag variable when the last step is complete, (3) sleeping thread checks the status of the flag after sleeping, and makes decision (e.g. exit) if failure. This allows for any pattern of time checks (intertwined, etc.)

[MIKE] - This is definitely useful.

- Looping ("while...do") can be handled by a thread invoking itself recursively, from an Assertion op.

[MIKE] - So a construct such as:



<Thread id="Thread01">

<TestAssertion split="Thread01">

..... (some assertion) ....

</TestAssertion>

</Thread>

 

should work.  That means we need to make <TestAssertion> available at the <Thread> level (currently it is defined as a child of the <TestStep> element only.  I don't think that that will be a problem.

[Jacques Durand] Yes (wouldn't the recusrive branching occur within one of these "when_true" attribute?)

 

Jacques

 



[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]