RE: [ebxml-iic] scripting of test cases: latest advances

[Jacques Durand <JDurand@us.fujitsu.com>]

Title: RE: [ebxml-iic] scripting of test cases: latest advances

A "precondition" is just a way to "guard" an assertion that
we don't want to test if precond is false. In which case, a likely
outcome for the test case is "undetermined".
But all this can now be controlled with just one op: "Assertion" which has been
generalized as a general verification operator, with "interpretation " attributes
(when_true, when_false)
if we want the above behavior, we can use:

<Assertion when_false="exitUndetermined" ...(verify the pre-cond) </Assertion>
<Assertion when_false="exitFail" when_true="exitPass" ...(verify main assertion) </Assertion>

So a  preCond element would just be syntactic sugar, and I personally believe that
it adds confusion. It is not like this would help to derive a test case straight from a test requ
that has indeed preconds and assertions... this is finer granularity, and orthogonal.

Jacques

-----Original Message-----
From: Monica J. Martin [mailto:Monica.Martin@sun.com]
Sent: Friday, May 28, 2004 6:52 PM
To: Jacques Durand
Cc: 'Michael Kass'; ebXML IIC - main list (E-mail) (E-mail);
tsakach@certivo.net
Subject: Re: [ebxml-iic] scripting of test cases: latest advances

>.Additionally, I believe we need to address
>     the <TestPreCondition> operation. It is semantically different
>     than <TestAssertion>. A precondition is something beyond the
>     control of the Test Driver and Test material. It is testable, but
>     we cannot fail a Test Case if that precondition does not exist.
>     Therefore, I believe we need to keep the <TestAssertion>
>     operation, and reduce its possible enumerated types for "onTrue"
>     and "onFalse" to "undetermined", "split"... since you would never
>     "pass" a Test Case based upon the existence of a precondition...,
>     nor would you "fail" a Test Case based upon a precondition not
>     being met.
>
>     [Jacques Durand] Couldn't we just consider the preCond (when we
>     need one) like being just another Assertion op that preceeds the
>     "normal" one, and with the right outcome
>     (when_false="undetermined" when_true="continue"), now that we have
>     this degree of control? (I feel that it can be confusing to
>     introduce a "preCOnd" to users: what special semantics can we give
>     to it that jsutifies a special operation, now that we can express
>     it with another Assertion op?)
>
>
>     I would suggest the "default" values for onTrue and onFalse for
>     <TestPreCondition> be "continue" and "undetermined".
>
>     Likewise, a <TestAssertion> operation is semantically different
>     from a TestPrecondition operation. And I believe the enumerated
>     list of values for "onTrue" and "onFalse" should be reduced. One
>     would not set a final state of a Test Case to "undetermined" based
>     upon the result of a <TestAssertion>. There are not ambiguities,
>     and no unsatisified preconditions at this point, so a Test Case
>     either "passes" or "fails" based upon that assertion. So I suggest
>     the possible values for "onTrue" and "onFalse" be "pass", "fail" ,
>     "continue" and "split".
>
>     Of course, using 2 optional attributes in our XML can create some
>     interesting paradoxes... i.e. onTrue="pass" and onFalse="pass"..
>     or onTrue="continue" onFalse="continue" ... again this means the
>     TestAssertion is meaningless.
>
mm1: Remember when we spoke about the fact that you would have a
negative test case that would raise an error, which is the expected
behavior? Isn't this a similar case? Therefore, if do we need another
type on the Test Assertion that gives meaning to the 'pass' (i.e. gives
it context)? Otherwise, the results could be misunderstood.

>     But perhaps we can live with this.. it's the semantic meaning of
>     <TestPreCondition> and <TestAssertion> that I am concerned with. I
>     definitely feel that we should limit the enumerated values of
>     those 2 test objects to meaningful values.
>
mm1: A pre-condition is meaningful, isn't it because the precondition
must exist for the conditions assumed in the Test Assertion are
available. As stated in the existing draft (from April, v1.1):

The TestPreCondition operation is semantically significant, in that a
“failure” to verify or validate the content of a message results in an
exit from the execution of the
Test Step with an exit condition of “undetermined” (meaning, because the
precondition could not be verified or validated, the ultimate result of
the Test Case could not be determined).

We have been speaking about external events that exist pre or post in
ebBP. We have not quantified specifically how those conditions affect
the process itself (or in your case a test step). However, the condition
makes it possible that the step to occur (it doesn't trigger it but
creates the condition for it to occur). If that is true, can we evaluate
the Test Assertion without determinism on the condition being present in
the preCondition?

Thanks.