[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: [ebxml-iic] scripting of test cases: latest advances
|
-----Original
Message----- Mike: sounds good. inline -----Original Message----- Jacques and all,
Here are my comments Regards, Mike - Branching
(like needed in Use Case #3) will be done by invoking threads by name, within a
split() op. [MIKE] - We may just be able to code: <TestAssertion
whenTrue="ThreadRef01">....</TestAssertion> ... (and if we wish to do more
complicated choreography (like split a series of Threads), place that
choreography inside of Thread01.. comments? That way we keep the scripting syntax simple.
[MIKE2] –
I agree with you. Using an attribute
to express this is not clear. So in the revised schema, I used the
<Split> syntax from a <TestAssertion> to maintain that clarity.
[MIKE] - This is true. Additionally, if the Thread were invoked from
within a Test Step, then it would be passed the parameters of the Test
Step. I think that it would be
useful to have a "scope" attribute for any parameter defined via
<SetParameter>. For example,
we could have a <SetParameter scope="self"> (meaning scope is
limited to the TestStep or Thread in which it is defined, and is
undefined/unavailable to any ancestor or descendant Threads and Test Steps) or
<SetParameter scope="selfAndDescendants", meaning the parameter's
value can be passed to any descendant Thread and its Test Steps. Doing this would avoid inadvertent
side-effects by passing parameters whose only purpose is for the current Test
Step to descendent Threads, where its value may be inadvertently substituted as
message content to be sent or as a comparison parameter used to verify received
message content. [MIKE2] –
Test Step scooping appears to be important, if we agree that a Test Step is a
series of message sends/receives for a common type of transaction (i.e.
Purchase Order). Setting a $CPAId
or $ConversationId value may only be appropriate for that Purchase Order
transaction, but may be inappropriate for subsequent Test Steps in a
Thread. If we treat all parameter
definitions as “global” to the Thread in which they are defined, then
collisions between sibling TestSteps can occur. [MIKE2] –
Thinking about this more, I am wondering if we should just “do away” with <TestStep> entirely,
and treat everything as a <Thread> (i.e. merge the operations of a
<TestStep> in to <Thread>).
While this removes the meaning of a <TesStep> as a container for common message request/receive
operations and test assertion verification, it would simplify the testing model
even more. Plus it would permit a
simpler syntax for recursion: <Thread
id=”Thread01”> <TestAssertion> <IfTrue><ThreadRef>Thread01</ThreaRef> </TestAssertion> </Thread> rather
than: <Thread
id=”Thread01”> <TestStep> <TestAssertion> <IfTrue><ThreadRef>Thread01</ThreaRef></IfTrue> </TestAssertion> </TestStep> </Thread> [MIKE2] –
This avoids one level (Test Step) of indirection to do recursion. Comments? - Because we did not want to spawn a thread from
within a test step, and for other reasons, we came to the conclusion that it is
better to extract the "Assertion" test outside of the GetMessage
step. [MIKE] - Removing
TestAssertion from within a GetMessage "operation" makes sense, since
we may Test Assertions without having to evaluate message content. This does not put TestAssertion outside
of a <TestStep> element however.. and there may be a need for it to be
available even at the <Thread> level (see "looping" comments
below) [Jacques
Durand] Wouldn't TestAssertion be considered as just
another (fancy) workflow operator? At same level as split()? (We have to
clarify what TestStep means) [MIKE2] –
Yes, that is the intention. <TestStep> <GetMessage> <Filter>
.. some XPath expression </Filter> </GetMessage> <TestAssertion> <VerifyContent>
.. some XPath expression </VerifyContent> </TestAssertion> </TestStep> [MIKE2] – Under
our new definition, <TestStep> is no longer an “atomic” level of
execution.. since we can now <Split> Threads from within a TestStep. This blurs the distinction between
<Thread> and <TestStep>, and raises the question if we should not
simply have <Threads> alone, capable of doing all the operations of a
TestStep. [MIKE] - Making <TestAssertion>
available at the <Thread> level wouldn't make sense from the logical
standpoint that <Threads> choreograph <Threads> and
<TestStep>s... and <TestStep>s is where the message sending, receiving and assertion
evaluation occur. - The Assertion becomes a control flow operator: it
may test FilterOutput material from several GetMessage inputs (these need be
named), or may test value of variables (no message data). [MIKE] - Yes the assertion becomes the key control flow operator now..
since we no longer evaluate the boolean result of a <TestStep>.. or its
<Thread> container... It
tests FilterResult from a "SINGLE" GetMessage <Filter>
operation (the current <Filter>.. although many <Filter> operations
are possible within a <TestStep>). - The outcome of the assertion test will be stated
explicitly (e.g. might be indicated via two attributes: "when_true",
"when_false" that are just a representation for if..then...else,
without the composability.) [MIKE] - I think that a reference to a single
Thread using "whenTrue" and/or "whenFalse" may be the way
to do this - These statements effectively
decouple the boolean result of an assertion, with the test case outcome. [MIKE] - I'm not sure what you mean there, since the boolean result of a
<TestAssertion> is fundamental to the test case outcome in your proposal,
when coupled with the enumerated attribute values you describe below. [Jacques Durand]
Yes, I only meant that a boolean result "false" does not
automatically mean "fail". These new attributes can map a true/false
result, to any of these test case outcomes: fail/pass/undetermined/continue. [MIKE2] - I see. This was needed
to handle "error-catching threads" that must fail the case when
assertion is true. [MIKE] - So do we need to provide a "whenTrue" and
"whenFalse" attribute for every TestAssertion? Shouldn't we have an
"implied" Test Driver behavior if one or both of these attributes
aren't' present? I would think the default attribute values should be
whenTrue="continue"
whenFalse="fail". [Jacques Durand]
that can be the default. (though you note that a test case would never
"pass" with these defaults, unless we say that if it terminates
gracefully, it passes.) Any outcome that we need to be different from these,
needs the attribute reset explicitly. [MIKE2] – Actually
, I should have said: be
whenTrue="continue"
whenFalse="exitFail".
This would seem to eliminate any ambiguity in Test Case result, as well
as provide an implicit/unambiguous Test Driver behavior, since in our new
schema, we can either “continue”, “split” or “exit” based upon a false result. Additionally, I believe we need to address the <TestPreCondition>
operation. It is semantically different than <TestAssertion>. A precondition is something beyond the
control of the Test Driver and Test material. It is testable, but we cannot fail a Test Case if that
precondition does not exist.
Therefore, I believe we need to keep the <TestAssertion>
operation, and reduce its possible enumerated types for "onTrue" and
"onFalse" to "undetermined", "split"... since you
would never "pass" a Test Case based upon the existence of a
precondition..., nor would you "fail" a Test Case based upon a
precondition not being met. [Jacques Durand] Couldn't we just consider the
preCond (when we need one) like being just another Assertion op that preceeds
the "normal" one, and with the right outcome
(when_false="undetermined" when_true="continue"), now that
we have this degree of control? (I feel that it can be confusing to introduce a
"preCOnd" to users: what special semantics can we give to it that
jsutifies a special operation, now that we can express it with another
Assertion op?) [MIKE2] – Mike.
We’ll break backward compatibility
with V1.0 Test Suites if we eliminate <TestPreCondition>.. but perhaps we
should do this to simplify more.
Likewise, a <TestAssertion> operation is semantically different
from a TestPrecondition operation. And I believe the enumerated list of values
for "onTrue" and "onFalse" should be reduced. One would not set a final state of a
Test Case to "undetermined" based upon the result of a
<TestAssertion>. There are
not ambiguities, and no unsatisified preconditions at this point, so a Test
Case either "passes" or "fails" based upon that assertion. So I suggest the possible values for
"onTrue" and "onFalse" be "pass",
"fail" , "continue" and "split". Of course, using 2 optional attributes in our XML can create some
interesting paradoxes... i.e.
onTrue="pass" and onFalse="pass".. or
onTrue="continue" onFalse="continue" .... again this means
the TestAssertion is meaningless.
But perhaps we can live with this.. it's the semantic meaning of
<TestPreCondition> and <TestAssertion> that I am concerned
with. I definitely feel that we
should limit the enumerated values of those 2 test objects to meaningful
values. Lastly, as I understand it, if the Test Case proceeds and finishes
execution to the end of its logic flow, and the last <TestAssertion>
evaluated to "true", then the Test Case passes. This also assumes all Threads run to
completion. If any Thread has not
completed execution, then I would think that the Test Case result would be
"undetermined". [Jacques Durand] This kind of implicit semantics becomes
complex... As best practice, I believe every final outcome should
result from an explicit exit statement, even the last one. [MIKE2] –
But we can’t be “ambiguous” with testing results, so our Test Driver behavior
must be deterministic. I believe
that it will be with regard to TestAssertion results with a default behavior of
onTrue=”continue”, onFalse=”exitFail”. As far as “unfinished” Thread results, I
would think that all threads MUST complete to make a final test case pass/fail
call. At best, the kind of implicit outcome we could keep is: when
the entire test case is over (all its threads are over) meaning that no exit
has been made on either pass/fail/undetermined, then it is a pass.
[MIKE] - This is definitely useful. - Looping
("while...do") can be handled by a thread invoking itself
recursively, from an Assertion op. [MIKE] - So a
construct such as:
<Thread id="Thread01"> <TestAssertion split="Thread01"> ..... (some assertion) .... </TestAssertion> </Thread> should work.
That means we need to make <TestAssertion> available at the
<Thread> level (currently it is defined as a child of the <TestStep>
element only. I don't think that
that will be a problem. [Jacques Durand] Yes (wouldn't the
recusrive branching occur within one of these "when_true" attribute?) Jacques |
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]