[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: [ebxml-iic] scripting of test cases: latest advances
Jacques, Inlined are some
additional comments, labeled [Mike3] Regards, Mike -----Original
Message----- inline [Jacques
Durand 2] ..... [Jacques
Durand] I thought doing that too, but then we have to be clear of the
semantics of starting a thread that way: since it can be done any place within
a thread (not just the end), it actually should have the semantics of: split(ThreadRef01) (i.e. the main thread keeps going concurrently, unless the next
op after this is a join(ThreadRef01). So we may keep teh above notation, but it has to be clear its a
short for split(ThreadRef01). [MIKE2] - I agree with you. Using an attribute to express this is not clear. So in the revised schema, I used
the <Split> syntax from a <TestAssertion> to maintain that clarity.
[MIKE] - This is true. Additionally, if the Thread were invoked from
within a Test Step, then it would be passed the parameters of the Test
Step. I think that it would be
useful to have a "scope" attribute for any parameter defined via
<SetParameter>. For example,
we could have a <SetParameter scope="self"> (meaning scope is
limited to the TestStep or Thread in which it is defined, and is
undefined/unavailable to any ancestor or descendant Threads and Test Steps) or
<SetParameter scope="selfAndDescendants", meaning the parameter's
value can be passed to any descendant Thread and its Test Steps. Doing this would avoid inadvertent
side-effects by passing parameters whose only purpose is for the current Test
Step to descendent Threads, where its value may be inadvertently substituted as
message content to be sent or as a comparison parameter used to verify received
message content. [MIKE2] - Test Step scooping appears to be important,
if we agree that a Test Step is a series of message sends/receives for a common
type of transaction (i.e. Purchase Order). Setting a $CPAId or $ConversationId value may only be
appropriate for that Purchase Order transaction, but may be inappropriate for
subsequent Test Steps in a Thread.
If we treat all parameter definitions as "global" to the
Thread in which they are defined, then collisions between sibling TestSteps can
occur. [MIKE2] - Thinking about this more, I am wondering if
we should just "do away"
with <TestStep> entirely, and treat everything as a <Thread> (i.e.
merge the operations of a <TestStep> in to <Thread>). While this removes the meaning of a
<TesStep> as a container for
common message request/receive operations and test assertion
verification, it would simplify the testing model even more. Plus it would permit a simpler syntax for
recursion: [MIKE3] – I think that a <Thread> can still
have context (like <TestStep> for a contained operation ( such as a PutMessage or
GetMessage ). And since,
<Thread> will perform the same operations, an “atomic” level
<Thread> could simply be one that (optionally ) does one single operation
instead of multiple <PutMessage> and <GetMessage> operations.
However, it is an advantage to be able to cluster <PutMessage> and
<GetMessage> operations together in a single <Thread>, since that
is realistically more in line with what a business transaction is ( i.e. as
described in BPSS ). Plus, we keep
the context ( CPAId, ConversationId ) the same for all of those operations..
instead of having to “pass” that context to the next sibling <TestStep> (
which creates parameter scoping problems anyway). So I am in favor of unifying the functionality of a
<Thread> and <TestStep> in to the <Thread> class, and
eliminating <TestStep> entirely. <Thread id="Thread01"> <TestAssertion> <IfTrue><ThreadRef>Thread01</ThreaRef> </TestAssertion> </Thread> rather than: <Thread id="Thread01"> <TestStep> <TestAssertion> <IfTrue><ThreadRef>Thread01</ThreaRef></IfTrue> </TestAssertion> </TestStep> </Thread> [MIKE2] - This avoids one level (Test Step) of
indirection to do recursion.
Comments? [Jacques Durand 2] If our Assertion becomes a 1st class
control flow op like split(), then it shold anyway be outside a Test
Step. That is, I thought the statement: <TestAssertion
whenTrue="ThreadRef01">....</TestAssertion> [MIKE3] – I agree, and it is now a 1st class
control flow operator. - Because we did not want to spawn a thread from
within a test step, and for other reasons, we came to the conclusion that it is
better to extract the "Assertion" test outside of the GetMessage
step. [MIKE] - Removing
TestAssertion from within a GetMessage "operation" makes sense, since
we may Test Assertions without having to evaluate message content. This does not put TestAssertion outside
of a <TestStep> element however.. and there may be a need for it to be
available even at the <Thread> level (see "looping" comments
below) [Jacques
Durand] Wouldn't TestAssertion be considered as just
another (fancy) workflow operator? At same level as split()? (We have to
clarify what TestStep means) [MIKE2] - Yes, that is the intention. <TestStep> <GetMessage> <Filter>
.. some XPath expression </Filter> </GetMessage> <TestAssertion> <VerifyContent>
.. some XPath expression </VerifyContent> </TestAssertion> </TestStep> [MIKE2] - Under our new definition, <TestStep>
is no longer an "atomic" level of execution.. since we can now
<Split> Threads from within a TestStep. This blurs the distinction between <Thread> and
<TestStep>, and raises the question if we should not simply have
<Threads> alone, capable of doing all the operations of a TestStep. [MIKE3] – This is true. But branching
only at the higher <Thread> level means you lose the “context” ( i.e. the
current <FilterResult> .. the current ConversationId, CPAId.. and other
context of the <TestStep>).
That was the original purpose of <TestStep> .. to provide the low
level context to which you refer previously in this discussion. So really, one must either keep
<TestStep> for the purpose of maintaining context and branching based upon a
<TestAssertion> within a <TestStep>.. or (what I favor)..
eliminating <TestStep>.. and giving a <Thread> all of a
<TestStep>’s properties.
That way, context is always maintained when you <Split> a
<Thread>. Comments? [MIKE] - So do we need to
provide a "whenTrue" and "whenFalse" attribute for every
TestAssertion? Shouldn't we have
an "implied" Test Driver behavior if one or both of these attributes
aren't' present? I would think the default attribute values should be
whenTrue="continue"
whenFalse="fail". [Jacques Durand]
that can be the default. (though you note that a test case would never
"pass" with these defaults, unless we say that if it terminates
gracefully, it passes.) Any outcome that we need to be different from these,
needs the attribute reset explicitly. [MIKE2] - Actually , I should have said: be
whenTrue="continue"
whenFalse="exitFail".
This would seem to eliminate any ambiguity in Test Case result, as well
as provide an implicit/unambiguous Test Driver behavior, since in our new
schema, we can either "continue", "split" or
"exit" based upon a false result. [Jacques Durand 2] Right. Additionally, I believe we need to address the <TestPreCondition>
operation. It is semantically different than <TestAssertion>. A precondition is something beyond the
control of the Test Driver and Test material. It is testable, but we cannot fail a Test Case if that
precondition does not exist.
Therefore, I believe we need to keep the <TestAssertion>
operation, and reduce its possible enumerated types for "onTrue" and
"onFalse" to "undetermined", "split"... since you
would never "pass" a Test Case based upon the existence of a
precondition..., nor would you "fail" a Test Case based upon a
precondition not being met. [Jacques Durand] Couldn't we just consider the
preCond (when we need one) like being just another Assertion op that preceeds
the "normal" one, and with the right outcome
(when_false="undetermined" when_true="continue"), now that
we have this degree of control? (I feel that it can be confusing to introduce a
"preCOnd" to users: what special semantics can we give to it that
jsutifies a special operation, now that we can express it with another
Assertion op?) [MIKE2] - Mike.
We'll break backward compatibility with V1.0 Test Suites if we eliminate
<TestPreCondition>.. but perhaps we should do this to simplify more. [Jacques Durand 2] I think backward com is probably broken
anyway? Plus if it were a real concern, test case scripts could be converted
using some XSLT from 1.0 to 1.1. [MIIKE3]
– Agreed.
Likewise, a <TestAssertion> operation is semantically different
from a TestPrecondition operation. And I believe the enumerated list of values
for "onTrue" and "onFalse" should be reduced. One would not set a final state of a
Test Case to "undetermined" based upon the result of a
<TestAssertion>. There are
not ambiguities, and no unsatisified preconditions at this point, so a Test
Case either "passes" or "fails" based upon that
assertion. So I suggest the possible
values for "onTrue" and "onFalse" be "pass",
"fail" , "continue" and "split". Of course, using 2 optional attributes in our XML can create some
interesting paradoxes... i.e.
onTrue="pass" and onFalse="pass".. or
onTrue="continue" onFalse="continue" ..... again this means
the TestAssertion is meaningless.
But perhaps we can live with this.. it's the semantic meaning of
<TestPreCondition> and <TestAssertion> that I am concerned with. I definitely feel that we should limit
the enumerated values of those 2 test objects to meaningful values. Lastly, as I understand it, if the Test Case proceeds and finishes
execution to the end of its logic flow, and the last <TestAssertion>
evaluated to "true", then the Test Case passes. This also assumes all Threads run to
completion. If any Thread has not
completed execution, then I would think that the Test Case result would be
"undetermined". [Jacques Durand] This kind of implicit semantics becomes
complex... As best practice, I believe every final outcome should result
from an explicit exit statement, even the last one. [MIKE2] - But we can't be "ambiguous" with
testing results, so our Test Driver behavior must be deterministic. I believe that it will be with regard
to TestAssertion results with a default behavior of onTrue="continue",
onFalse="exitFail". As
far as "unfinished" Thread results, I would think that all threads
MUST complete to make a final test case pass/fail call. |
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]