Jacques and all,
Here is my response to the ongoing
thread regarding test case scripting.
It looks like we have nearly reached
convergence. I am attaching a revised schema
diagram reflecting changes, along with a ZIP
containing the 3 use case instances,
along with schemas an XSL
stylesheet.
With the revised schema, ( including
both <If><Then><Else> along with <Split> and
<Join> elements ),
I am also including revised use case instance
files reflecting those changes.
Some suggestions based upon these use
cases:
1) If we define a "pass" <Thread>
and a "fail" <Thread>.. we may want to make the "global" ( i.e.
defined
at the <TestSuite> level ) since they can
then be used everywhere. If we assume that the scope of a
<Thread> definition is only local
to a <TestCase>... it just seems like a logical thing to do I
think.
[Jacques
Durand] still skeptical about the value of sharing "global "threads"...
won't each thread vary based on the steps and test assertions content ? the
conditions that lead to failure or success are different for each
case.
[Jacques] - Not necessarily.
You may also have a <Thread> that just sends
a <PurchaseOrder> request. That is all that it does every
time.
Or perhaps a <Thread> that always does a
<PutMessage> then a <GetMessage> and a <PutMessage>..
always sending and expecting messages ( of course Timestamps and
ConversatoinIds will change.. but nothing else does ). Since
BPSS testing could involve a series of <Threads> ( or business
transactions ) , some of them may be repeatable for other
<TestCase>s.
2) I agree that there is no need to have a
<Thread> evaluate to "true"... if we can instead just verify
that
the last <TestAssertion> in the
<Thread> evaluates to "true".. however.. there may be
<TestStep>s in a <Thread> where
there is only a
<TestPreCondition> or where there is only a
<PutMessage> operation. If a <PutMessage>
happens to be final "leaf node" in our
<Thread> branch.. then there is no <TestAssertion> ( a
<TestStep> with
a <PutMessage> simply sends a message, and returns a "true" if it is successful.
) So I would suggest that
we simply state that as long as the final leaf
node in a <Thread> branch, or a <TestStep> branch returns a
value of "true", then the logic flow may
proceed after that branch. I am avoiding saying that the
<Thread> or <TestStep>
therefore evaluates to "true".. but in essence, that is what I am
saying.
[Jacques
Durand] see in my attached comments: we may need to consider an
"undetermined" result for test cases. Even if a precond fails, that
should not mean the test case fails: precisely the semantics of a precond is
that if it fails, the test case is not meaningful ! In other cases, if no
Assertion - e.g. just PutMessage, default could be "undetermined" too. The
"pass" or "fail" semantics should ideally result from an explicit statement,
and I think we moved toward this direction with the "exitCondition"
attribute in Assertions that explicitly says what should be the
outcome (i.e. not implicitly based on the boolean outcome). Should we
require this attribute at every exit point, even at the last step, or on a
PutMessage? (default would be "undetermined" or
"notApplicable")
[MIKE] I agree that the "pass" and "fail"
result should be reserved for <TestAssertion> (meaning a
"leaf node" in the logic tree, from which there is no more possible
branching). However, the implication that the <TestCase> "fails"
if a <TestPreCondition> is "false" was not my intention. I think
that the specification needs more clarity. The Test Driver should
interpret a "fail" boolean result of a <TestPreCondition> differently,
and set the final <TestCase> result to "undetermined". I would
say the same thing for <PutMessage>. Its boolean result could be
"fail", but ultimately, the Test Driver would recognize that the failure
occured on a <PutMessage>, and the result would be
"undetermined". I do not see the need to "set" this, since
the Test Driver should make the determination that the failure did not occur
on a <TestAssertion>
[MIKE] - In fact, I do not see the need for
an "exitCondition" for <TestPreCondition> or <PutMessage>.. or
for that matter <TestAssertion>. Since the logic flow scripted
in the XML scripting ultimately determines flow, there is really no need to
use an "exitCondition" attribute anywhere ( i.e. this is just a GOTO
that circumvents the logic defined in the scripting
).
I agree that the Test Framework
specification needs clarity here.
Comments?
[MIKE] - I've added a few more comments re: the
MessageStore and "synchType" attribute of
<Thread>s
3) After revising the 3 use case instances, I
agree with your earlier suggestion that a <Thread> be "synchronous" by
default,
and that a <Split> be used to define it
as "asynchronous". It makes for much cleaner structure.. plus it
enforces the
"synchronization" of <Thread>s through
the <Join> operator. I
think that ( although a bit more verbose ).. it removes
ambiguities and ultimately makes it more understandable.
Regards,
Mike