[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: [ebxml-iic] minutes and more - response and use case examples
|
Jacques, Attached are my
comments to your comments. I am
hopeful that we can reach an agreement on the fundamental issues Today regarding: 1) Branching at the
<TestStep> level: (I’ll
agree, even though it may cause serious parameter scoping issues to allow <TestSteps> to have child <Threads> (and child TestSteps
also?) That means that if I
assign a new ConversationId for a Test Step, all of its child <Thread>s get that ConversationId also… which may
be an unwanted side-effect.. since <TestSteps> are “atomic” level tests (much like BPSS <Transactions> are atomic-level activities
). I may want to change a
ConversationId for a single send/receive.. but I would not want to propogate that ConversationId to any child
<Threads> .. which is unavoidable if I allow logical branching at such a
low level in my testing script. 2) Branching at the
<Thread> level : I don’t see how you can avoid this if you want to do
high level (i.e. BinaryCollaborationActivity) choreography.
Restricting all branching at the <TestStep> level, based upon
<TestAssertion> results, means you are driving all choreography based upon message content at the Transaction level, and
are not interested in perhaps doing choreography based upon success/failure of a TransactionActivity, or perhaps even
success/failure of a BinaryCollaborationActivity. This appears to be trying to “micro-manage” business process choreography based upon individual message
tests. A
solution would be to allow <TestAssertion> Boolean results to “bubble up”
to their parent <TestStep> and possibly ITS parent <Thread> result, so that logical Test Case flow can occur at a
higher level. I agree that it is “simpler”
to understand choreography if everything branches from a <TestAssertion> result, but
<TestAssertions> may be part of testing a bigger Transaction.. which
could in turn be part of testing a bigger Binary Collaboration. Restricting all branching to the <TestStep> level
destroys that testing (and business) hierarchy. 3) Addition of a <VerifyTime>
operator within a <TestAssertion>.
This looks like “must” in order to support time comparisons, especially since XPath does not time comparison expressions. 4) Dealing with
TimeToPerform, TimeToAcknowledgeAcceptance and TimeToAcknowledgeReceipt
messages (BPSS requirements). Regards, Mike -----Original
Message----- Mike: that's definitely
better, but there are some places where the test
scripts don't seem to work (or do what the use case expects) So I point to these
problems and outline a solution. Also we still do not
handle the "test case outcome" semantics in a satisfying way , I
think, and that becomes obvious on one of the use cases. See my proposal (I
try to dissociate explicitly the boolean value of an Assertion, and the
thread operational outcome for the test case: fail/pass/undetermined/continue) See attached the original
Use Case defs, and for each, my comments labelled: "...comments from jacques on latest
solution from Mike (May 20):" Cheers, Jacques -----Original Message----- Jacques and all, I studied BPSS v1.05 specification and modified the
Test Framework schema to reflect possible modifications for performing BPSS testing. Attached is a ZIP of BPSS use cases #1-3 POC, using
modified Test Framework scripting to support BPSS testing.
Also attached are my comments/response to Jacques notes re: Test
Framework modifications/suggestions. Regards, Mike |
Test use cases, representative of test case control patterns. They should help focus on the minimal set of operators for control flow. -------------------------------------- Use Case #1: Basic Business Transaction, (e.g. PIP 3A4) with TimeToPerform, and TimeToAcknowledge -------------------------------------- Test Case: (test driver plays "buyer") Step 1: Buyer Send a P.O. to Supplier. Step 2: Receive a signal ack, correlated with the P.O. based on Conversation ID Step 3: Receive a P.O. Confirmation or Rejection. Step 4: Send a signal Ack. Constraints: - total time for 1-2-3 is bounded by TimeToPerform. - total time for 1-2 is bounded by TimeToAcknowledge. Success: - within TimeToPerform, receive either a P.O. Confirmation or Rejection, but not both. - TimeToAcknowledge is satisfied. Variant: The correlation is based on a PO reference # that is in the payload. --------- Use Case #1: comments from jacques on latest solution from Mike (May 20): - I am not convinced that this new "Transaction" element is necessary (one more construct, also), especially for a general purpose test framework (seems too focused on BPSS bus transactions)... [MIKE2] - Actually, we could eliminate <Transaction> if we assume that a BPSS Transaction maps to a <TestStep>.. that would be simplify without having to introduce a new tag. However, if we wish to time "any" subset of Test Steps, then we'll need to introduce some new construct. As we discussed, we may want to measure time for any subset of test step sequence, and possibly overlapping ones (which prohibits a "bloc" approach) [MIKE2]- If we don't wish to "block" ... then we'll need some mechanism (i.e. a <Tag>) to "group" any subset of steps that we choose. If we don't want to call it a <Transaction>, then we can call it something else. I thought we opted for some timestamping mechanism. For example: we could split a thread that starts a counter (e.g. just use "sleep" op) and at the end of the sleep, checks (Assertion) if some global var (e.g. "Transaction1Done") is true. [MIKE2] - We could do that if we change 2 things in the current Test Framework. 1) Relax the parameter scoping rules so that any parameter defined in a <TestStep> is "global" to its parent <Thread>. Currently, that is not the case. *** Of course, the danger of relaxing the scoping rule is that you can get "side effects" from sibling <TestStep>s that may use a parameter value previous defined in another Test Step rather than the value that is automatically generated by the Test Driver. (such as $ConversationId .. or $MessageId...) However, perhaps this is an by-product we should accept *** 2) If a "timeToPerform" is to be a critical testable assertion, then I suggest it should be done within the context of a <TestAssertion> as: <TestAssertion> <VerifyTimeToPerform limit="T600S"> <Transaction> <StepId="a"/> <StepId="b"/> <StepId="w"/> </Transaction> </VerifyTimeToPerform> </TestAssertion> I don't see the need for a <Sleep> Thread if we write the Test Case as: <TestCase> <TestStep> id="a">... send purchase order ...</TestStep> <TestStep> id="b">... receive confirmation ...</TestStep> ..... more TestSteps .... <TestStep> id="g">... some other action ...</TestStep> <If> <TestStep id="z"> <TestAssertion> <VerifyTimeToPerform limit="T600S"> <Transaction> <StepId="a"/> <StepId="b"/> <StepId="f"/> </Transaction> </VerifyTimeToPerform> </TestAssertion> </TestStep> <Then> <SetParameter>name="Transaction1Done" value="true"/> </Then> .... now if we wish to continue based on the result of the TimeToPerform operation ... <If> <VerifyParameter name="Transaction1Done" value="true"/> <Then> .... do something .. </Then> <Else> .... do something else... </Else> </If> </TestCase> Note that this would mean adding a new instruction called <VerifyParameter> (currently we don’t have one) that would be used for control flow And the thread that does the time-bound transaction (e.g. BPSS bus transaction) would at the end set Transaction1Done to "true". (so we can use SetXPathParameter) [MIKE2] - SetXPathParameter sets a parameter value (a string ) based upon the content of a returned XPath query result argument. So, rather than using <SetXPathParameter>, it would be necessary to use <SetParameter> (which assigns a string value to a parameter name as shown above). [MIKE2] - ***One thing that concerns me is that in order to be truly accurate with "TimeToPerform", we should be using message timestamps ONLY To determine such things as TimeToPerform, TimeToAcknowledgeReceipt and TimeToAcknowledgeAcceptance business signal messages. Using "clock time" of a Test Driver is really not accurate, since network latency could influence message timing. I'm thinking that what we really need is a generic <VerifyTime> operation within a <TestAssertion> that accepts 2 or 3 arguments, and can calculate if timestamp of message (A) < or > or = message (B)... or if the timestamp from message (A) +/- timestampe of message (B) < or > or = timestampe of message (C).. otherwise, a TimeToPerform test is a very ambiguous result*** - About 'TimeToAcknowledgeReceipt': not sure how that works?(where is used/set the TimeToAcknowledge value for the test case). [MIKE2] - TimeToAcknowledgeReceipt is the time necessary for a business application to send a “business acknowledgment” back to the sending party. This operation will only work if the Test Driver assumes it must extract the Timestamp from the current “business acknowledgment” message, and determine the time difference from the timestamp of the last sent message. This is a feature not currently in the Test Framework, but perhaps should be to support this..not just for BPSS, but for any Business Process, since they share this common functionality. That also could be done using a global signal var and a "sleeper thread". [MIKE2] - I would argue that a “sleeper thread” is not necessary, because a sleeper thread would not be necessary for us to measure time difference between two parameters.. which is all that is really necessary for this Test Case. - Second GetMessage op: <ebTest:VerifyContent>//*[Confirmation and Rejection] Shounldn't we use : <ebTest:VerifyContent>/FilterResult//*[Confirmation and Rejection]? [MIKE2] - Yes, I should have included /FilterResult. You are correct. I used a "shorthand" notation. Also, we want to fail teh case if this condition is satisfied ! So we need to actually wait for TimeToPerform to make sure we have not gotten "too many" messages. [MIKE2] - I would think that this particular <TestAssertion> would not have to wait to test TimeToPerform.. since the presence of both Confirmation and Rejection fail the Test Case.. regardless of TimeToPerform, correct? I suggest using the "sleeper" thread suggested above, and add one more test of failure: if count(/FilterResult//*[Confirmation or Rejection]) > 1. [MIKE2] - But we are not testing for duplicates are we? We're testing to see if both a Confirmation AND Rejection are present.. which would be a failure of the Test Case. So how would a "sleeper" thread, and counting the number of Confirmations or Rejections verify that both do not exist? -------------------------------------- Use Case #2: Catching unexpected ebXML Error messages (a test case "design pattern") -------------------------------------- Test Case: (test driver plays "buyer") Step 1: buyer sends a message M1 Step 2: receive and verify message M2 Correlation M1-M2 based on COnversation ID. But in case an error is received that correlates with M1 (in addition or instead of M2), at any point in time within 300sec after sending M1, before or after receiving M2, and regardless of the outcome of verifying M2, we want the test case to fail. --------- Use Case #2: comments from jacques on latest solution from Mike (May 20): - Do we really need the "wrapper" element ThreadRef nameRef='thread_01' ? that seems redundant. [MIKE2] - I just rendered that in HTML to show that we are using "references".. not inlining <Thread>s in the scripting. I'll remove it - I believe the Assertion test for the error-catching thread: /FilterResult//*[not(eb:ErrorList)] will not work !! because this thread will exit on success if we catch the response message M2 BEFORE an error message arrives! So we need to actually explicitly test for catching the error message (/FilterResult//eb:ErrorList ), and explicitly fail the test case if we catch it. [MIKE2] - I see what you mean.. Changing to the opposite would keep the Filter from automatically declaring "success".. when perhaps an Error message would follow later. We discussed this kind of use case before, and frankly I don't see how we can do without a more explicit statement to tell the semantics of the outcome of teh Assertion check, with regard to the overall test case, something of the "exitCondition" type we already have in Appendix E (but don't seem to use anywhere) (fail / pass / undetermined / continue). [MIKE2] - I agree to the use of an “exitCondition”.. however, I don’t see how you will get away from having to travers back up the logic tree to determine if branching is necessary. Otherwise, you will have a “micro-managed” scripting language, not capable of doing complex, higher-level choreographies.. because you are “making the call” on whether to contine Test Case execution with each <TestAssertion>.. which I do not believe is very flexible. Comments? Any "implicit" semantics, like (Assertion fails-->test Case fails), (Asssertion pass --> test case pass "except if there is something after"...) will be confusing, thread-context-dependent, and unclear to users. [MIKE2] - But I believe that this is unavoidable if you want to do high-level ( i.e. Thread level_ choreographies that emulate binary collaborations.. or possible binary collaborations within other binary collaborations). Trying to micro-manage Test Driver state at the <TestAssertion> level is very inflexible for complex choreographies involving multiple binary collaborations. So we could put all our logic control at the <TestStep> level, but we’ll have a very inflexible Scripting language unless we allow a “failed” <TestAssertion> to fail its parent <TestStep> dnd possibly its “grandparent” <Thread>. Particularly if we want to branch at a higher level in the test object ( or in BPSS terms “Activity”) hierarchy. Comments? - A minor point: in step #2 the test on "action=mute" would rather be done in the VerifyContent part, as we want to test all messages that correlate, including those with bad Action field (for errors) (so we don't want to filter these out). [MIKE2] - I see. OK, I'll change that. -------------------------------------- Use Case #3: conditional branching -------------------------------------- Test Case: (test driver plays "buyer") - buyer sends M1 - received M2 (e.g. an approval, or rejection) ---> if M2 is "approval", we will expect a sequence of: - receive M3 (e.g. a quotation) - send M4 (e.g. approval of quote and payment info) - receive M5 (final delivery details). ---> if M2 is "rejection", we will expect a sequence of: - receive M6 (proposed alternative) We need to verify all received messages, as the test case would fail if they do not comply. Any error message received, would also fail the test case (see use case #2) --------- Use Case #3: comments from jacques on latest solution from Mike (May 20): - the error cases could be handled all separately from main thread, like suggested in use case #1, [MIKE2] - But “splitting” a Thread that checks from any received ErrorLlist within 300 seconds would seem to be a “separate” process wouldn’t it? so we don't need to complicate the main thread with them (step #2 would not have to worry about receiving both 1 acceptance and a rejection, which anyway needs to be tested over a longer period of time, e.g. till TimeToPerform expires.) [MIKE2] - I agree. I think that it should be a “split” Thread that behaves much like the split thread that listens for ErrorList messages over a period of time. - another unclear pattern: we have these two sections: if (GetMessage = approval) then... endif; if (GetMessage = rejection) then... endif; The risk here is taht both conditions may succeed in case we get an Approval (first) then a Rejection. It would be much safer if we GetMessage just once, and whatever this response is (first message we catch), we then branch to the right thread. [MIKE2] - In any event, if our “ error listener” Thread does its job, then however we script this, the result should be OK. But if we do it that way, it seems that we may have to put the "if...then" inside the step, e.g. wrapping the Assertion check. One possible solution: If we use the proposed "when_true" and "when_false" attributes proposed in use case #2, then we could do without an "if" ! e.g. VerifyContent when_true="threadABC" when_false="threadXYZ" Or, if we want to still have two tests for "Approval" and "Rejection": VerifyContent when_true="threadApprove" when_false="continue" ... schemaLocation=http://http://www.eBusines.org/approval.xsd VerifyContent when_true="threadReject" when_false="fail" ... schemaLocation=http://http://www.eBusines.org/rejection.xsd - other than that, that seems to work (though steps don't match exactly those in use case)
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]