[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: FW: starting a thread on upgrading test case scripting
|
This message did not appear to have made
it to the list last week: From: All: Starting this thread on the upgrade of TestFramework 1.1
script language, under interest from NIST and KorBIT to move on to a new
version, and also under the increased interest in B2B testing from various
orgs. Objectives are: (a)- get to a more minimal
script language yet powerful enough. Fewer operations and concepts, in order to
foster adoption by making implementation easier. (b)- pave the way for easier
extensions (e.g. semantic tests for message content) (c)- get feedback from KorBIT
factored-in. (d)- better event handling,
and enhanced "message store" capability. (e)- make sure we support
all functions needed to support ebBP testing. The ambition is by end of 2006 to publish a test script language
upgrade that will be the reference in most B2B testing activities and test
suites. Here is attached an outline to start the discussion and
requirements gathering. I have appended at the end early comments from KorBIT /
PosTech. We will observe also relate to other initiatives in test
control (e.g. TTCN-3) and/or scripting. Please feel free to send out requirements, use cases, or
comments. Jacques |
Outline of proposed upgrade of the executable test case scripting language
(TestFramework 1.1):\
(Jacques Durand)
======================================================
Objectives:
(a)- get to a more minimal script language yet powerful enough. Fewer operations
and concepts, in order to foster adoption by making implementation easier.
(b)- pave the way for easier extensions (e.g. semantic tests for message content)
(c)- get feedback from KorBIT factored-in.
(d)- better event handling, and enhanced "message store" capability.
(e)- make sure we support all functions needed to support ebBP testing.
What is outlined below does not address all the above objectives - only outline
design for (a) and (d).
Overall (re)design: (note: many of the below is already supported)
------------------
1- Test case:
o test case = a set of testflows, one of them is the "main" testflow.
o testflow = a sequence of test steps, that is triggered by an event (on-event).
Only the main testflow is triggered by "executing the test case".
o A testflow may branch to a non-consecutive test step (label) using a simple
and unique "check-and-branch"operator (a conditional "go to". No more if-then-else).
Branching may be inconditional (always occurs) or based on simple condition,
(involving a test case variable or based on the outcome of previous test step).
2- Test step:
o a test step is an atomic unit of execution. Either (a) a test action, (b) a test control
o Each test step is labelled (uniquely identified within test case).
o Every test step execution leaves a trace in the test event loeg (see below TEL),
where the current state of the test case execution is kept for recovery and audit.
3- Message templates:
o XML documents that are parameterized, for craeting message instance representations.
o once interpreted by an "adapter", generates a real transport-dependent message (e.g. MIME/HTTP)
4- Test control (a kind of test step):
o check-and-branch (either "blocking" or not: branching synchronously (no new thread)
or async (spawn a new thread). the conditions ("check" part ) uses XPath, variables,
and constants.
o exit --> fail / pass / undetermined / system exception.
o sleep(~ "sleep" in Unix) (note: to schedule an event, use this on a
specific testflow as step#1, then put-event=step#2)
o set timer (initialize a timer to be checked later)
5- Test-Event Log (TEL) (formerly message store)
o test-event = XML object, with XML event "wrapper" (event format is TBD, e.g.
standard attributes: type, origin, timestamp, etc.)
o An adapter will apply a transform to a protocol-specific message to generate a test-event
o for example: test-event = received message (e.g. SOAP message w/o attachments),
after appropriate transformation from an adapter. It could also result from direct
input from a human (via GUI),
o A reverse adapter will generate a real protocol message from a test-event of the right type.
o the test-event log (TEL) mediates every message transfer. For example, when a message
is received by the test driver during a test case for B2B exchanges, it is first logged
as a test-event. When a message is generated by the test case, it is first generated as a
a test-event and logged, only then an adapter picks it up and send it out.
6- Test actions:
o "Get-event" action: e.g. for getting a received message (from TEL) based on a
selector (filter):
It defines a selector + timer. This action may be either "lasting" or instantaneous,
and either immediate or scheduled.
- If immediate, it starts executing immediately.
- If scheduled, it will execute at a scheduled date.
- If instantaneous, once it executes it performs a single check on the TEL and returns
whatever event(s) satisfy the selector.
- If lasting, once it executes it keeps checking the log until a selection is successful,
or until a timeout expires, whichever comes first.
o The Get-event action has a read-only attribute (true/false)(true: it does not remove
any event from the Log. False: it remove what has been selected.)
o "Put-event" action: e.g. for sending a message (to TEL, then outside via an adapter)
7- adapters (to external systems):
o a reader of the event log (selector + a special action)(e.g. format and send a message)
o a writer of the event log
---------------- Use cases to satisfy:
--------------------------------------
Use Case #1: Basic Business Transaction,
(e.g. PIP 3A4) with TimeToPerform, and TimeToAcknowledge
--------------------------------------
Test Case: (test driver plays "buyer")
Step 1: Buyer Send a P.O. to Supplier.
Step 2: Receive a signal ack, correlated with the P.O. based on Conversation ID
Step 3: Receive a P.O. Confirmation or Rejection.
Step 4: Send a signal Ack.
Constraints:
- total time for 1-2-3 is bounded by TimeToPerform.
- total time for 1-2 is bounded by TimeToAcknowledge.
Success:
- within TimeToPerform, receive either a P.O. Confirmation or Rejection, but not both.
- TimeToAcknowledge is satisfied.
Variant:
The correlation is based on a PO reference # that is in the payload.
--------------------------------------
Use Case #2: Catching unexpected ebXML Error messages (a test case "design pattern")
--------------------------------------
Test Case: (test driver plays "buyer")
Step 1: buyer sends a message M1
Step 2: receive and verify message M2
Correlation M1-M2 based on COnversation ID.
But in case an error is received that correlates with M1 (in addition or instead of M2),
at any point in time within 300sec after sending M1, before or after receiving M2,
and regardless of the outcome of verifying M2, we want the test case to fail.
--------------------------------------
Use Case #3: conditional branching
--------------------------------------
Test Case: (test driver plays "buyer")
- buyer sends M1
- received M2 (e.g. an approval, or rejection)
---> if M2 is "approval", we will expect a sequence of:
- receive M3 (e.g. a quotation)
- send M4 (e.g. approval of quote and payment info)
- receive M5 (final delivery details).
---> if M2 is "rejection", we will expect a sequence of:
- receive M6 (proposed alternative)
We need to verify all received messages, as the test case would fail if they do not comply.
Any error message received, would also fail the test case (see use case #2)
--------------- early comments:
---------------------- commented by Jaewook Kim ----------------------
- I think we need a test control operation to check, log, and then continue
tests (without exiting tests).
In some tests, we may need to simultaneously verify multiple elements in one message.
I proposes 'check-and-log' test control as follow:
o check-and-log --> result(fail / pass / undetermined / system exception) and description.
- I think we should define test recovery mechanism using the TEL.
Because some tests may require a long test time (for days),
when testbed is abruptly shutdown it should be able to recover the tests
that have been under execution.
Although That is only implementation issue, the TEL should support functions
and formats to keep test execution status or event.
- In my opinion, the "Put-event" action should define a reverse adaptor
to decide how to send a message.
When "Put-event" action is manually performed in output mapping test,
it should be executed by using user interface to show user a message.
Or, "Put-event" action may be executed by using other transport protocols.
I think "ConfigurationGroup" may define this parameter.
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]