question about testing

[Jacques Durand <JDurand@us.fujitsu.com>]

Pim contacted me about a potential project about testing ebMS3 that he might be involved in:

>I'm in a project that uses ebMS3 and needs to start testing.  I am interested in reusing as much as possible e.g. the WS-I BP 1.2/2.0 test tools, Tamelizer,  any deliverables from the CEN testbed project.   I am wondering what approach you would recommend. 

My first advice is to create “test assertions”, the blueprints or design for actual test cases that comprise an executable test suite. The test assertions rephrase in a more concrete, testable way the specification requirements. See an example below (a test assertion for a Cloud API). The TA defines typical sequences of message (“target”) , and the conditions these must fulfill.

The Test Assertions Guideline (TAG) TC has produced some guidelines to do that (has been used by other TCs). See https://www.oasis-open.org/committees/download.php/44696/taguidelines-v1.0-wd02.pdf  I can help/assist the writing of these.

Then you could define scenarios driven from one endpoint, to “exercise” these test assertions. One scenario may exercise one or more test assertions. Several scenarios may exercise same test assertion(s) .

Then you have the choice of test implementation:

(a)    “integrated” test suite is made of test cases that combine both the scenario execution AND conformance verification (the test assertion logic).

(b)   “decoupled” test suite where the scenarios are executed (manually or test driver) separately from the actual verification that implements test assertion logic.

I favor (b) which also allows for routinely verifying real business exchanges even if there is not particular scenario. And we have a tool to automate this verification (used in WS-I, perfected in the OSS Tamelizer). However (b) assumes you can capture the message exchange (produce a log of these). If you use tamelizer, this log has to be in XML (and the test assertions be scripted in XML + XPATH too as described in http://code.google.com/p/tamelizer/ ).

Then we can consider contributing your test suite to the future testbed prototype(s) of GITB project (if phase 3 approved & funded by the EC early 2013…)

 

TA_Id: CIMI-TA-Example1a (for Cloud Infrastructure Management Interface specification)

NormativeSource: “If the request to create a Machine resource is successful, then the Machine URI must be recorded in the corresponding machines collection of the CEP.” (§4.2.1.1, §5.5.12.1)

Target: A time-ordered sequence of messages m1, m2, m3 such as:

·         m1 = create Machine request

·         m2 = successful response to create message (HTTP 201 “created”),

·         m3 = a response to a “GET CEP/machines”. (with no “delete Machine” or “delete System” message logged between m1 and m3).

Predicate: the Machine URI returned in the response message m2, also appears in the list of Machines in response m3.

PrescriptionLevel: mandatory

Tag: SUT=Provider

 

-jacques