[ebxml-iic-msg] [CTTF-MS]comments on ebXMLTestSuite

[Jacques Durand <JDurand@fsw.fujitsu.com>]

Title: [CTTF-MS]comments on ebXMLTestSuite

Hello:

here are some (lengthy) comments on ebXMLTestSuite doc/design.

Cheers,

jacques

  <<CTTF_MS_comments_Apr_29.txt>>

Mike, Matt: 

A first review of the material you posted some time ago - so far focused here
on ebXMLTestSuite.xml, which is really key to the automation of our tests.
Overall, I am impressed with the level of automation we seem to be able
to achieve, especially in the "analysis" part of the test.
Looks like XPath will be of great help here, as you and Matt suggested.
In addition, Schematron is to investigate also (Matt, what do you think of Schematron?)

My comments below are mostly driven by the "operational" aspects of the tests
(so that this XML material is easy to process by a testbed implementation) 
and by the expectation we can reuse most to drive Interoperability tests as well,
in addition to Conformance tests.

First comment [C1]:
As a context to my subsequent comments, I will assume that these conformance tests
will run on a testbed architecture (harness) that involves two Test driver components:
(a) an Application Test Driver (or AT-driver) (simulating application layer,
for feeding  message data and config data to
the candidate MSH, through its application API, but also for analyzing 
messages received by candidate and transferred to app layer)
(b) a Message Test Driver (or MT-driver) (simulating other party MSH, interacting
at wire-level with canidate MSH, sending messages to the
candidate MSH, and receiving messages from it. Analyzing these messages.)
So I assume that each Test Step of a given Test Case is executed by either one 
of these test components. 
NOTE: of course both test drivers could be implemented by the same piece of code, running
in same process, but this is the most general case.

[C2]
Assuming these two drivers are set-up, the TestStep element of a Test Case 
could specify which one (AT-driver or MT-driver) is concerned by the test. 
We could do this with the new attribute "stepDriver" below:
<ebTest:TestStep stepId="s1" stepName="Send a SOAP message from the candidate party" stepDriver="AT-Driver">
<ebTest:TestStep stepId="s2" stepName="Parse sent message" stepDriver="MT-Driver">
 That would take us one step further toward automation.

[C3]
The main input of step S1 is message data. You provide a set of operation elements
(ebTest:SetMessagePackage, ebTest:SetHeader,..), to enable the test driver to drive
the candidate MSH (through adequate API calls) so that it builds the right message to send out.
Two suggestions here: 
- (1)  Could we wrap all these operations that prepare the same message, 
in a same XML parent element? e.g. <ebTest:SetMessage>. 
- (2) maybe we can assume some pre-existing "message templates" (or sample messages) 
so that we do not have to set all message elements each time, but only those we want 
to override on the template. 
So the <ebTest:SetMessage> tag would also specify the template we want to derive from, 
e.g. <ebTest:SetMessage mesgtmpl="mesg01_testXYZ">.

[C4]
we may also consider an option (in addition to the way you do it now), 
where we do NOT specify the message data content directly
inside the TestStep element, but instead do it in another document 
that our TestStep element would reference, example: 
<ebTest:TestStep stepId="s1" stepName="Send a SOAP message from the candidate party" 
		 stepDriver="AT-Driver"
		 stepMessage="setmessageXYZ">
(or we could use more formal referencing, e.g. XLink?)
Such an option is motivated by: 
- several test cases may reuse same message input 
(e.g. identified by "setmessageXYZ" above), so that we do not want to repeat them. 
Same for other operations like message analysis.
- keeping test cases and test steps definitions more high-level or abstract
and make the document more readable. The Test Case doc for simple r1.1.6 is already busy,
future tests cases might get cluttered with too detailed message or analysis data. 

[C5] The TestTrace schema seems to be general enough so that it could be used
not only as output of the MT-driver (as in S2), but also of the AT-driver, for 
monitoring the messages received by the candidate MSH.
That would especially be useful for Interoperability tsets - but also for
some conformance tests (e.g. when checking if the right message is passed
from MSH candidate, to its app.) We need to explore this.

[C6] I am a little unclear whether the way we associate a trace with a test, and with
the input messages for this test, is sufficient or not.
When a message sent by the candidate MSH is received by the MT-driver
(as in r1.1.6 step S2), I'd like to be convinced we are going to use it in the
"right test": because I am assuming that the tests may not always be done in a 
purely sequential way. Possible following scenarios should be supported
(maybe they are, but not sure):

(sc1)- the same test case is repeated several times, and we may want to distinguish
the test data ( and test condition) for each. I.e. the input message crafted by
each instance of the same test will be different - e.g. using different messageID -
and the test condition, as specified by the parseTrace element, will target the
corresponding message. I guess that can be done by adding messageID condition
(in XPath or Schematron rule context) in order to distinguish the right trace,
in addition to the Test Req#.
(sc2)- Also, it may happen that for some reason, the message 
currently received by the MT-driver is not related to the "current" test, 
but to a previous one (e.g. due to resendings in case of retries, etc.)

So these scenarios require unambiguous correlation of input messages with output trace.
Also: the trace is supposed to contain the test req (e.g. r1.1.6, and possibly the step # e.g. S2)
how/when are these added to the trace? If this info comes from the TestCase doc that
is fed to the MT-driver, then that puts a strong requirement on the order of
messages received, that should match the order the MT-driver is getting test case data.
Could we have the trace built only from message data? Can we then introduce test req references
- e.g. r1.1.6 - in this mesge data? (e.g. in Service/Action field, or PartyID...)

[C7]
Here is the operational behavior of the test execution I would expect.
Let me know if that's what you had in mind:

(a)- The testbed (AT-driver + MT-driver) is being fed with a TestSuite doc, which is
a sequence of TestCases. Each TestCase describes a sequence of TestSteps.
Some TestSteps are to be executed by the AT-driver, some by the MT-driver.
Normally, everything will execute in the right order. But we do not assume
a perfectly sequential order of the tests. (e.g. some messages may arrive in
different order, and we do not assume a notification of end of Test #123 before
starting Test#124.)

(b)- any message received (e.g. by MT-driver) is appended to a Trace file, 
in the format described by the TestTrace schema, in the order of reception.
The "requirementReferenceID" attribute is set by using the test case # (e.g. 1.1.6)
that is reported in some field of the message. (Same for the test case step?)  

(c)- the "ParseTrace" command (e.g. as described in TestStep S2 of 1.1.6) 
will be passed to the MT-driver, at a time that may not match exactly the
time the test is executed (message sent).
The command has enough data to discriminate the right trace subset (the Test # e.g. 1.1.6, 
and possibly the expected MessageID).

(d)- the MT-driver maintains a queue of "ParseTrace" orders, but does not trust the next
message received to be for the next ParseTrace order - depends on when the associated 
message is received,
due to network latency, "noise" messages, overlapping tests, etc... 
So the MT-driver is periodically watching the
trace file, and tries to identify messages in it that match its next parseTrace order. 
If a parseTrace order can be executed on an instance of the trace, it executes the 
related test condition.

So the interest of this way to operate, is that the trace can be parsed/checked
at any time, in a de-coupled way with the actual execution of the tests.