[ebxml-iic-conform] review test case material for MS conformance

[Jacques Durand <JDurand@fsw.fujitsu.com>]

Title: review test case material for MS conformance

Mike, Matt:

here is my take on remaining issues (see attached), after review of
last version Mike sent out 9/8.

I propose to close on several issues with solutions that do not pretend to be final,
but will keep us going - and will allow for refinment later on (i.e. no throw away...).

The general idea is: Once the basics of the test case material are defined
(e.g. vocabulary/naming, msg material identification, CPA parameters, service/actions invoked,
error messages and types...) then it is enough to get going on remiaining test cases.
And we are already there (almost...). Let us not be stopped by:
-  issues raised by shortcomings of our material when describing more complex test cases,
- or by the ultimate formal representation of some artifacts (CPA, MIME construction)

Once we have an inital version of all cases, we'll be in a better position
to tackle difficult cases, and we'll be also in better position to refine our test material in a second pass.

Let us try to finalize these few remaining issues this week at latest,
so that the test case definitions can proceed.

Cheers,

jacques
<<ebXMLIICTestCaseCmts_closure.txt>>

Proposed closure on 7 remaining issues on Test Case material:
----------------------------------------------------------


1. CPA: 

Proposal:
-------- 
let us just use a list of name/value pairs for defining
each CPA, in our first version of MS Test Cases.
A small set of CPAs instances is sufficient, each with a defined CPAId. 
We refer to these, from each test case (message expressions), by the CPAId.
When we need to name a CPA attribute in "message expression", 
we use one of these CPA attribute "names" with $ notation, and "CPA_" as prefix.
E.g.: "Signature" will be referred to as : $CPA_Signature.

Comments:
---------
we can just provide a list CPA parameters, as an abstract list of
name / values (we started doing that in current draft, 
see MS COnformance 5.2.2 "CPA data" section.)
Any more specific format (e.g. XML doc) is not essential here.
(Can be defined later, mostly for automation.)


2. Message material: 

Proposal:
---------
identify a small set of predefined mesge envelopes - as we have now. 
(see defined in 5.2.5). We can name them and refer to them in the "Template"
column, as we did for header tmpl and payload tmpl.

Comment:
--------
Jeff T. will refine this, but for now it does not hurt to "abstract" the
mesg envelope building process.
These pre-build message envelopes represent the outcome of 
a MIME env building expression, as Jeff will come-up with, so later will be 
replaced by such an expression.


3. Verification step:

Proposal:
---------

Not all testCases in latest version seem to have a specific "Verification" step.
For the sake of standardization, we need to: all steps will have a name,
either GetMessage, PutMessage or Verification, (and maybe also "preCOndition"?
see below)


4. General description of a test case: some suggestions, 
illustrated on TestCase:id:2 

Proposal: 
---------

to improve this test case description, and some others in the same way.

Comment:
--------

- it is simpler to send to "Reflector", than to Initiator, as Initiator 
requires to fully specify the embedded message to be re-sent, 
as well as the "carrier" message. (so in general "Initiator" should be avoided 
when not needed).
- In TestCase:id:2, We would send a message with 1 or more payloads, 
and check that when "reflected", this msge is well formed according to the test.
-PutMessage step: same as current, but with "Reflector" instead. 
-GetMessage step: would only express a filter to correlate with the
previous message:
(/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:ConversationId==’$ConversationId’) and 
(/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:MessageData/eb:RefToMessageId==’$MessageId’ )
-Verification step: would express  the condition "SOAP env is in root part". 
Here, we can try use a formal notation, like you did in testcase:id:5,
or like: MIME-Envelope.part(1) == /SOAP:Envelope/*. 
(the MIME envelope is an object that we can manipulate for conditions, in same
way as we can manipulate for construction.) 
For example, in testcase:id:5: you have the condition:
($MimeMessageStart !== ’’) We should standardize on the manipulation of MIME env/parts (should we use an XPath-like notation for
MIME elements? otherwise would we need too many identifiers such as "MimeMessageStart" ?)
Let us see how that works in other test cases.


5. Expressing Verification COnditions: 

Proposal:
--------

In case it is difficult to "formally" express some conditions, 
in "test message expressions": For test cases where that is really tricky,
then let us use a precise English text description for now.
We'll refine this in a next pass.


6. About some steps that just check if some optional condition
is satisfied (like in testcase:id:3 and testcase:id:10):

Proposal:
---------

We need to give a name to these steps. 
The closest name I can imagine to what the step is doing,  could be "precondition" ?

Comment:
--------

So we would have 4 kinds of steps:
- GetMessage
- PutMessage
- Precondition
- Verification
Then I propose below (section 7) that errors related to such steps
is really about pre-condition failure (no need for "FatalOption"). 


7. About error types: (minor point) We might simplify further,
by not distinguishing "FatalOption" from others: 

Proposal:
---------

we do not need FatalOption: just replace by preCondition.nonApplicable,
Also We need refine the preCondition failure for other steps (specify
"system" vs. "nonApplicable")

Comment:
--------

If we agree that the outcome of a test is either:
(a)- failure because of testbed technical problem (system).
(b)- failure because the logical precondition of a test could not 
be realized for whatever reason (other than system failure).
(c)- failure because the test condition (assertion) was not verified
under normal conditions.
Then it seems we have all we need: if the step is about "optional"
condition, then it should really be a precondition failure:
Let us take two examples: 
- in testcase:id:3, the step that generates "FatalOption" error is
really about a precondition to the test case: if it fails, that means the 
test is non-applicable. 
- in testcase:id:10, there are two steps that check the presence of
the attributes we want to compare. If these attributes are not there
(steps 3 and/or 4 fail) then it is a pre-condition failure: the test is
not applicable. (we should consolidate these 2 steps). 
- So it seems that we could just use: FatalPrecondition.nonApplicable
instead of all these FatalOption.
- For the "PutMessage" steps: a failure could mean either (1) sending was
unsuccessful, (2) message material needed to craft the message was not available.
In such cases, I would say this is a FatalPrecondition.system failure.
- For the GetMessage steps: a failure to receive in time the right message, 
could be again a FatalPrecondition.system. (no matter who is the culprit:
the testbed or the MSH). Of course, in case the "filter" associated
with GetMessage goes beyond correlating the right message (i.e. RefToMesgId +
conversationID) and has some additional condition on some option
that could fail if the right message does not have such option, then
that would be a  FatalPrecondition.nonApplicable. But if
we want really to distinguish, we would then create two steps, like
you did in testcase:id:3: 
- the GetMessage would fail with FatalPrecondition.system
- the "precondition" step would fail with FatalPrecondition.nonApplicable