OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

ebxml-iic-conform message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Subject: Re: [ebxml-iic-conform] review test case material for MS conformance


  Attached is an annotated update to your comments.


At 04:30 PM 9/17/2002 -0700, Jacques Durand wrote:

Mike, Matt:

here is my take on remaining issues (see attached), after review of
last version Mike sent out 9/8.

I propose to close on several issues with solutions that do not pretend to be final,
but will keep us going - and will allow for refinment later on (i.e. no throw away...).

The general idea is: Once the basics of the test case material are defined
(e.g. vocabulary/naming, msg material identification, CPA parameters, service/actions invoked,
error messages and types...) then it is enough to get going on remiaining test cases.
And we are already there (almost...). Let us not be stopped by:
-  issues raised by shortcomings of our material when describing more complex test cases,
- or by the ultimate formal representation of some artifacts (CPA, MIME construction)

Once we have an inital version of all cases, we'll be in a better position
to tackle difficult cases, and we'll be also in better position to refine our test material in a second pass.

Let us try to finalize these few remaining issues this week at latest,
so that the test case definitions can proceed.



Proposed closure on 7 remaining issues on Test Case material:

1. CPA: 

let us just use a list of name/value pairs for defining
each CPA, in our first version of MS Test Cases.
A small set of CPAs instances is sufficient, each with a defined CPAId. 
We refer to these, from each test case (message expressions), by the CPAId.
When we need to name a CPA attribute in "message expression", 
we use one of these CPA attribute "names" with $ notation, and "CPA_" as prefix.
E.g.: "Signature" will be referred to as : $CPA_Signature.

[MIKE] - We can take this approach ( assuming a minimum number of CPAs is required )
and/or treat our "mini-cpa" as just another message, and modify using our XPath-like syntax
if necessary.
   I agree with the $CPA_..... synatax for the message expression.

we can just provide a list CPA parameters, as an abstract list of
name / values (we started doing that in current draft, 
see MS COnformance 5.2.2 "CPA data" section.)
Any more specific format (e.g. XML doc) is not essential here.
(Can be defined later, mostly for automation.)

[MIKE] - Agreed

2. Message material: 

identify a small set of predefined mesge envelopes - as we have now. 
(see defined in 5.2.5). We can name them and refer to them in the "Template"
column, as we did for header tmpl and payload tmpl.

Jeff T. will refine this, but for now it does not hurt to "abstract" the
mesg envelope building process.
These pre-build message envelopes represent the outcome of 
a MIME env building expression, as Jeff will come-up with, so later will be 
replaced by such an expression.

[MIKE] - At the F2F meeting, Jeff and others ( including me) 
felt ( and I may be mistaken )  that the message building ( or "MIMING-UP" ) was best handled through 
an interface, as opposed to using "templates" for MIME messages.. 
Use of an "abstract" MIME message would be just that.. for informational
purposes only.  
  I sent out an "MIMING-UP" example schema a few weeks ago to the group.  Perhaps
something like that can be incorporated into our XML syntax.

3. Verification step:


Not all testCases in latest version seem to have a specific "Verification" step.
For the sake of standardization, we need to: all steps will have a name,
either GetMessage, PutMessage or Verification, (and maybe also "preCOndition"?
see below)

[MIKE] - I believe that all test cases have a verification step.  It is represented
through the ErrorStatus.. perhaps should be changed just to "StepType"
to indicate "Precondition" "Test" or "Option".   
  I would propose avoiding confusing the step names further by mixing "Puts" and "Gets",
with "Verify" and "Precondition" ( like mixing apples with oranges ).  Semantically, I 
believe we should use two descriptors Put/Set/Get for what the step does, and 
Precondition/Verification/Option for describing the nature of the test step. 
I believe that "Option" is important.. to tell the test driver that this is not just another
normal precondition.. but that this is an OPTIONAL precondition.. This is important
when ultimately the test case must be evaluated for PASS/FAIL.

4. General description of a test case: some suggestions, 
illustrated on TestCase:id:2 


to improve this test case description, and some others in the same way.


- it is simpler to send to "Reflector", than to Initiator, as Initiator 
requires to fully specify the embedded message to be re-sent, 
as well as the "carrier" message. (so in general "Initiator" should be avoided 
when not needed).

[MIKE] - I see what you mean. Agreed.

- In TestCase:id:2, We would send a message with 1 or more payloads, 
and check that when "reflected", this msge is well formed according to the test.
-PutMessage step: same as current, but with "Reflector" instead. 
-GetMessage step: would only express a filter to correlate with the
previous message:
(/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:ConversationId==’$ConversationId’) and 
(/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:MessageData/eb:RefToMessageId==’$MessageId’ )
-Verification step: would express  the condition "SOAP env is in root part". 
Here, we can try use a formal notation, like you did in testcase:id:5,
or like: MIME-Envelope.part(1) == /SOAP:Envelope/*. 

[MIKE] - I would recommend trying to stay with syntax that is something that we could
transform into a real useable implementation ( XUpdate, XPath or XSLT ).  Once we begin introducing
more "abstract" notions into our formalized syntax.. the harder it will be to map to 
a real implementable language.
In this example, evaluating a returned message to see if the /SOAP:Envelope is present in what
should be the root part of the message ( which our own ( or other ) TestDriver implementation should logically
do ) sould seem to be an easy way to make this determination, without the need to define
something that would fall outside the scope of simple XPath syntax.
I would strive to define the syntax in an implementable form until we can't ( for example,
human intervention needed to "interrupt/fail" an MSH ).

(the MIME envelope is an object that we can manipulate for conditions, in same
way as we can manipulate for construction.) 
For example, in testcase:id:5: you have the condition:
($MimeMessageStart !== ’’) We should standardize on the manipulation of MIME env/parts (should we use an XPath-like notation for
MIME elements? otherwise would we need too many identifiers such as "MimeMessageStart" ?)
Let us see how that works in other test cases.

[MIKE] - $MIMEMessageStart is a paramater name for the MIME "start" attribute value, which would be passed
through to an XSLT/XPath processor for evaluation of a particular message.  So my answer is, yes I would use XSLT/XPath
parameter syntax to describe MIME values for evaluation.  That is the intention of using the notation:
$MimeMessageStart !== ’’

5. Expressing Verification COnditions: 


In case it is difficult to "formally" express some conditions, 
in "test message expressions": For test cases where that is really tricky,
then let us use a precise English text description for now.
We'll refine this in a next pass.

[MIKE] - Like for human "interrupts".  I agree.

6. About some steps that just check if some optional condition
is satisfied (like in testcase:id:3 and testcase:id:10):


We need to give a name to these steps. 
The closest name I can imagine to what the step is doing,  could be "precondition" ?

[MIKE] - I chose "Option". It is a precondition.. but a special one.  We need to identify
these steps as "special", because we cannot "fail" a test if they are not present.. only
state that the test cannot be evaluated.


So we would have 4 kinds of steps:
- GetMessage
- PutMessage
- Precondition
- Verification
Then I propose below (section 7) that errors related to such steps
is really about pre-condition failure (no need for "FatalOption"). 

[MIKE] - As I suggested above, we should not mix apples/oranges.  Get/Set describe
what the step does.  Precondition/Verificaiton describe the type of step. I propose
2 different columns/attributes for a step.  Both are necessary.

7. About error types: (minor point) We might simplify further,
by not distinguishing "FatalOption" from others: 

[MIKE] - Based on your comments above, the need to define "option" steps is
important.  I agree that if we define an additional type of step as "option",
then the ErrorType  of "FatalOption" is not needed.  Only 
.notApplicable and .system would seem to be necessary.
We would have:


I would recommend keeping: ( to identify just what these steps "do" )



we do not need FatalOption: just replace by preCondition.nonApplicable,
Also We need refine the preCondition failure for other steps (specify
"system" vs. "nonApplicable")

[MIKE] - I would suggest Option.nonApplicable


If we agree that the outcome of a test is either:
(a)- failure because of testbed technical problem (system).
(b)- failure because the logical precondition of a test could not 
be realized for whatever reason (other than system failure).
(c)- failure because the test condition (assertion) was not verified
under normal conditions.
Then it seems we have all we need: if the step is about "optional"
condition, then it should really be a precondition failure:
Let us take two examples: 
- in testcase:id:3, the step that generates "FatalOption" error is
really about a precondition to the test case: if it fails, that means the 
test is non-applicable. 

[MIKE] - Agreed. But we HAVE to somehow tell the Test Driver that this is a
"special" optional precondition.  I bundled that into a "FatalOption" error status.
I think that things will be simpler if we simply change the "ErrorStatus"
field name to "StepType".  Make the step type "Option"
 Failure will then be either Option.system or Option.nonApplicable

- in testcase:id:10, there are two steps that check the presence of
the attributes we want to compare. If these attributes are not there
(steps 3 and/or 4 fail) then it is a pre-condition failure: the test is
not applicable. (we should consolidate these 2 steps). 
- So it seems that we could just use: FatalPrecondition.nonApplicable
instead of all these FatalOption.

[MIKE] - I use the same argument as above. We need to somehow differentiate what
type of steps these are.  I suggest naming them "Option" instead of "Precondition".

- For the "PutMessage" steps: a failure could mean either (1) sending was
unsuccessful, (2) message material needed to craft the message was not available.
In such cases, I would say this is a FatalPrecondition.system failure.

[MIKE] - I agree.

- For the GetMessage steps: a failure to receive in time the right message, 
could be again a FatalPrecondition.system. (no matter who is the culprit:
the testbed or the MSH). Of course, in case the "filter" associated
with GetMessage goes beyond correlating the right message (i.e. RefToMesgId +
conversationID) and has some additional condition on some option
that could fail if the right message does not have such option, then
that would be a  FatalPrecondition.nonApplicable. But if
we want really to distinguish, we would then create two steps, like
you did in testcase:id:3: 
- the GetMessage would fail with FatalPrecondition.system
- the "precondition" step would fail with FatalPrecondition.nonApplicable

[MIKE] - I Agree.  I think we are converging on the meaning of errors, and hopefully
on types of steps.

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Powered by eList eXpress LLC