[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Subject: RE: [ebxml-iic-conform] review test case material for MS conforma nce
-----Original Message-----Jacques,
From: Michael Kass [mailto:michael.kass@nist.gov]
Sent: Wednesday, September 18, 2002 4:18 PM
To: Jacques Durand; 'ebxml-iic-conform@lists.oasis-open.org'
Subject: Re: [ebxml-iic-conform] review test case material for MS conformance
I am attaching a jpg image of a potential schema showing how MIME header values could be
incorporated into our XML syntax for specifying test cases and test steps. Please disregard the
naming "Sendmessage" ( as opposed to "PutMessage") and "ReceiveMessage" ( as opposed to "GetMessage").
This should be a relatively simple way to "tweak" values for messages we wish to send.
MIME header values in a returned message could be imported into XSLT/XPath parameter
values for evaluation in an XPath expression.
Cheers,
Mike
At 04:30 PM 9/17/2002 -0700, Jacques Durand wrote:
Mike, Matt:
here is my take on remaining issues (see attached), after review of
last version Mike sent out 9/8.
I propose to close on several issues with solutions that do not pretend to be final,
but will keep us going - and will allow for refinment later on (i.e. no throw away...).
The general idea is: Once the basics of the test case material are defined
(e.g. vocabulary/naming, msg material identification, CPA parameters, service/actions invoked,
error messages and types...) then it is enough to get going on remiaining test cases.
And we are already there (almost...). Let us not be stopped by:
- issues raised by shortcomings of our material when describing more complex test cases,
- or by the ultimate formal representation of some artifacts (CPA, MIME construction)
Once we have an inital version of all cases, we'll be in a better position
to tackle difficult cases, and we'll be also in better position to refine our test material in a second pass.
Let us try to finalize these few remaining issues this week at latest,
so that the test case definitions can proceed.
Cheers,
jacques
<<ebXMLIICTestCaseCmts_closure.txt>>
Proposed closure on 7 remaining issues on Test Case material:
----------------------------------------------------------
1. CPA:
Proposal:
--------
let us just use a list of name/value pairs for defining
each CPA, in our first version of MS Test Cases.
A small set of CPAs instances is sufficient, each with a defined CPAId.
We refer to these, from each test case (message expressions), by the CPAId.
When we need to name a CPA attribute in "message expression",
we use one of these CPA attribute "names" with $ notation, and "CPA_" as prefix.
E.g.: "Signature" will be referred to as : $CPA_Signature.
[MIKE] - We can take this approach ( assuming a minimum number of CPAs is required )
and/or treat our "mini-cpa" as just another message, and modify using our XPath-like syntax
if necessary.
I agree with the $CPA_..... synatax for the message expression.
Comments:
---------
we can just provide a list CPA parameters, as an abstract list of
name / values (we started doing that in current draft,
see MS COnformance 5.2.2 "CPA data" section.)
Any more specific format (e.g. XML doc) is not essential here.
(Can be defined later, mostly for automation.)
[MIKE] - Agreed
2. Message material:
Proposal:
---------
identify a small set of predefined mesge envelopes - as we have now.
(see defined in 5.2.5). We can name them and refer to them in the "Template"
column, as we did for header tmpl and payload tmpl.
Comment:
--------
Jeff T. will refine this, but for now it does not hurt to "abstract" the
mesg envelope building process.
These pre-build message envelopes represent the outcome of
a MIME env building expression, as Jeff will come-up with, so later will be
replaced by such an expression.
[MIKE] - At the F2F meeting, Jeff and others ( including me)
felt ( and I may be mistaken ) that the message building ( or "MIMING-UP" )
was best handled through
an interface, as opposed to using "templates" for MIME messages..
Use of an "abstract" MIME message would be just that.. for informational
purposes only.
I sent out an "MIMING-UP" example schema a few weeks ago to the group. Perhaps
something like that can be incorporated into our XML syntax.
<jacques> OK, maybe I did not understand well what you guys were proposing.
When you say "interface", you mean mostly that a test case designer will only need
to specify the values of the MIME parameters (that this interface would consume),
right? I.e. the elements you show in MIME-up schema under Sendmessage,
and SetPaylod, e.g. Content-ID, Content-type, etc.
I assume the interface itself only needs be described in the Test Framework,
right? Then that is fine.
3. Verification step:
Proposal:
---------
Not all testCases in latest version seem to have a specific "Verification" step.
For the sake of standardization, we need to: all steps will have a name,
either GetMessage, PutMessage or Verification, (and maybe also "preCOndition"?
see below)
[MIKE] - I believe that all test cases have a verification step. It is represented
through the ErrorStatus.. perhaps should be changed just to "StepType"
to indicate "Precondition" "Test" or "Option".
<Jacques> having thought about it, I retract what I proposed above...
see proposal later in this file.
I would propose avoiding confusing the step names further by mixing "Puts" and "Gets",
with "Verify" and "Precondition" ( like mixing apples with oranges ). Semantically, I
believe we should use two descriptors Put/Set/Get for what the step does, and
Precondition/Verification/Option for describing the nature of the test step.
<Jacques> OK, agree there is a confusing mix here.
See proposal later in file. (I propose we don't even need a "step type".)
BTW: I see a SetPayload in TestCase:id:3, I guess this is where the MIME is built.
Could we just merge this step with the "PutMessage" step before (Step 1)?
Maybe they are, but having two table rows is confusing...
Because the PutMessage is actulally resulting in sending the message, so all assembly
should be under it. (So a single step will assemble all parts of a same message).
(so "SetPayload" op could actually be changed into an expression inside same
"message expression" table field as PutMessage expressions, something like
a statement:
MessageEnvelope = MIME(__,__,__, header1, paylod2)
after the statements that set/modify header and payloads templates.
Opinion?
</jacques>
I believe that "Option" is important.. to tell the test driver that this is not just another
normal precondition.. but that this is an OPTIONAL precondition.. This is important
when ultimately the test case must be evaluated for PASS/FAIL.
<Jacques> I am not sure I follow you on this one: assuming a "precondition" step is
a step used to implement/realize the "precondition" of a Test Requirement,
then if it fails, the test cannot be carried out. We can only say that the test could
not be done (no implication for conformance): that is a "precondition" failure.
Seems to me that all "options" fall into this category?
(We refined further this failure as either "fatalprecondition:notapplicable" or
"fatalprecondition:system".)
So the error we associate with each step, will ultimately define the outcome of
teh entire test:
- FAIL ("fataltest")
- N/A ("fatalprecondition:notapplicable" or "fatalprecondition:system")
- PASS (no error generated, last step completed)
Here is what I suggest:
- we don't really need to specify a "step type". We already have step "operation"
(the puts and gets). We can add another type of "operation", which is pure
logical expression checking (no receiving or sending of messages).
That could be called "check" or "condition". So the fatalOption step
in testcase #3 is a "condition" operation, as well as the ultimate step ("FatalTest").
- the error type associated with any step is actually what determines the outcome
of the test. If we want to show the "optional" character of any step,
("condition", or "get" or "put") we just associate a "fatalprecondition" error to it.
Similarly, "FatalTest" could be in theory associated with any step, not just the
last one. Would that address your "optional" requirement?
</jacques>
4. General description of a test case: some suggestions,
illustrated on TestCase:id:2
Proposal:
---------
to improve this test case description, and some others in the same way.
Comment:
--------
- it is simpler to send to "Reflector", than to Initiator, as Initiator
requires to fully specify the embedded message to be re-sent,
as well as the "carrier" message. (so in general "Initiator" should be avoided
when not needed).
[MIKE] - I see what you mean. Agreed.
- In TestCase:id:2, We would send a message with 1 or more payloads,
and check that when "reflected", this msge is well formed according to the test.
-PutMessage step: same as current, but with "Reflector" instead.
-GetMessage step: would only express a filter to correlate with the
previous message:
(/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:ConversationId==’$ConversationId’) and
(/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:MessageData/eb:RefToMessageId==’$MessageId’ )
-Verification step: would express the condition "SOAP env is in root part".
Here, we can try use a formal notation, like you did in testcase:id:5,
or like: MIME-Envelope.part(1) == /SOAP:Envelope/*.
[MIKE] - I would recommend trying to stay with syntax that is something that we could
transform into a real useable implementation ( XUpdate, XPath or XSLT ). Once we begin introducing
more "abstract" notions into our formalized syntax.. the harder it will be to map to
a real implementable language.
In this example, evaluating a returned message to see if the /SOAP:Envelope is present in what
should be the root part of the message ( which our own ( or other ) TestDriver implementation should logically
do ) sould seem to be an easy way to make this determination, without the need to define
something that would fall outside the scope of simple XPath syntax.
I would strive to define the syntax in an implementable form until we can't ( for example,
human intervention needed to "interrupt/fail" an MSH ).
<Jacques> sounds fair. I was only making my case for the message material
that is normally beyond XML (MIME envelope). But if we can define an XML "canonical representation"
of a MIME envelope, then XPath expressions can indeed apply there too.
(We'll have to provide a Schema for the entire message envelope then...)
</jacques>
(the MIME envelope is an object that we can manipulate for conditions, in same
way as we can manipulate for construction.)
For example, in testcase:id:5: you have the condition:
($MimeMessageStart !== ’’) We should standardize on the manipulation of MIME env/parts
(should we use an XPath-like notation for
MIME elements? otherwise would we need too many identifiers such as "MimeMessageStart" ?)
Let us see how that works in other test cases.
[MIKE] - $MIMEMessageStart is a paramater name for the MIME "start" attribute value, which would be passed
through to an XSLT/XPath processor for evaluation of a particular message.
So my answer is, yes I would use XSLT/XPath
parameter syntax to describe MIME values for evaluation. That is the intention of using the notation:
$MimeMessageStart !== ’’
<Jacques>OK, so XPath notation would be used here also like we did for headers and payload
(asignment expr and logical expr.) (what XSLT needed here for ?).
5. Expressing Verification COnditions:
Proposal:
--------
In case it is difficult to "formally" express some conditions,
in "test message expressions": For test cases where that is really tricky,
then let us use a precise English text description for now.
We'll refine this in a next pass.
[MIKE] - Like for human "interrupts". I agree.
6. About some steps that just check if some optional condition
is satisfied (like in testcase:id:3 and testcase:id:10):
Proposal:
---------
We need to give a name to these steps.
The closest name I can imagine to what the step is doing, could be "precondition" ?
[MIKE] - I chose "Option". It is a precondition.. but a special one. We need to identify
these steps as "special", because we cannot "fail" a test if they are not present.. only
state that the test cannot be evaluated.
<Jacques> would my previous point on this answer this?
(I proposed we don't even qualify steps as "precondition" or "option",
we just describe:
(1) the operation of the step (and I proposed to add a "condition" operation that
would be used to do final verification as well as intermediate "optional" preconditions.),
(2) the error status, which really determines the status of a step "operation" failure
and how it should be interpreted for the entire test (FAIL, N/A)
(do you suggest that sometimes, a step failure should not even "stop" the test, and should
allow to proceed to next step? if so, simply leaving a "blank" error status could permit that.)
</jacques>
Comment:
--------
So we would have 4 kinds of steps:
- GetMessage
- PutMessage
- Precondition
- Verification
Then I propose below (section 7) that errors related to such steps
is really about pre-condition failure (no need for "FatalOption").
[MIKE] - As I suggested above, we should not mix apples/oranges. Get/Set describe
what the step does. Precondition/Verificaiton describe the type of step. I propose
2 different columns/attributes for a step. Both are necessary.
<Jacques> I retracted this proposal... (would the "new" one described address the issue?)
7. About error types: (minor point) We might simplify further,
by not distinguishing "FatalOption" from others:
[MIKE] - Based on your comments above, the need to define "option" steps is
important. I agree that if we define an additional type of step as "option",
then the ErrorType of "FatalOption" is not needed. Only
.notApplicable and .system would seem to be necessary.
We would have:
Precondition.fatalSystem
Precondition.nonApplicable
Option.fatalSystem
Option.nonApplicable
Verification.fatalSystem
Verification.nonApplicable
<Jacques> maybe we have already address that above?
(what would the semantics of "Option.---" be with regard to the final
test outcome (FAIL, N/A, PASS)?) do you see that a step failure with "option"
error type would not stop the test case execution?
I would recommend keeping: ( to identify just what these steps "do" )
PutMessage
SetPayload
SetMime
GetMessage
GetPayload
GetMime
<Jacques> would a consolidation, as suggested on top, be possible?
(just trying to reduce the various operation types, as usually you always have
to do together the "Sets/Put" variants, asme for the "Get". Or,
could a single step gather several of these operations? Just trying to avoid
having too many small steps.)
</jacques>
Proposal:
---------
we do not need FatalOption: just replace by preCondition.nonApplicable,
Also We need refine the preCondition failure for other steps (specify
"system" vs. "nonApplicable")
[MIKE] - I would suggest Option.nonApplicable
Comment:
--------
If we agree that the outcome of a test is either:
(a)- failure because of testbed technical problem (system).
(b)- failure because the logical precondition of a test could not
be realized for whatever reason (other than system failure).
(c)- failure because the test condition (assertion) was not verified
under normal conditions.
Then it seems we have all we need: if the step is about "optional"
condition, then it should really be a precondition failure:
Let us take two examples:
- in testcase:id:3, the step that generates "FatalOption" error is
really about a precondition to the test case: if it fails, that means the
test is non-applicable.
[MIKE] - Agreed. But we HAVE to somehow tell the Test Driver that this is a
"special" optional precondition. I bundled that into a "FatalOption" error status.
I think that things will be simpler if we simply change the "ErrorStatus"
field name to "StepType". Make the step type "Option"
Failure will then be either Option.system or Option.nonApplicable
<Jacques> maybe already addressed above... (you seem to assume
a step failure may still allow driver to proceed further, is that
your definition of "optional"?)
- in testcase:id:10, there are two steps that check the presence of
the attributes we want to compare. If these attributes are not there
(steps 3 and/or 4 fail) then it is a pre-condition failure: the test is
not applicable. (we should consolidate these 2 steps).
- So it seems that we could just use: FatalPrecondition.nonApplicable
instead of all these FatalOption.
[MIKE] - I use the same argument as above. We need to somehow differentiate what
type of steps these are. I suggest naming them "Option" instead of "Precondition".
- For the "PutMessage" steps: a failure could mean either (1) sending was
unsuccessful, (2) message material needed to craft the message was not available.
In such cases, I would say this is a FatalPrecondition.system failure.
[MIKE] - I agree.
- For the GetMessage steps: a failure to receive in time the right message,
could be again a FatalPrecondition.system. (no matter who is the culprit:
the testbed or the MSH). Of course, in case the "filter" associated
with GetMessage goes beyond correlating the right message (i.e. RefToMesgId +
conversationID) and has some additional condition on some option
that could fail if the right message does not have such option, then
that would be a FatalPrecondition.nonApplicable. But if
we want really to distinguish, we would then create two steps, like
you did in testcase:id:3:
- the GetMessage would fail with FatalPrecondition.system
- the "precondition" step would fail with FatalPrecondition.nonApplicable
[MIKE] - I Agree. I think we are converging on the meaning of errors, and hopefully
on types of steps.
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Powered by eList eXpress LLC