[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Subject: Re: [ebxml-iic-conform] review test case material for MS conformance
Mike, Matt:
here is my take on remaining issues (see attached), after review of
last version Mike sent out 9/8.
I propose to close on several issues with solutions that do not pretend to be final,
but will keep us going - and will allow for refinment later on (i.e. no throw away...).
The general idea is: Once the basics of the test case material are defined
(e.g. vocabulary/naming, msg material identification, CPA parameters, service/actions invoked,
error messages and types...) then it is enough to get going on remiaining test cases.
And we are already there (almost...). Let us not be stopped by:
- issues raised by shortcomings of our material when describing more complex test cases,
- or by the ultimate formal representation of some artifacts (CPA, MIME construction)
Once we have an inital version of all cases, we'll be in a better position
to tackle difficult cases, and we'll be also in better position to refine our test material in a second pass.
Let us try to finalize these few remaining issues this week at latest,
so that the test case definitions can proceed.
Cheers,
jacques
<<ebXMLIICTestCaseCmts_closure.txt>>
Proposed closure on 7 remaining issues on Test Case material: ---------------------------------------------------------- 1. CPA: Proposal: -------- let us just use a list of name/value pairs for defining each CPA, in our first version of MS Test Cases. A small set of CPAs instances is sufficient, each with a defined CPAId. We refer to these, from each test case (message expressions), by the CPAId. When we need to name a CPA attribute in "message expression", we use one of these CPA attribute "names" with $ notation, and "CPA_" as prefix. E.g.: "Signature" will be referred to as : $CPA_Signature. [MIKE] - We can take this approach ( assuming a minimum number of CPAs is required ) and/or treat our "mini-cpa" as just another message, and modify using our XPath-like syntax if necessary. I agree with the $CPA_..... synatax for the message expression. Comments: --------- we can just provide a list CPA parameters, as an abstract list of name / values (we started doing that in current draft, see MS COnformance 5.2.2 "CPA data" section.) Any more specific format (e.g. XML doc) is not essential here. (Can be defined later, mostly for automation.) [MIKE] - Agreed 2. Message material: Proposal: --------- identify a small set of predefined mesge envelopes - as we have now. (see defined in 5.2.5). We can name them and refer to them in the "Template" column, as we did for header tmpl and payload tmpl. Comment: -------- Jeff T. will refine this, but for now it does not hurt to "abstract" the mesg envelope building process. These pre-build message envelopes represent the outcome of a MIME env building expression, as Jeff will come-up with, so later will be replaced by such an expression. [MIKE] - At the F2F meeting, Jeff and others ( including me) felt ( and I may be mistaken ) that the message building ( or "MIMING-UP" ) was best handled through an interface, as opposed to using "templates" for MIME messages.. Use of an "abstract" MIME message would be just that.. for informational purposes only. I sent out an "MIMING-UP" example schema a few weeks ago to the group. Perhaps something like that can be incorporated into our XML syntax. 3. Verification step: Proposal: --------- Not all testCases in latest version seem to have a specific "Verification" step. For the sake of standardization, we need to: all steps will have a name, either GetMessage, PutMessage or Verification, (and maybe also "preCOndition"? see below) [MIKE] - I believe that all test cases have a verification step. It is represented through the ErrorStatus.. perhaps should be changed just to "StepType" to indicate "Precondition" "Test" or "Option". I would propose avoiding confusing the step names further by mixing "Puts" and "Gets", with "Verify" and "Precondition" ( like mixing apples with oranges ). Semantically, I believe we should use two descriptors Put/Set/Get for what the step does, and Precondition/Verification/Option for describing the nature of the test step. I believe that "Option" is important.. to tell the test driver that this is not just another normal precondition.. but that this is an OPTIONAL precondition.. This is important when ultimately the test case must be evaluated for PASS/FAIL. 4. General description of a test case: some suggestions, illustrated on TestCase:id:2 Proposal: --------- to improve this test case description, and some others in the same way. Comment: -------- - it is simpler to send to "Reflector", than to Initiator, as Initiator requires to fully specify the embedded message to be re-sent, as well as the "carrier" message. (so in general "Initiator" should be avoided when not needed). [MIKE] - I see what you mean. Agreed. - In TestCase:id:2, We would send a message with 1 or more payloads, and check that when "reflected", this msge is well formed according to the test. -PutMessage step: same as current, but with "Reflector" instead. -GetMessage step: would only express a filter to correlate with the previous message: (/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:ConversationId==’$ConversationId’) and (/SOAP:Envelope/SOAP:Header/eb:MessageHeader/eb:MessageData/eb:RefToMessageId==’$MessageId’ ) -Verification step: would express the condition "SOAP env is in root part". Here, we can try use a formal notation, like you did in testcase:id:5, or like: MIME-Envelope.part(1) == /SOAP:Envelope/*. [MIKE] - I would recommend trying to stay with syntax that is something that we could transform into a real useable implementation ( XUpdate, XPath or XSLT ). Once we begin introducing more "abstract" notions into our formalized syntax.. the harder it will be to map to a real implementable language. In this example, evaluating a returned message to see if the /SOAP:Envelope is present in what should be the root part of the message ( which our own ( or other ) TestDriver implementation should logically do ) sould seem to be an easy way to make this determination, without the need to define something that would fall outside the scope of simple XPath syntax. I would strive to define the syntax in an implementable form until we can't ( for example, human intervention needed to "interrupt/fail" an MSH ). (the MIME envelope is an object that we can manipulate for conditions, in same way as we can manipulate for construction.) For example, in testcase:id:5: you have the condition: ($MimeMessageStart !== ’’) We should standardize on the manipulation of MIME env/parts (should we use an XPath-like notation for MIME elements? otherwise would we need too many identifiers such as "MimeMessageStart" ?) Let us see how that works in other test cases. [MIKE] - $MIMEMessageStart is a paramater name for the MIME "start" attribute value, which would be passed through to an XSLT/XPath processor for evaluation of a particular message. So my answer is, yes I would use XSLT/XPath parameter syntax to describe MIME values for evaluation. That is the intention of using the notation: $MimeMessageStart !== ’’ 5. Expressing Verification COnditions: Proposal: -------- In case it is difficult to "formally" express some conditions, in "test message expressions": For test cases where that is really tricky, then let us use a precise English text description for now. We'll refine this in a next pass. [MIKE] - Like for human "interrupts". I agree. 6. About some steps that just check if some optional condition is satisfied (like in testcase:id:3 and testcase:id:10): Proposal: --------- We need to give a name to these steps. The closest name I can imagine to what the step is doing, could be "precondition" ? [MIKE] - I chose "Option". It is a precondition.. but a special one. We need to identify these steps as "special", because we cannot "fail" a test if they are not present.. only state that the test cannot be evaluated. Comment: -------- So we would have 4 kinds of steps: - GetMessage - PutMessage - Precondition - Verification Then I propose below (section 7) that errors related to such steps is really about pre-condition failure (no need for "FatalOption"). [MIKE] - As I suggested above, we should not mix apples/oranges. Get/Set describe what the step does. Precondition/Verificaiton describe the type of step. I propose 2 different columns/attributes for a step. Both are necessary. 7. About error types: (minor point) We might simplify further, by not distinguishing "FatalOption" from others: [MIKE] - Based on your comments above, the need to define "option" steps is important. I agree that if we define an additional type of step as "option", then the ErrorType of "FatalOption" is not needed. Only .notApplicable and .system would seem to be necessary. We would have: Precondition.fatalSystem Precondition.nonApplicable Option.fatalSystem Option.nonApplicable Verification.fatalSystem Verification.nonApplicable I would recommend keeping: ( to identify just what these steps "do" ) PutMessage SetPayload SetMime GetMessage GetPayload GetMime Proposal: --------- we do not need FatalOption: just replace by preCondition.nonApplicable, Also We need refine the preCondition failure for other steps (specify "system" vs. "nonApplicable") [MIKE] - I would suggest Option.nonApplicable Comment: -------- If we agree that the outcome of a test is either: (a)- failure because of testbed technical problem (system). (b)- failure because the logical precondition of a test could not be realized for whatever reason (other than system failure). (c)- failure because the test condition (assertion) was not verified under normal conditions. Then it seems we have all we need: if the step is about "optional" condition, then it should really be a precondition failure: Let us take two examples: - in testcase:id:3, the step that generates "FatalOption" error is really about a precondition to the test case: if it fails, that means the test is non-applicable. [MIKE] - Agreed. But we HAVE to somehow tell the Test Driver that this is a "special" optional precondition. I bundled that into a "FatalOption" error status. I think that things will be simpler if we simply change the "ErrorStatus" field name to "StepType". Make the step type "Option" Failure will then be either Option.system or Option.nonApplicable - in testcase:id:10, there are two steps that check the presence of the attributes we want to compare. If these attributes are not there (steps 3 and/or 4 fail) then it is a pre-condition failure: the test is not applicable. (we should consolidate these 2 steps). - So it seems that we could just use: FatalPrecondition.nonApplicable instead of all these FatalOption. [MIKE] - I use the same argument as above. We need to somehow differentiate what type of steps these are. I suggest naming them "Option" instead of "Precondition". - For the "PutMessage" steps: a failure could mean either (1) sending was unsuccessful, (2) message material needed to craft the message was not available. In such cases, I would say this is a FatalPrecondition.system failure. [MIKE] - I agree. - For the GetMessage steps: a failure to receive in time the right message, could be again a FatalPrecondition.system. (no matter who is the culprit: the testbed or the MSH). Of course, in case the "filter" associated with GetMessage goes beyond correlating the right message (i.e. RefToMesgId + conversationID) and has some additional condition on some option that could fail if the right message does not have such option, then that would be a FatalPrecondition.nonApplicable. But if we want really to distinguish, we would then create two steps, like you did in testcase:id:3: - the GetMessage would fail with FatalPrecondition.system - the "precondition" step would fail with FatalPrecondition.nonApplicable [MIKE] - I Agree. I think we are converging on the meaning of errors, and hopefully on types of steps.
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]
Powered by eList eXpress LLC