Re: Durand [ebxml-iic] 9/2/2002: Test Case Material Development

[Michael Kass <michael.kass@nist.gov>]

To all,

    Attached is an updated ebXML MS v2.0 Abstract Test Suite document, 
along with
additional comments for discussion at today's meeting. My comments begin 
with [MIKE] -

Thanks,
MK
In order to complete Test Case material design:
----------------------------------------------

1. [Matt]: specify CPA subset used. Which format should we pick? 
So far we get two candidates: tpaSample.xml? minicpa.xml from Hatem?
(quick comments on tpaSample.xml:
- SyncReplyMode options missing
- what is the distinction between Ack "expected" and "requested"?)

2. [Jeff]: we need to finalize message template data, in particular
- the way we parameterize these templates (XPath?)
- the way we build complete MIME envelopes and their content (either
using again a template approach - restrictive but simple - or some other
doc building.)

3. [Mike, Monica?] mapping of Test Cases to Test Assertions.
Can we really assume that there is always 1 test case for each test assertion?
I am sure that is the case for 98% of them, but it would be prudent to not preclude
the possibility of more than 1 test case for an assertion. A test case is always
more concrete than an assertion, could there be situations where it makes sense to
have two or more tests for a same assertion that we would not split?


My question is in fact: do we really have to decide on this, or can we adopt an
Test Case ID scheme that allow for this if we need it later:
it can be same as current assertion ID (e.g. urn:semreq:id:3), and in case we have 1-to-n, 
we can use additional letters: e.g. urn:semreq:id:3a, urn:semreq:id:3b, ... ? 
or dot numbering: urn:semreq:id:3.1, urn:semreq:id:3.2... 
would that be an issue?

[MIKE] - I think that we can incorporate multiple test cases for one requirement/assertion.  We could
allow multiple test cases with the same ID.  The test harness could be designed to traverse 
through the test requirement tree and execute ALL test cases having the same 'base' ID as the requirement.
I favor a 'urn:semreq:id:3:1  urn:semreq:id:3:2 ...  3:3' etc.

[mm1: In all honesty, I think over time we could have M-M (test-case to assertion) so I suggest we
provide for extensibility.  This is a discussion item similar to my previous one regarding
aggregation of test cases for a type of scenario or lifecycle of functionality testing.
And, to, we need to allow for granularity of the test assertion and cases.]

4. Test Case table formatting [Mike,...]:
- Test Case ID field: see above remarks on numbering. (by the way, why "semreq"?)

[MIKE] - This URN was used because it matched the name of the to name conformance testing 
'semantic requirements'. We could shorten it to "req" as that may be a little clearer to someone looking at the abstract
test list.

- "Action Element" field: we could use more intuitive "step names", 
e.g. for the sending of message: "SendMessage" instead of "SetMessage".

[MIKE] - The current reasoning for useing "Set" and "Get" is because we are constructing/modifying
templates and payloads.  "Send" and "Receive" would not seem to be appropriate verbs to use for 
"setting" MIME attributes, message content and payloads attributes.

- Also I strongly suggest that we make the "verification" of the test, a separate
and final step. (could be called "Verification").\

[mm1: YES! Separate the test from the verification - this is a key concept for a test
framework and just because there is a test, this does not infer verification.]

[MIKE] - The thing to remember with these "abstract" tests is that they are just that.. and do not
represent how we will ultimately code and run the tests.  I agree that the actual verification test should be
"broken out" from other portions of the test case ( such as preconditions verification and message ID correlation  ). 
Whether or not it is a seperate "test step" is an implementation detail. 
( For example, if the test driver was implemented using Schematron, then all filtering of a received message 
could be performed using a single test step... since Schematron provides for individual reporting of each filter
result. )
In the abstract test description, the verification test can/should be seperated from other portions. I have done that with the
attached modified abstract test suite.

- "Party" field: probably not needed, as it is always the TestDriver, as per our
definition of what a "step" is: an event that is always observable in TestDriver.


[mm1: What if in the future the Party is actually observable in the Test Service?]

[MIKE] - Based upon our discussion at the F2F, I removed this field.  If, however the 
the implementation of the test harness will allow observation/initiation 
of other nodes besides the testing party, then I agree with Monica that we should include a Party field.


- "ErrorStatus" field needs revision. See below "Test failures".
- ErrorMessage: for each step is fine.
- "XPath" field: let us use a better name... should be more general , 
like "message expression" or something like that.


[MIKE] - I've addressed "ErrorStatus below", and Changed "XPath Expression" to "Test Message Expression" for now.

5. XPath / message expressions [Mike, Matt]:
- some XPath expressions are for building message material ("SetMessage" action),
some are for expressing "filters" to select the right message (GetMessage).
It would be good to distinguish them in syntax, e.g. the assignment operator "="
be distinguished from equal operator, like in programming languages (e.g. "==").


[MIKE] - I agree that this makes things clearer ( assignment vs. comparison )  Then again,
the use of the SetMessage vs. GetMessage operations can also delineate between an assignment expression and a comparison expression.

- GetMessage steps should not be aggregated with the final Verification
condition: GetMessage only contains filters to select the right message.

[MIKE] - Agreed.  I believe that this is an implementation detail.  If our test harness can
differentiate between the actual test and test correlations and test preconditions, then it may not 
be necessary to break the test into seperate test "steps".  It remains to be seen how this will be
implemented.

- For the final step (or Verification): will contain the boolean expression
that defines success (currently merged with the "filter" expression of GetMessage step
in current draft.)

[MIKE] - I have seperated the final verification from the rest of the test case in the latest version of the abstract test suite.
How it is actually implemented remains to be determined.

[mm1: See question above about where we could have 1-M (test assertion to test case).  Is it not applicable
that the verification as a complement to but not part of the test, we do have this condition - 1 test assertion,
that results in (1) test of the case, and (2) verification of the case?  Either way the verification should be separate.]

[MIKE] - I am not sure that I completely understand Monica's question here.  However, I believe that we can have a 
"higher level" evaluation of a test requirement ( assertion ), that can consist of an aggregation of 
test cases that must all be verified in order to "pass/fail" the requirement.  Simply allow a 1-M 
relationship between a requirement and test cases, and aggregate their result.


- Use of parameters ($MessageId, etc.): it seems that these parameters need sometimes
to be set to current (e.g. received) material. That is not clear how it is done (see Case id:3)

[MIKE] - It is true that this is not evident how this will be done.  The abstract tests only define
in a general sense how the tests will be done.. not the implementation specifics.  However, I believe that all of the 
parameters defined in the XPath statements can be "set" based on the filtering of these parameters for 
an incoming message that also passes the examination of any message that "passes" the XPath correlation filters. 
In such a case, MIMEContentType, MIMEStart... and other parameters would be "Set"
based on the 'current' message that satisfies a message correlation filter.  

We face two issues:
(a) how to "remember" message material from past test steps?
We could use XPath-based assignment, e.g. a GetMessage could contain filter
expressions as well as assignment expressions: e.g. $MessageId = <xpath expr>

[MIKE] - We need to decide what we need to "remember".  Do we really need anything
more than RefToMessageId, ConversationId and CPAId for correlating messsages?  I think that
we need some examples here.

(b) Across several steps, as several message are involved, and we may want to
refer material from  more than 1 step, we can use step# to identify the parameter: 
$1MessageId, $2MessageId...

[MIKE] - I see what you are saying.  That is certainly an option, if it is necessary.


- advanced verification conditions: sometimes verification conditions need more
than just constraints on message material: e.g. check that step N completed
within 10sec from Step M. It seems anyway we need to set a timeout for step completion.
What else? How to improve script language for this? When it comes to checking
that we got say 3 messages of a kind, e.g. for retries in reliability, 
could that be an enhancement of the GetMessage step? (where we would specify how
many messages of this kind need be received for the step to complete?)

[MIKE] - I believe that we can script these tests by expanding our <SetMessage> and
<GetMessage> elements to include additional attributes that run our Test Driver. Including
additional attributes to define our test cases should allow us to provide both additional filters
outside of and parameters to XPath expressions.


[mm1: Regardless of what path is chosen, keep in mind - extensibility and discrete definition.  If we continue
to embed important data elements in the expressions, our capability to identify it (should it need to be searched
for and found) may be more difficult.  I keep thinking about the database days when we concatenated data and then
expected to search for an text string, augh.]


[mm1: ON advanced verification conditions, I believe we are seeing we not only have conditions on the assertion but pre-
and post-conditions on the test case itself as well as the verification phase (not really metadata though).
Can we attach those conditions to the test case itself, where the conditions are "included" with the test case?]



6. Test Verification , and Test failures [Mike, Matt]:
- A separate step for this would be good, as mentioned above.

[MIKE] - I agree, if test harness implementation requires a separate test step for verification, then we should do this.

- sometimes a successful test needs to verify that no error message wwas received,
in addition to completing all its steps. How do we do that? Should we define
"Exception step(s)" to a test case, that will capture messages that should NOT occur...
and then when completed, generate test failure?


[mm1: We have discussed exceptions at length in BCP, and exceptions may not always be errors - they may be less 
traveled paths.  So, to err on the side of future function, I would suggest you allow for both exception
steps and outcome (test failure or test incomplete? - do we see these as different? - links with Jacques question
below).

[MIKE] - If the specification is specific about what errors SHOULD NOT OCCUR, then we should feel confident 
in writing tests to verify this.  If the spec specifically says that NO ERROR MESSAGES SHOULD BE GENERATED,
then I would follow its literal interpretation and design a test accordingly. Exceptions can be handled as
"informational", and flagged accordingly.. but they should not fail a test if the specification does not
indicate that they are illegal, and all other requirements for that test have been met.


- important to distinguish two types of failure for a Test Case:
(a) "operation" failure, resulting from the impossibility to carry out the test properly.
e.g. some test step could not complete, for some reason unrelated to the spec requirements
that we are trying to test. 
Typically, this happens when the Test Requirement "pre-condition" cannot be realized.
In such case, the conformance report should NOT conclude that MSH implementation 
is not conforming, just that the test could not be performed.

[MIKE] - I added an error status of 'FatalStep' to flag these particular test steps.
It applies to any test step ( other than a conformance test or a special 'pre-condition evaluation step' )
I believe, based on the way the first 70 tests are designed, that we can set up a sequential evaluation 
of test steps, and exit gracefully from a test case whenever an error occurs at any test step.  I have not
seen any test cases yet where we would wish continue to proceed with the test case 
 upon encountering either an operational failure, pre-condition or conformance test step failure.


(b) "conformance" failure, clearly showing that the spec requirement is not satisfied
by the MSH implementation.

[MIKE] - I named the error status 'FatalTest' in the attached abstract test suite list for this kind of test step.

Generally, the failures (a) correspond to some step that could not be completed.
So we could associate with each step either type of error: (1) failure causing
"operation" failure, (2) failure causing "conformance" failure.
- should we also make room for a "failure" expression in the verification step?
In other words, in case the "success" expression is not satisfied, we may
still need to distinguish the kind of test failure. A specific error message
could be associated with each kind.

[mm1: May be 3 types of 'failures' - not certain if failures is the best word - (1) system - outside
of the test condition that affects the test, (2) operation - condition completion to test the assertion, and
(3) conformance failure - affecting testing test assertion itself as defined in a test case(s).  Can
we differentiate a system from an operation failure? How will we be able to differentiate which failure
type occurred?  Some of it may be outside of the visibility or scope of the test framework - does this require
data from the testing node we interact with?]

[MIKE] - I do not think that we can differentiate between system failure and an operational failure, particularly
in "correlation test steps", where we are trying to match up received messages with sent ones.  It would be hard to tell
if a message simply was not sent, or didn't correlate ( i.e.. mismatched refToMessageIds/ConversationIds ).  I'm
submitting an attachment with 3 proposed "error status" classifications:
FatalStep - a step operation that failed due to either message examination ( particularly in message "correlation" test steps )  or due
to system failure
FatalPrecondition - a step operation that failed due to either system failure (unlikely) 
or because an optional feature was not present in candidate MSH - THIS ERROR CONDITION DOES NOT RESULT IN A CONFORMANCE TEST CASE FAILURE
FatalTest - a step operation that failed due to either system problems (unlikely) or because of conformance problems with candidate MSH
*** Since the message correlation step should always come FIRST in the test step sequence for evaluating received messages, all other errors that follow
for a received message should not be a "FatalSystem" nature, but either a precondition or a conformance type of error.

Attachment: ebxml_ms_20_abstract_tests.doc
Description: MS-Word document