[ebxml-iic-msg] Review of Tes Reqs Level 2

[Jacques Durand <JDurand@fsw.fujitsu.com>]

Title: Review of Tes Reqs Level 2

Mike, Matt:

here is some comments on MS Level 2 reqs, attached.
Once we converge on Level 1,2 and 3 (next week), we should
proceed quickly with the corresponding Test Case definitions (test steps, mark-up),
given the architecture we assumed (Test Driver, Test Service).

NOTE: I noticed that what we describe here (that we call Test Reqs), are also
called "Test Assertions" by others (see OASIS / NIST testing literature)...
We fit quite well the definition they give - a test description between
specification and test case (see below) - and our terminology
(" assertions" field) is also in sync, so thats good! (we just added
"precondition", as a way to better distinguish the conditions under which the test
need apply, so that we have clear idea of how to produce or detect
these conditions ).

-----------------
"A Test Assertion is a statement of behavior, action or condition
that can be measured or tested. It is derived from the specification's
requirements and bridges the gap between the narrative of the specification
and the test cases. Each test assertion is an independent, complete, testable
statement for requirements in the specification.  Each test assertion results
in one or more test Cases. Examples of specifications that included test
assertions as part of their specification include several IEEE (e.g. POSIX)
and ISO standards (e.g. STEP)."

Regards,

jacques

 

NOTE 0: most of my comments below are about rewording so that 
we get consistent wording with what we did in Level 1, and 
also making some test reqs more precise (closer to test cases).
Technically, I realize that what we describe here (that we call Test Reqs), are
called "Test Assertions" by others (see OASIS / NIST testing literature)... 
We fit quite well the definition they give - a test description between
specification and test case - and our terminology
(" assertions" field) is also in sync, so thats good!!! (we just added
"precondition", as a way to specify the conditions under which the test
need apply). 

NOTE 1: this Level 2 doc need be in-synch with Level 1 new format
("preconditions", contexts: received M / sending M)

NOTE 2: the "name" field values need probably be renamed due to test req content
changes below. I did not touch them below.
Ex: r2.1.3 "PersistReliableMsg" must be narrowed.

NOTE 3: When there is an assumed CPA particular set-up, a way to handle
it is to mention it  in "pre-condition" field. You did it for some,
I generalized. (see r2.1.30 below)

NOTE 4: You should use MS spec 2.0 as published on OASIS site: 
your section numbers are not in sync with it after Section 6. 

NOTE 5: my comments below stop just before reviewing r2.2.x (Ordering)
Will be done later...

----------------------------------------------------------------------------

r2.1.1:
Just re-formulating: 
(note: I'm surprised the spec uses MAY - optional - for resending behavior!! (line 1433)
how can we be reliable if not? So in order to translate that in something
testable, I am assuming here that MSH behavior is to resend, when using 
reliabilty - making this part of the test precondition.
So if that is not the case, I'm considering this test simply does not apply...
nothing to verify!!! )

[precondition]: sending M reliably, in case MSH behavior is to resend (per CPA contract?),
in case an Ack is not received.
[assertion]: REQUIRED: the MSH resends M as many times as needed, as specified in "retries",
at the expected time interval.

[precondition]: sending M reliably, in case MSH behavior is to resend,
if an Ack is received before  all retries are expired.
[assertion]: REQUIRED: the MSH stops resending M.

(two subcases)
r2.1.2: 
[precondition]: sending M reliably, in case MSH behavior is to resend,
in case no Ack is received at all.
[assertion]: REQUIRED: the MSH resends M as many times as specified in "retries",
then sends back delivery failure notice to application.



r2.1.3:
I believe the general "persistent" req should be split in subcases, as you are
right it cannot be tested as general statement - but some sub cases may be:
we should split in 2 cases illustrating failures that require persistent store:

[precondition]: sending M, in case transport connection is broken, then reestablished
within the time-to-live value of the message.
[assertion]: REQUIRED: the MSH should resend the message as expected.

[precondition]: sending M reliably, then in case no Ack is sent back, 
and MSH system is shutdown, then if MSH restarted within the time-to-live value of M. 
[assertion]: REQUIRED: the MSH should resend the message as expected, totaling exact number of
 retries.

In addition, we can still restate the overall "persistent" requirement asyou did,
but with a "partial" only coverage, as it is hard to create failures that would demonstrate
right behavior in all cases, especially for recived messages.


r2.1.4:
I believe the spec ref is 6.1, not 7.1. (looks like all your refs are shifted:
see official MS spec as published on OASIS MS TC site)

r2.1.5:
I think this test req is subsumed by future test reqs on duplicate elimination?

r2.1.10: note that this is more a req for multi-hop, as this happens
when several MSHs add such elts. Same remark for r2.1.9 and r2.1.11, that we should move
in Level 3?
Here, At most, we can check that in single-hop
situation that never happens (note: we could combine with r2.1.12):
[precondition]: sending M reliably with an AckRequested element
[assertion]: REQUIRED: only one AckRequested element is present.

r2.1.12:
[precondition]: sending M reliably with an AckRequested element
[assertion]: REQUIRED: the AckRequested element must be targeted to the toParty (either 
no (default) SOAP actor, or SOAP actor must have same URI).

r2.1.13:
[precondition]: receiving M reliably, with an AckRequested element with Signed 
attribute set to "true")
[assertion]: REQUIRED: MSH is sending a signed Ack.
[precondition]: receiving M reliably, with an AckRequested element with Signed 
attribute set to "false")
[assertion]: REQUIRED: MSH is sending a un-signed Ack.

r2.1.18 (rewording)
[precondition]: sending an Ack message unbundled with other (no payload) 
[assertion]: REQUIRED: Message should not contain AckRequested element.

r2.1.19 (rewording)
[precondition]: sending an Error message 
[assertion]: REQUIRED: Message should not contain AckRequested element.

r2.1.21 (rewording)
[precondition]: sending an message with an Ack element
[assertion]: REQUIRED: .... UTC time ...

r2.1.25: maybe we could combine with r2.1.13a ? 

r2.1.26:(rewording)
[precondition]: sending an message with an Ack element
[assertion]: REQUIRED: .... namespace...

r2.1.29:(rewording)
[precondition]: sending an Ack message with no payload and no Error elt
[assertion]: REQUIRED: Service / Action should be...


r2.1.30:(spec ref 6.4.1) subcases:
[precondition]: sending M reliably, in the context of an agreement (CPA)
that requires duplicate elimination (CPA DuplicateElimination = always).
[assertion]: REQUIRED: the DuplicateElimination element is present in header of M.

r2.1.31 is fine (minor rewording:)
[precondition]: sending M reliably, in the context of an agreement (CPA)
that forbids duplicate elimination (CPA DuplicateElimination = never).
[assertion]: REQUIRED: the DuplicateElimination element is not present in header of M.

missing:
[precondition]: sending M reliably, in the context of an agreement (CPA)
that is neutral about duplicate elimination (CPA DuplicateElimination = per message),
in case the party requires duplicate elimination for M.
[assertion]: REQUIRED: the DuplicateElimination element is present in header of M.

r2.1.32: we may try to make it more "testable" (sounds too much like the spec itself)
Two sub-cases involving persistent store:
[precondition]: when receiving twice the same M reliably, with DuplicateElimination element.
[assertion]: REQUIRED: the message is presented only once to the application.
[precondition]: when receiving once M with DuplicateElimination element, shutting down MSH, 
restarting MSH, then receiving a second time the same message with same MEssageId.
[assertion]: REQUIRED: the message is presented only once to the application.


r2.1.33a: precondition should describe more precisely the context of test.

r2.1.33b: this is the "double" of r2.1.31 and  r2.1.30, on Receiving side:
[precondition]: when receiving M reliably, in the context of an agreement (CPA)
that forbids duplicate elimination (CPA DuplicateElimination = never), with 
a DuplicateElimination element in header.
[assertion]: OPTIONAL: error .... is generated
(same for "always and NO DuplicateElimination element )

r2.1.34: probabbly falls under revised r2.1.1 above.

r2.1.35: can we express a more precise test req for that?

r2.1.36:
[precondition]: when sending M reliably, with TimeToLive element, 
[assertion]: the value of TimeToLive satisfies...

r2.1.37: can we express a more precise test req for that?
We can verify that by doing duplicate elimination:
[precondition]: when receiving M reliably, with a PersistDuration value,
and a DuplicateElimination element,
and receiving later the same message M before PersistDuration expires.
[assertion]: REQUIRED: the message is presented only once to the application.

missing:
[precondition]: when receiving M reliably, with a PersistDuration value,
and a DuplicateElimination element,
and receiving later the same message M before PersistDuration expires.
[assertion]: REQUIRED: an Ack similar to first one is sent back to sender.

r2.1.38: can only be partially verified: is stating uniqueness of MessageId.

r2.1.40: use similar wording as r2.1.36 above.

r2.1.41a: (rewording)
[precondition]: when sending M , in context of an agreement (CPA)
where SynCReplyMode is present and not "none"
[assertion]: REQUIRED: the message has a SyncReply element.

r2.1.41b: (rewording)
[precondition]: when receiving M , in context of agreement (CPA)
where SynCReplyMode is present and not "none", and when responding to M. 
[assertion]: REQUIRED: the response is returned on same HTTP connection.

r2.1.43: probably belongs to Level 3 (multi-hop)