[ebxml-iic-msg] Further comments on refining level 1 MS ConformanceTest Requirements

[Michael Kass <michael.kass@nist.gov>]

Jacques,

   Attached are further comments and clarifications regarding level 1 MS 
test requirements.

Mike
My comments tagged: [MIKE2] 
---------------------------

NOTE1: it is useful to specify, for each test item, the context of its verification:
Should it be verified: 
(1) for messages received by the candidate MSH? 
(2) for messages generated  by the candidate MSH?
(3) on the callback from application interface (e.g. some errors)?
So I'll add this info for each test item.

[Jacques]: one way to do that, is to use the "Condition" part of the tset req, 
which states what are the pre-conditions required for applying this test
(more on that below)
So for example, we could now have in a COndition part that was empty before: 
"for each generated message" , in case the test applies to generated messages.
Or, we could have "for each received message", in case it applies to received
messages. 
And if Condition used to contain a cond "C1", then well have:
"for each received message that satisfies C1", etc.
What do you think?

....

[MIKE2] - I think that a better place for a "for each generated/received message" ( IF THERE IS NO
EXISTING CONDITION )  would be in the Assertion.  The <Condition> and <Assertion> elements are going to logically drive the test
harness.  Each <Condition> must have a corresponding <TestCase> to execute . Each <Assertion> must have a corresponding
<TestCase> to execute. Including a "for each" in each "empty" <Condition> means that the test driver must evaluate that <Condition> by
executing a corresponding <TestCase>.. for which there is none.  The only TestCase that exists in the
case of an "empty" <Condition> is the <TestCase> for the <Assertion>. So if you want to add this piece of 
information to make the requirement more understandable, a better place for it would be in the 
<Assertion>. I have no problems modifying the requirements to add this to the <Assertion>. Please
see below for more explanation of this topic.



r1.1.2:(generated M)
can we detail a little more what that means?

[MIKE] - I believe that the following excerpt from the SOAP With Attachmentts 
specification provides  the requirements for validating SOAP MIME headers:

The primary SOAP 1.1 message must be carried in the root body part of the 
Multipart/Related structure. The type parameter of the Multipart/Related 
media header will always equal text/xml. 
Referenced MIME parts must contain either a Content-ID MIME header structured 
in accordance with RFC 2045, or a Content-Location MIME header structured 
in accordance with RFC 2557. 
It is strongly recommended that the root part contain a Content-ID MIME 
header structured in accordance with RFC 2045, and that in addition to 
the required parameters for the Multipart/Related media type, the start 
parameter (optional in RFC 2387) always be present.

I added 4 conditions against the above.

[Jacques]: That is definitely more precise than before.
a bit of editorial job still needed I think: i am still uncomfortable with the
way we use condition/assertion, although we improved a lot:
even though the distinction Condition / Assertion is not much important 
from a test execution viewpoint, I believe we could/should use it with the 
following meaning:

[MIKE2]: Because of the way that we are designing the test harness, Condition/Assertion IS 
important from a test execution standpoint. A Condition <Clause> and its nested <Condition>s 
must be evaluated as true/false before the <Assertion> is evaluated true/false.  
Failure of the <Condition> or its enclosing <Clause> prohibits evaluation of the <Assertion>.  
Conditions are not just provided for human readability.. they must have a matching <TestCase> with real test data associated
with them.  A parser is going to traverse the test requirements logic tree and execute
<TestCase>s based upon the structure of that tree.  So if we supply a <Condition>
where none exists, we must have a matching <TestCase> to test that condition. So putting something
like "for all received messages" as a <Condition> does not really evaluate to a true/false.. it 
cannot stand alone as testable.  

A solution for making more readable ( and functional )  test <Conditions> and <Assertions> is:

1) If the requirement is a conditional assertion ( requires certain conditions to be met before the
assertion can be evaluated ).. then put the "for each genereated/received message" at the beginning
of the <Condition>

2) If the requirement is an unconditional assertion ( no <Condition> element is present ), 
put the "for each generated/received message" at the beginning of the <Assertion>

How does this sound?


o it should help readers understand *in which case* the core of the 
test requirement applies, i.e. if the COndition is not satisfied, 
then we cannot say that the test req has been actually verified: 
just that no matching situation occurred.

[MIKE2] - I agree. It would not be hard to modify the requirements using the solution I 
suggest above to better clarify the context of the requirement.

o In case we cannot drive the test harness so that it generates such a situation
that satisfies the "Condition" part of a test req, that means we cannot guarantee
coverage of this test req (coverage = null). In case we can only generate
a subset of the situations, coverage is "partial".

[MIKE2] Agreed


[Jacques]: so to get back to our test req item:
- in r1.1.2, I think the interpretation should be:
[Condition]:"IF the message is multipart MIME (or in any case is NOT a pure SOAP msge) 
[assertion]: THEN it should satisfy SwA requirements"
- So the COndition part (which maybe should be renamed "Pre-condition" to avoid
ambiguity) tells that this test verification only applies to multipart messages
(because straight SOAP messages are allowed when no payload...).
- So the test req item could be:
[precondition]: for each generated message, if is multipart MIME, OR is not 
enveloped like a pure SOAP ("text/xml")
[assertion]: the message must then be SwA compliant, which means: 
(...all you listed  previously in "Condition")
- if all you listed in Condition part is enough to capture SwA, we don't need 
to restate what you had before in previous Assertion (as it is just a recap of SwA).
- We might have to revisit the content of [Pre]condition/Assertion
throughout the other test reqs (I'll try for some). 


[MIKE2] - Here is how we can restructure and reword this requirement:

<Precondition>For each generated mesage, if it is multipart MIME or not text/xml</Precondition>
<Assertion>The primary SOAP message is carried in the root body part,
the "type" paramater of the Multipart/RElated media header is "text/xml" and
MIME parts contain a CID MIME header or a Content-Location MIME header structed in
accordance with RFC2557, and the "start" parameter is present.</Assertion>


r1,1,5: (generated M)
[Jacques]:
A way to state that in a "testable" way is:
[precondition]: In case the mesage is NOT a multipart MIME
[assertion]: it must have no payload.
(Note 1: only "partial" testing can be done here, as we can't force an MSH to
satisfy the pre-condition: we have no control on it.)
(Note 2: this is also a "double test": according to spec 2.1.2, we should check 
for Received M as well:
[precondition]: For each received message, if it is NOT a multipart and is
a well-formed SOAP message without payload,
[assertion]: The MSH should accept it.)

[MIKE2] - Here's my coding to restructure this ( also "doubled" it):

<Precondition>For each receive message, if the message is not multipart MIME AND
the message has no payload</Precondition>
<Assertion>The receiving MSH should accept it.</Assertion>


<Precondition>For each generated message, if the message is not multipart MIME</Precondition>
<Assertion>The message must have no payload.</Assertion>


[Jacques]:
r1.1.9: 
- The way this test req is stated is not wrong, but too restrictive:
it does not cover cases where there are SOME payload elts that are identified 
in Manifest, but SOME are not... (in fact, that is stated in r1.1.10,
but I believe r1.1.9 is actually completely subsumed by r1.1.10 as stated,
i.e. if 10 is satisfied, then 9 always is...)
It needs be rephrased:

[MIKE2] - Section RS 2.1.4, paragraph #4 states that "The contents of each Payload Container
MUST be identified in the ebXML Message Manifest element within the SOAP body."  So, if 
I read this right, there would not be a case where SOME payload elts are identified in the
Manifest, and SOME are not. r1.1.9 tests the case where there are no application payloads, and
should therefore have no Manifest References.


[MIKE2] On the other hand, r1.1.10 addresses the case where there are payloads present, and tests
whether a matching Manifest Reference element is present.  So I see both r1.1.9 and 1.1.10 as
legitimate tests against the spec.


[precondition]: for each generated Message M, if M has some payload containers 
[assertion]: each container content must be identified in Manifest.
- we should state "for generated Message" (generated M)
I really believe we should always add this "context", either as a new column,
or as part of the "precondition" ("generated Message M has no payload
identified in Manifest.) 

[MIKE2] - As stated above, I will modify all requirements to reflect this context either in the
<Condition> or <Assertion> as appropriate.

- now, that sums-up previous r1.1.9 and r1.1.10. But another point in 2.1.4 is:
"If there is no application payload within the Message Package then a Payload Container MUST NOT be present."
So here, I think we need to check that if there are payload containers, then they are not empty.

[MIKE2] - Agreed.. will add this requirement


r1.1.7:  (received M)
The "charset = UTF-x" MIME parameter can still - and should - be verified?

[MIKE] - Designated test coverage as "partial".... will only veritfy that 
UTF-X is the value supplied for the MIME content-type

r.1.1.11:(received M)
Partial check can be done? (by falsification of a message: we put all possible
other MIME headers in, and just see that the message is well received.)

[MIKE] - Fair enough.  Designated test coverage as "partial", based on the assumption
that we will be putting in every possible ( but perhaps not all ) MIME headers.

[Jacques]:
We should interpret a little more what "ignore" means (spec 2.1.5) from a tset req viewpoint:
I would say it means no error with code "Error" is generated, and behavior is same
as if feature wasn't there. 

[MIKE2] - I'm a little concerned about the vaguery of this spec statement.  It is certinly wide
open to interpretation. This is another spec reference that should be brought to the attention of 
the MS spec writing team. In the meantime, we can check that no "Error" elements are generated... 
but our interpretation is not certain.  We should perhaps "tag" these test requirements as our own
interpretation, rather than give the impression that this is a specified requirement.


r1.1.25: (received M)
The rejection must be in "accordance with SOAP", I believe there should be a SOAP Error,
that should then translate into an MSH error. So by falsification, we should be
able to partially check this (i.e. just for one instance of a non-recognizable header.)

[MIKE] - Agreed.


r1.2.1: (generated M)
Actual test is that the "From" and the "To" are present. (identifying the 
party can't be verified)

[MIKE] - Agreed. Since we are now only checking if "From" and "To" are present, 
I removed this requirement because schema validation ( done for every message 
generated by the candidate ) 
will always verify whether both  "From" and "To" are present.

r1.2.3: (generated M)
As worded, I believe we have no way to verify that... however other validation can be done:
(uniqueness of type attr value, position of "role" element if any)

[MIKE] - Agreed. Substituted partyId attribute "uniqueness" requirement
for this requirement.


r1.2.5:  (received M)
Note: this is the kind of test that can be "doubled", i.e. verified for both received
and generated messages. In that case, we should make it two test req items:
For rceived M: stated as you said (we SHOULD observe an error message)
For generated M: 
(generated M) either the PartyId type attr is present, or if it is not, its value MUST be URI.

[MIKE] - "Doubled" this requirement into a "send" and "receive" scenario.

r1.2.7:  (received M)
Is the spec ref wrong (3.2.1)? it seems to refer to XLinks in my version.
Anyway the scope of that test seems too broad - too high level - normally should be split in several
test items. Can we ignore it and rely on more specific discrepancy checks as they 
show up later in the spec? 

[MIKE] - Agreed. Removed this requirement. CPPA/message discrepency tests will have to
be done on a lower-level, individual basis.

r1.2.9: (generated M)
Only partial verification possible: this requirement actually involves the application 
behavior: this is a usage guideline for the app... if the app does not set the conv id right, 
no way can check that: could be another conversation. We may just verify that the conv id
set by our test service is well reported in the msg?
We should also add a falsification test (generated M, here) for uniqueness of convID 
within a CPAid, where the app (our test service) on top of candidate MSH try to send two messages 
with different convID but same CPAid. Normally, the candidate MSH should complain - report
error to the app.

[MIKE] - If tests are performed serially, then it would seem that this 
type of a test could be done.  You assume that other messages will be 
sent simultaneously while testing is taking place.  Why not require that 
this test suite be run standalone against candidate ebXML MS implementations?
For now, I am following your recommendation and changing the test coverage to "partial"

[Jacques]: no, even for a "standalone" conversation: I mean that convID may change under
the control of an application, from one message to the other,
and we are not authorized to see anything wrong in this... This is an application
behavior. Unless we refer to the CPAId: the uniqueness of convID within
the same CPAId: 
[precondition]:  for generated messages , if several of them relate to same CPAId
[assertion]: these messages should all have same conversation ID. 

[MIKE2] - Here is my coding of your suggestion:

<Precondition>For all generated messages with the same CPAId</Precondition>
<Assertion>The message contains the same ConversationId</Assertion>

So again this depends on tha app side: can't cover that fully in a testbed. 
But if we assume an Error must be generated
if that is not the case, then we can cover the following test req: 
[precondition]:  for generated messages , if two of them relate to same CPAId 
but have different conv ID:
[assertion]: the MSH should generate an error. 
(and the test could be doubled with a "Received" version.)


[MIKE2] - Here is my coding for this:

<Precondition>For all received messages with the same CPAId but different ConversationId</Precondition>
<Assertion>An Error element element should be generated in the response message.</Assertion>


r1.2.10: (generated M)
Remind in test description that this is about the Service element.
And also, this is a "double" test item:
(generated M): "condition C must be satisfied."
(received M): "if condition C is not satisfied, an error should/must be generated."


[MIKE] - Done.

r1.2.11:  (received M)
probably partial verification: we can only verify for some particular case of bad message
we generate (falsification).

[MIKE] - Done.

r1.2.14: (generated M)
Only partial verification possible: it depends on the application to set the RefToMsgId properly.
So we can simulate some unauthorized case, (but that is r1.2.15) by initiating a new conversation 
with our test service, that sets RefToMsgId (should not), and see an error. 
(yet, nothing is said about Error generated.
Is there an implicit assumption throughout the spec that failed reqs generate errors?)


[MIKE] - The spec is unclear as to what type of Error message is generated if the 
condition is not met ( because the previous message never existed, for example ).  
I do not find any "implied" assumption that all failed requests generate an error.  
This should be brought to the attention of the MS spec team as an item to be clarified. 
 For now, I would mark this test coverage as "none", since we cannot write a test for 
this requirement.

[Jacques]: I propose that, when the spec states MUST or SHALL, and does not specify 
what happens when that is not satisfied, that we interpret this as a RECOMMENDATION
for an Error generation. 
This should be the case for r1.2.9.


[MIKE2] - If we are going to treat such requirements as valid, we MSUT provide some kind
of additional metadata tagging this requirement as not specified.  We are already accumulating
a list of spec ambiguities that we must keep track of.  We can tag the <Assertion> or <Condition>
as RECOMMENDED, as long as we tag the requirement as "interpreted" and not specified.


r1.2.18:  (received M)
The error msg should be returned to the "From" party.
Other test items should be derived from (3.1.6.4) , like TimeToLive if present should have 
valid UTC value (and not local time...)

[MIKE] - Correction made.. and an additional derived requirement for UTC created

r1.2.19:  (received M)
Even though this req is in "MS Core features", it really relates to Reliable Messaging
(MS additional features). We should handle this in Level 2.


[MIKE] - This requirement, along with r1.2.20 were both moved to level 2 requirements

r1.2.20:  (generated M)
(An example of the way r1.2.7 should be split into sub-items.)
Yet another "double" test, if in case we send a "bad" message: should there be an error?

[MIKE] - Moved to level 2 ( reliable messaging ), also "doubled" the requirement


r1.2.24:  (received M)
Yet another double test: we should also add the (generated M) version of this test req item:
For any generated messages from the MSH: "either the xlink:href does NOT contain a cid URI, or 
if it does, a MIME part with content-id must be present in payload container."


[MIKE] -  Done

r1.4.5:  (generated M)
I believe we can test that if no error are reported in a message sent by MSH, then 
the ErrorList elt must not be present.


[MIKE] - Done

r1.4.6:    (generated M)
Only partial check, as it may be hard to observe this from candidate MSH: we would need to 
make it generate such a message with several error elements.


[MIKE] - Done

r1.4.9:  (generated M)
reword as: the Error code must be valid. (we can't check more...)


[MIKE] - This is done in requirement r1.4.13... so this requirement will be removed.

r1.4.12:  (generated M)
I believe we can at least check the XPointer well-formedness, and in case of payload-related
error, this is not app-specific (all these errors are MSH level): 
we can send a message with a bad payload *container*.
so that we can check if the error generated is OK.


[MIKE] - Done.

r1.4.18:  (generated M)
I believe its very hard to test that, as it is the case where an error message is bundled
with a business message response. Some MSH may simply not support this feature.
We may not have any control on that from our test service. So I would say no coverage.


[MIKE] - Done.

r1.5.1:  (generated M)
We can test if we got the business response (generated by our test service)
as an HTTP response to the previous sending... so I would say coverage is OK 
(req level: RECOMMENDED, as SHOULD is the same).

[MIKE] - Done.

Note: spec 4.3.1 also report yet another case of CPA discrepancy that is not allowed:
SyncReplyMode. Should be in a separate test req item.

[MIKE] - Added this requirement, actually "doubled" it for generated and received