Re: [ebxml-iic] Conformance Test Case Coverage - Issues to be Resolved

[Pete Wenzel <pete@seebeyond.com>]

Here are my comments on the remaining issues.  In some cases, I've posed 
deeper questions that may need to be discussed, and things that need to 
be brought to the attention of the Messaging TC.

--Pete
Pete Wenzel <pete@seebeyond.com>
SeeBeyond
Standards & Product Strategy
+1-626-471-6311 (US-Pacific)

Michael Kass wrote:

> To all,
>  
>    Attached is the Test Case Coverage document for those conformance 
> test requirements where
> the specification, the nature of the test ( application vs. MSH testing 
> ) or the Test Framework itself
> is an issue regarding testability.
>   I integrated comments from the last conference call, but I believe 
> that we still need a further iteration
> on these, as we did not get to cover all of the requirements at issue.
>  
>   Those covered in the last call include:
>  
> 86
> 86
> 87
> 88
> 89
> 148
> 155
> 156
> 161
> 175
> 159
> 193
> 194



funreq_id_9:
  Future work: extend test interface, giving it access to full MIME
  message, including headers.

funreq_id_27:
  Add to assertion:  Any @id values that are present must be unique.
  (Can such an assertion be expressed in XPath/XSLT?)
  The full list of extension elements is:
    Acknowledgment, AckRequested, Error, ErrorList, Manifest,
    MessageHeader, MessageOrder, StatusRequest, StatusResponse, SyncReply.
    (Pong is not an element, it is a Service.)  Obviously not all of
    these elements can be exercised in a single test case, so do we
    revisit the assertion in future test cases to get full coverage?

funreq_id_29:
  Presence and correct namespace qualification of this attribute is
  asserted by the echema and tested by virtue of schema validation.
  Required for these elements:
    Acknowledgment, AckRequested, ErrorList, MessageHeader, MessageOrder,
    SyncReply.
  (Side note, perhaps for ebMS TC: value of "1" is required in each case;
  why is it not a fixed value in the schema?)

funreq_id_31:
  Not only is this application-level testing, but whether or not
  uniqueness of the type value is required depends on the nature of the
  party identification system chosen.

funreq_id_47:
  Remove "or Manifest" from Name and Assertion (it is not possible to
  place application data inside Manifest anyway).
  Manifest, StatusRequest, StatusResponse are the allowed SOAP Body
  extension elements.

funreq_id_48: (not on this list, but I noticed it when looking into the
  next one).
  The applies only if href URI scheme is "cid".

funreq_id_50:
  Not application-level.  Test by verifying that each Manifest/Reference
  CID matches the Content-ID: of an attached MIME part.

funreq_id_56:
  Will test driver report an error if unexpected ErrorList element is
  present in any received message?

funreq_id_61:
  Value of XPointer is predictable; assert that its value matches the
  path of the generated "bad" element of the original, referenced message.
  Should precondition be "for each received message"?

funreq_id_62:
  If test driver is capable of generating a "bad" MIME part, the cid
  value is predictable; assert that its value matches the Content-ID of
  the generated MIME part in error in the original, referenced message.
  Should precondition be "for each received message"?

funreq_id_63:
  Raise this issue with ebMS TC?  I think there's no problem here, just
  that the spec contains a rather odd statement, since Error element has
  no content anyway (it is a complex type).  The spec does say Description
  content is vendor-defined.  So we just don't know why the prohibition
  against using "Short Description" is there.

funreq_id_77-84:
  I suspect that many customers will demand assurance of data reliability
  in the face of certain types of system failure.  Glad to see that
  the abstract test cases will be described, so they can at least be
  performed manually if desired.

funreq_id_85:
  Is message persistence proven implicitly by reqs 77-84 above?

funreq_id_89:
  Can abstract test case be described?  Something like, send request
  to MSH, ignore response, interrupt/restart MSH, expect MSH to resend
  response.

funreq_id_115:
  In addition, this is app-level testing.  (We can't tell whether a
  duplicate is ever delivered to the application.)  Does this mean that
  we could/should provide abstract test cases for all other reqs we
  determine are app-level?

funreq_id_146:
  Issue for ebMS TC: Error for invalid presence of SyncReply is covered
  in section 9.2 and tested by case funreq_id_157, but what is to be
  done if DuplicateElimination is absent?

funreq_id_154:
  Not application-level, but impractical to test (we would have to
  instruct MSH to send 100,000,001 messages to observe the wraparound
  condition).

funreq_id_177, 178, 180, 181:
  Future work: Define a way for framework to query transport protocol
  security parameters that were used.

funreq_id_197:
  Not application-level, but may be framework limitation.  Future:
  extend framework to provide a way of instructing MSH to generate a Ping.