Re: [ebxml-iic] Conformance Test Case Coverage - Issues to be Resolved

["Michael Kass" <michael.kass@nist.gov>]

Pete,

  I provided my ideas for the items in question inlined below.  Attached is
the list of test
requirements in question.  I'd like to go through the ones we've discussed
as quickly as we can
on the conference call, so that we'll have  a final disposition on them, and
I can go forward with
updating the Conformance Test Suite Specification.

Thanks for your help!

Mike

funreq_id_9:
  Future work: extend test interface, giving it access to full MIME
  message, including headers.

[MIKE] - The testing interface currently supports only access to Content-ID,
Content-Type and charset MIME headers.  I agree that the next version of the
test interface should support examination of any MIME headers


funreq_id_27:
  Add to assertion:  Any @id values that are present must be unique.
  (Can such an assertion be expressed in XPath/XSLT?)

[MIKE] - I am sure that you can express it in XPath, although it might not
be
pretty or efficient

  The full list of extension elements is:
    Acknowledgment, AckRequested, Error, ErrorList, Manifest,
    MessageHeader, MessageOrder, StatusRequest, StatusResponse, SyncReply.
    (Pong is not an element, it is a Service.)  Obviously not all of
    these elements can be exercised in a single test case, so do we
    revisit the assertion in future test cases to get full coverage?

[MIKE] - Sorry about the "Pong" error.  Lost my focus there :)
The reason I created a "sublist" of these [MessageHeader, Acknowledgment,
ErrorList, Error and
StatusRespone] is because [AckRequested, Manifest, MessageOrder,
StatusRequest and SyncReply] are, in my opinion
"application generated" elements, and would not test an MSH conformance, but
instead
ebXML application conformance.   So I would suggest testing only the 4
above, and in
particular only ErrorList or Errors generated at the MSH (not application)
level.
 Comments?



funreq_id_29:
  Presence and correct namespace qualification of this attribute is
  asserted by the echema and tested by virtue of schema validation.
  Required for these elements:
    Acknowledgment, AckRequested, ErrorList, MessageHeader, MessageOrder,
    SyncReply.
  (Side note, perhaps for ebMS TC: value of "1" is required in each case;
  why is it not a fixed value in the schema?)

[MIKE] - Schema validation could be used as a "blanket" check of the
document,
but would only verify the positive test result, that the correct namespace
qualification
was used for the "mustUnderstand" attribute.  A negative result ( validation
failure )
would not necessarily be due to an imporoper namespace, but could be due to
any
other validation problem with the document.  That is why individual testing
of namespace value is necessary to individually test this.  Currently, the
Test
Framework does not support schema validation, and traceability back to
individual
test requirements.


funreq_id_31:
  Not only is this application-level testing, but whether or not
  uniqueness of the type value is required depends on the nature of the
  party identification system chosen.

[MIKE] - Agreed


funreq_id_47:
  Remove "or Manifest" from Name and Assertion (it is not possible to
  place application data inside Manifest anyway).
  Manifest, StatusRequest, StatusResponse are the allowed SOAP Body
  extension elements.

[MIKE] - Done.


funreq_id_48: (not on this list, but I noticed it when looking into the
  next one).
  The applies only if href URI scheme is "cid".

[MIKE] - I fixed this in the Conformance Test Suite spec.


funreq_id_50:
  Not application-level.  Test by verifying that each Manifest/Reference
  CID matches the Content-ID: of an attached MIME part.

[MIKE] - So the application does not control CID name or the paylod?
I thought this was in control of the particular application.... I can test
this
against our own "Reflector" action, but what does this mean as far as
MSH conformance testing, since the MSH isn't really aware of message
appplication content... Comments?


funreq_id_56:
  Will test driver report an error if unexpected ErrorList element is
  present in any received message?

[MIKE] - No, although we could incorporate some "warnings".  If the
resulting
returned message satisfies our test requirement, then additional errors or
warnings
"could" be flagged in the test report, but we do not fail a candidate MSH
for
unexpected errors unless those errors are specifically determined to be
criteria for failing the test.  I would suggest "extending" the Test
Framework
to incorporate warning messages for all test cases when unexpected errors
are generated.


funreq_id_61:
  Value of XPointer is predictable; assert that its value matches the
  path of the generated "bad" element of the original, referenced message.
  Should precondition be "for each received message"?

[MIKE] - My reason for not making this test "complete" coverage is that I do
not
see in the specification exactly what the standard XPointer syntax would be
to point to an element within the SOAP Header or Body of a message.  Without
knowing that, I don't know how to specify the XPointer. Is that specified
somewhere?
For example, what would be the XPointer to an ErrorList element in the SOAP
Header?


funreq_id_62:
  If test driver is capable of generating a "bad" MIME part, the cid
  value is predictable; assert that its value matches the Content-ID of
  the generated MIME part in error in the original, referenced message.
  Should precondition be "for each received message"? [MIKE] - Yes, FIXED
THAT

[MIKE] - You ar right.  The CID of a "BAD" MIME part, generated by the Test
Driver
is predicatlabe.  However, none of the Test Service Actions are defined to
highlight a "bad" MIME part, which I assume means an application payload .
And even if they did, wouldn't that put this test in the realm of
"application-level
testing"?  What would be the meaning of writing an application that returns
the
correct CID of an erroneous message, since the MSH has no real part in this?
Again, I would view this as "application level testing".. not MSH testing.
Comments?


funreq_id_63:
  Raise this issue with ebMS TC?  I think there's no problem here, just
  that the spec contains a rather odd statement, since Error element has
  no content anyway (it is a complex type).  The spec does say Description
  content is vendor-defined.  So we just don't know why the prohibition
  against using "Short Description" is there.

[MIKE] - I read it as "use the Long Description" but the spec never
precisely says that.


funreq_id_77-84:
  I suspect that many customers will demand assurance of data reliability
  in the face of certain types of system failure.  Glad to see that
  the abstract test cases will be described, so they can at least be
  performed manually if desired.

[MIKE] - Agreed


funreq_id_85:
  Is message persistence proven implicitly by reqs 77-84 above?

[MIKE] - My vote would be "yes", since the received or response message
HAS to exist somewhere during the period of an interruption...


funreq_id_89:
  Can abstract test case be described?  Something like, send request
  to MSH, ignore response, interrupt/restart MSH, expect MSH to resend
  response.


[MIKE] - I think that we can do this, and that is the kind of wording that
I would use



funreq_id_115:
  In addition, this is app-level testing.  (We can't tell whether a
  duplicate is ever delivered to the application.)  Does this mean that
  we could/should provide abstract test cases for all other reqs we
  determine are app-level?

[MIKE] - Well, if we are going to do i,  then we would
have to say something like:  "Candidate MSH notifies Test Driver that a
duplicate message is presented to the application after system
interruption".
I think that we should be consistent if we are going to do this.  If we are
going to abstractly describe system interrupts, then we should describe
any other test steps that fall outside of the capability of the Test
Framework.



funreq_id_146:
  Issue for ebMS TC: Error for invalid presence of SyncReply is covered
  in section 9.2 and tested by case funreq_id_157, but what is to be
  done if DuplicateElimination is absent?


[MIKE] - Agreed

funreq_id_154:
  Not application-level, but impractical to test (we would have to
  instruct MSH to send 100,000,001 messages to observe the wraparound
  condition).

[MIKE] - Agreed

funreq_id_177, 178, 180, 181:
  Future work: Define a way for framework to query transport protocol
  security parameters that were used.

MIKE] - Agreed


funreq_id_197:
  Not application-level, but may be framework limitation.  Future:
  extend framework to provide a way of instructing MSH to generate a Ping.

[MIKE] - I see this as application-level, since we would have to (at the
application
level, generate a message with a Ping <Action> element ) and make sure it
has no
application payload... this sounds like application testing to me, not MSH
testing...
Comments?

ebMSConfTestCoverage.doc