ebxml-iic-msg message

Subject: Re: [Fwd: FW: [ebxml-iic-msg] MS Conformance checklist: comments]

From: Michael Kass <michael.kass@nist.gov>
To: ebxml-iic-msg@lists.oasis-open.org
Date: Wed, 03 Apr 2002 13:19:04 -0500

Jacques and Matt,

-----Original Message-----
From: Matthew MacKenzie [mailto:matt@xmlglobal.com]
Sent: Wednesday, March 27, 2002 11:12 PM
To: Jacques Durand
Cc: ebxml-iic-msg@lists.oasis-open.org
Subject: Re: [ebxml-iic-msg] MS Conformance checklist: comments

On Wednesday, March 27, 2002, at 07:18 PM, Jacques Durand wrote:

> Matt / Michael:
>
> Quite good material here from XMLG - I assume you guys will use this to
> complete
> the current (XML) "test reqs" document?

That is the plan. We have one person devoted to doing this stuff,
including some test case definition for these conformance issues, and I
hope to donate this stuff where possible to kickstart our work here.

> Some comments:
>
> 1. About some test items:
> - the DuplicateElimination tests (section 3.1.7, testing actual
> duplicates), really
> belong to "Additional features" (Reliability module), however the
> "wellformedness" of
> the header element (if present) can still be considered as core Level
> (schema compliance)...
> - a few items in this list are CPA-dependent. We may want to put them
> aside for now until we
> introduce CPA-content modeling in our tests(3.1.2, 3.1.5, 4.1.2,
> 4.2.4.2...).
>

[Mike]
A schema that requires a person creating requirements to provide a level number with the
requirement could provide some integrity checking to the requirement, and
help eliminate invalid tests based on level of a <Condition> or <Assertion>

Please see:

http://xw2k.sdct.itl.nist.gov/mkass/oasis/ebxml/schema/ebXMLRequirements.html

This gives a general description of the Test Requirements schema that I am proposing to use as
a base for the testing framework.

> 2. As we have noticed before, many test items are in the form of
> IF-THEN rules.
> Either the "IF" is controlled :
> (a)- by the MSH :
> (i.e. is an implementation choice, e.g. 2.2.1 &2.2.2 "XML prolog in
> SOAP")
> In that case, this test item may never apply, as the MSH will never
> generate messages
> that satisfy the "IF".
> (b)- or the "IF" is controlled by the application (e.g.
> or 3.1.8 "if the Description elt is present...", 2.1.4 "if there is no
> payload...")
> In that case, a well-rounded test suite is expected to produce messages
> that will
> satisfy the "IF" of these test items.

[Mike]

I agree, that tests should be written to be deterministic. However, there will be cases where
a feature is not required according to the specification ( like the "Content-type" MIME value ).. but if present...
it must match the value specified in the SOAP header. Such a test must not "fail" if the Content-type
attribute is not present. We can support this in the schema by providing a "MUST" or "OPTIONAL" ..etc..
attribute in the <Assertion> or <Condition>. See:

http://xw2k.sdct.itl.nist.gov/mkass/oasis/ebxml/schema/ebXMLRequirements.xml ( you should see a stylized HTML page.. not XML if you are running the latest MSXML in you Internet Explorere Browser ). If running Netscape or other... look at:
http://xw2k.sdct.itl.nist.gov/mkass/oasis/ebxml/schema/requirements.html

> (c)- or the "IF" is controlled by the CPA (e.g. all test items
> conditional to the presence
> of Security elements)
> In order to test (b), relevant messages need be generated. That means
> we need also to define
> a sampling of "test messages" that we want to see sent by the tested
> MSH.
> We need then to define a format (XML) for message data to be fed to the
> MSH.
> To be more precise, to be fed to the Test Driver that controls the
> tested MSH.
> These test messages should be expressed in an "envelope-neutral" XML
> format (i.e. NOT ebXML!)
>

[Mike]
The CPA <Condition> could be the first one specified in a test requirement, to allow for
gracefully exiting the test requires a particular CPA feature to be present.

> 3. The XML format for message sampling used by test driver in (2)
> above, could adopt
> a JMS style: Part 1 will specify header elements as a list of
> "property / value" pairs,
> Part 2 will specify the payload(s). Roughly, that could be of the form:
>
> <messagedata name="testABC">
> <header_properties>
> <fromPartyID>123</fromPartyID>
> <toPartyID>456</toPartyID>
> <Service>PurchaseOrdering</Service>
> <Action>Accept</Action>
> ...
> </header_properties>
> <payloads>
> <payload> .... an XML payload, or a ref to a binary payload...</payload>
> </payloads>
> </messagedata>
>
> The test driver would have to "translate" this in API calls to the MSH
> (using
> a small adapter specific to the candidate MSH's API).
>

[Mike]
I took a stab at this using a "homegrown" syntax. See:

http://xw2k.sdct.itl.nist.gov/mkass/oasis/ebxml/schema/ebXMLTestSuite.html

I think that XPath would be an excellent candidate for succinctly describing what
must be present in a returned message ( also in a sent message, but XPath is
a "query" language... so I do not know how acceptable such a concept is )

Why not just have a library of ebXML Message templates that we want to
send, and use a simple preprocessor and a tool like netcat to send the
messages to the MSH? I don't think we care to test a candidate MSH's
client side API, just the behaviour of the MSH itself.

[Mike] Would the same thinking apply to the "server side"? I'm just trying to
understand the role of the testing API in ebXML MSH client/server roles. Will
it always be simply a "pass thru" for messages to the MSH only?

We were thinking of defining a journal or log format that candidate
msh's would write during conformance testing. We would define the
format of the log, and all of the points where a log must be written.
There would also be a Conformance service that the MSH would implement
allowing the conformance test suite to "download" the journal after a
test run. e.g.

<journal start="2002-8-6T12:43:45">
        <entry time="2002-8-6T12:43:47" type="msg_recv">
        <![CDATA[ message here ]]>
        </entry>

        <entry time="2002-8-6T12:44:00" type="ack_sent">
        <![CDATA[ ack here ]]>
        </entry>
</journal>

> Defining such samples of messages will also help down the road to check
> the
> "service-conformance", that is, the MSH really sends the message
> content that
> the application (here the test driver) expects it to send out.
>
> 4. We may have a similar approach for defining the CPA under which a
> test is to be
> performed (i.e. we define an XML format that encodes the subset of CPA
> needed to
> "configure" the MSH).

A simple property file will likely suffice.

>
> 5. As our "test requirements" definition document will also somehow
> "bind" the
> test items, with the test procedures that check them, it may be good to
> distinguish
> the test procedures these test items require. So far I see the test
> items you
> described fall into two test procedures:
> (P1)- "drive-and-receive" procedures, where the test driver is fed some
> message data,
> and controls the candidate MSH so that it generates a messages that we
> capture on
> the HTTP destination. Most of the test items you described are tested
> that way.
> A Test case definition for such procedures is made of:
> (1) test item requirement def (and associated procedure),
> (2) message data to be fed to driver.
> (P2)- "send-and-receive" procedures, where we send a message to the
> candidate MSH,
> and observe its response-message. This usually involves the test driver
> on candidate
> MSH side, to provide response data (e.g. 3.1.6.3a), but sometime the
> response is
> generated directly by the candidate MSH (in case of errors, e.g.
> 3.1.6.3bc, 3.1.6.4)
> A Test case definition for such procedures is made of:
> (1) test item requirement def (and associated procedure),
> (2) message data to be sent to candidate MSH (described using same XML
> neutral format).
> (3) message data to be fed to driver for response to test driver of
> candidate, if applicable.
>
> We probably should start thinking of how to more completely describe
> such test cases.
> I can give a first cut at a "message data" XML format, and give it for
> review.

The message data format will certainly be useful, and could be
integrated with the requirements definition, however, you must remember
that the requirements document simply states the requirements, not
necessarily give you the details of how to test compliance to the
requirements. Nonetheless, being able to pass this kind of data onto
the test framework is probably very useful. I'll let Michael weigh in
here, as he is definitely the expert on this sort of thing.

[Mike]
It is always nice to keep the requirements separate from the actual
test implementation. However, XML is a very powerful tool for integrating
both, and I believe that we could do that without blurring the line between
requirements and procedures. Including the procedural XML into the
testing framework could be done through the requirements document, as
Matt and I discussed earlier ( including a file reference as an attribute to the
<TestRequirement> element.

--
Matthew MacKenzie
XML Global R&D
PGP Key available upon request.

----------------------------------------------------------------
To subscribe or unsubscribe from this elist use the subscription
manager: <http://lists.oasis-open.org/ob/adm.pl>