Re: [ebxml-iic-msg] MS Conformance checklist: comments

[Matthew MacKenzie <matt@xmlglobal.com>]

On Wednesday, March 27, 2002, at 07:18  PM, Jacques Durand wrote:

> Matt / Michael:
>
> Quite good material here from XMLG - I assume you guys will use this to 
> complete
> the current (XML) "test reqs" document?

That is the plan.  We have one person devoted to doing this stuff, 
including some test case definition for these conformance issues, and I 
hope to donate this stuff where possible to kickstart our work here.


> Some comments:
>
> 1. About some test items:
> - the DuplicateElimination tests (section 3.1.7, testing actual 
> duplicates), really
> belong to "Additional features" (Reliability module), however the 
> "wellformedness" of
> the header element (if present) can still be considered as core Level 
> (schema compliance)...
> - a few items in this list are CPA-dependent. We may want to put them 
> aside for now until we
> introduce CPA-content modeling in our tests(3.1.2, 3.1.5, 4.1.2, 
> 4.2.4.2...).
>
> 2. As we have noticed before, many test items are in the form of 
> IF-THEN rules.
> Either the "IF" is controlled :
> (a)- by the MSH :
> (i.e. is an implementation choice, e.g. 2.2.1 &2.2.2 "XML prolog in 
> SOAP")
> In that case, this test item may never apply, as the MSH will never 
> generate messages
> that satisfy the "IF".
> (b)- or the "IF" is controlled by the application (e.g.
> or 3.1.8 "if the Description elt is present...", 2.1.4 "if there is no 
> payload...")
> In that case, a well-rounded test suite is expected to produce messages 
> that will
> satisfy the "IF" of these test items.
> (c)- or the "IF" is controlled by the CPA (e.g. all test items 
> conditional to the presence
> of Security elements)
> In order to test (b), relevant messages need be generated. That means 
> we need also to define
> a sampling of "test messages" that we want to see sent by the tested 
> MSH.
> We need then to define a format (XML) for message data to be fed to the 
> MSH.
> To be more precise, to be fed to the Test Driver that controls the 
> tested MSH.
> These test messages should be expressed in an "envelope-neutral" XML 
> format (i.e. NOT ebXML!)
>
> 3. The XML format for message sampling used by test driver in (2) 
> above, could adopt
> a JMS style: Part 1 will specify header elements as a list of 
> "property / value" pairs,
> Part 2 will specify the payload(s). Roughly, that could be of the form:
>
> <messagedata name="testABC">
> <header_properties>
> <fromPartyID>123</fromPartyID>
> <toPartyID>456</toPartyID>
> <Service>PurchaseOrdering</Service>
> <Action>Accept</Action>
> ...
> </header_properties>
> <payloads>
> <payload> .... an XML payload, or a ref to a binary payload...</payload>
> </payloads>
> </messagedata>
>
> The test driver would have to "translate" this in API calls to the MSH 
> (using
> a small adapter specific to the candidate MSH's API).
>

Why not just have a library of ebXML Message templates that we want to 
send, and use a simple preprocessor and a tool like netcat to send the 
messages to the MSH?  I don't think we care to test a candidate MSH's 
client side API, just the behaviour of the MSH itself.

We were thinking of defining a journal or log format that candidate 
msh's would write during conformance testing.  We would define the 
format of the log, and all of the points where a log must be written.  
There would also be a Conformance service that the MSH would implement 
allowing the conformance test suite to "download" the journal after a 
test run. e.g.

<journal start="2002-8-6T12:43:45">
	<entry time="2002-8-6T12:43:47" type="msg_recv">
	<![CDATA[ message here ]]>
	</entry>

	<entry time="2002-8-6T12:44:00" type="ack_sent">
	<![CDATA[ ack here ]]>
	</entry>
</journal>




> Defining such samples of messages will also help down the road to check 
> the
> "service-conformance", that is, the MSH really sends the message 
> content that
>  the application (here the test driver) expects it to send out.
>
> 4. We may have a similar approach for defining the CPA under which a 
> test is to be
> performed (i.e. we define an XML format that encodes the subset of CPA 
> needed to
> "configure" the MSH).

A simple property file will likely suffice.


>
> 5. As our "test requirements" definition document will also somehow 
> "bind" the
> test items, with the test procedures that check them, it may be good to 
> distinguish
> the test procedures these test items require. So far I see the test 
> items you
> described fall into two test procedures:
> (P1)- "drive-and-receive" procedures, where the test driver is fed some 
> message data,
> and controls the candidate MSH so that it generates a messages that we 
> capture on
> the HTTP destination. Most of the test items you described are tested 
> that way.
> A Test case definition for such procedures is made of:
> (1) test item requirement def (and associated procedure),
> (2) message data to be fed to driver.
> (P2)- "send-and-receive" procedures, where we send a message to the 
> candidate MSH,
> and observe its response-message. This usually involves the test driver 
> on candidate
> MSH side, to provide response data (e.g. 3.1.6.3a), but sometime the 
> response is
> generated directly by the candidate MSH (in case of errors, e.g. 
> 3.1.6.3bc, 3.1.6.4)
>  A Test case definition for such procedures is made of:
> (1) test item requirement def (and associated procedure),
> (2) message data to be sent to candidate MSH (described using same XML 
> neutral format).
> (3) message data to be fed to driver for response to test driver of 
> candidate, if applicable.
>  
> We probably should start thinking of how to more completely describe 
> such test cases.
> I can give a first cut at a  "message data" XML format, and give it for 
> review.


The message data format will certainly be useful, and could be 
integrated with the requirements definition, however, you must remember 
that the requirements document simply states the requirements, not 
necessarily give you the details of how to test compliance to the 
requirements.  Nonetheless, being able to pass this kind of data onto 
the test framework is probably very useful.  I'll let Michael weigh in 
here, as he is definitely the expert on this sort of thing.
--
Matthew MacKenzie
XML Global R&D
PGP Key available upon request.