[ebxml-iic-msg] RE: other Schema docs - usage?

[Jacques Durand <JDurand@fs.fujitsu.com>]

Title: Welcome

Michael, Matt:

 

I believe the "Test requirements" document and schema are on good tracks.

Let me try to summarize where we are, and raise some remaining issues:

 

1- The Requirements schema/doc, so far captures Level 1 ("core", or most of it).

When the corresponding tests are run (controlled by ANT + JXUnit), one can check that

an ebXML message is well-formed and also semantically sound, according to Level 1.

That actually captures "wire-conformance" Level 1.

 

2- We expect to drive the candidate MSH so that it generates a diverse-enough sampling of messages,

each of which will undergo the Level 1 "wire-conformance" tests above.

 

3- Issue: as we will drive the candidate MSH to generate messages to be tested, it is not enough

to test each of these through the "Level 1 tests" above. Indeed, we want to make sure in addition that

the set of generated messages (test set), uses all features expected in Level 1.

Example: if for some reason the candidate MSH fails to generate some optional message elements - e.g.

digital signature, etc. - then its message format may still be correct, as tested on the wire, but

we cannot claim Level 1 passes, as the test coverage would not have checked all expected message features.

Solutions: we need to have a way to "require" and to check, in the generated message test set,

that all expected message features have ben produced at least once. That could be done by (a) either

keeping a checklist of all message features that are concerned by Level 1, and match it with what has

actually been teted, (b) or by making sure that the test associated with each Level 1 Requirement has been

used at least once (<condition> part successful?), (c) or by doing a more through comparison between

what the test driver has been fed (input test cases) and what is observed on the wire. note

that would have  some "service-conformance" aspect.

 

 

Just following up on Michael schemas (ebXMLCompare.xsd, ebXMLSet.xsd, ebXMLTestSuite.xsd):

 

4- Michael: could you give a quick overview / use case of how you see instances of these schemas being used?

(I assume that instances of ebXMLTestSuite.xsd will be input to a "comparison engine"?)

 

5- I assume these schemas will then be used to compare actual output / expected output, right?

(that could then help on issue #3-c above.)

 

6- Issue: We'll need to define how to compare schema instances, e.g. in some cases, some distance is allowed

with the reference document (e.g. optional element, order, different values...)

 

7- COnformance reports - should it be instance of an extended Requirements schema with test results?

 

-Regards,

 

jacques

-----Original Message-----
From: Jacques Durand [mailto:JDurand@fs.fujitsu.com]
Sent: Monday, March 18, 2002 3:42 PM
To: 'ebxml-iic-msg@lists.oasis-open.org'
Subject: [ebxml-iic-msg] Welcome

Hi:

Welcome to the ebxml-iic-msg sublist.

This is where you will read or post mails about MS conformance testing design,
and more generally about the "Conformance Testing Task Force" activities and material.

Regards,

Jacques Durand
ebXML IIC chair