RE: ebXMLTestFramework

["Michael Kass" <michael.kass@nist.gov>]

Laura,

   Thank you for your comments.  They are being added to the errata,  and
will be posted online soon, along with changes to the specification sections
you mention..  I will notify you when the are available on the OASIS site.
We can add you as a prospective IIC member ( giving you automatic email
messages from our IIC discussion team ) if you wish.  Right now, I am CC'ing
other IIC members with our discussion.

   You can monitor the IIC mailing list right now at:
http://lists.oasis-open.org/archives/ebxml-iic/

    If you wish to become a member of the IIC, I would suggest you contact
Jacques Durand at: JDurand@us.fujitsu.com  He is the Technical Committee
chair and can advise you how to
get on the mailing list and OASIS membership details.

   Below are my comments ..preceded by   [MIKE]



-----Original Message-----
From: Laura Bianchini [mailto:bianchin@cs.unibo.it]
Sent: Monday, March 15, 2004 9:59 AM
To: michael.kass@nist.gov
Subject: ebXMLTestFramework


Dear Michael Kass,

at this moment, my attention is focused on the Conformance Test
Requirement - ErrorHandling (req_id_4) and related Test Cases.
I know req_id_4 is very particular, because it isn't simple to implement
and test a module for error reporting and handling.
I would like to make you some questions on the analyzed requirements and
to report the discrepancies between:

- IIC_MS_20_Conformance_TestSuite_V1.0.doc [Test]
- ebxml_ms_20_executable_testsuite.xml [ExeT]

1) In the [ExeT] the TestCase_53 is absent.

[MIKE] - We opted to remove <TestCase> 53 from the Executable Test Suite
because it was really SOAP conformance testing, and beyond the scope of
ebXML MSH testing.  I will remove it from [Test]


2) In the [ExeT] TestCase_54 there are 1 precondition and 1 assertion but
the assertion is wrong: the requirement impose to verify that "no SOAP
fault is generated for warning" and the assertion is wrong because it's a
precondition.

[MIKE] - Agreed. I re-wrote it in a way that may permit testing this.  This
will be published in the latest update to the Executable Test Suite and the
Abstract Conformance Test Suite.

3) In the [ExeT] TestCase_57, in PutMessage the action element is not
defined
(also in other TestCases, when the MessageHeader is created
with unresolveable To/PartyId, the action element is absent. Is it
an error?)
[MIKE] - No, this is not an error. If an Action is not defined in the
Message Declaration, then the Test Driver MUST insert the "default" Action
name, as defined by the default ConfigurationGroup in the Test Suite. In
this case, that would be the "Dummy" action.   An "unresolvable" PartyId of
"null" was explicitly added for the purpose of causing the candidate MSH to
generate an error message when it is not able to resolve that ID.


The assertion is different in the 2 version of the testcase.



4) In TestCase_58, the purpose is to test if codeContext
attribute is an URI, but in [Test] the abstract message content seems to
only verify the existence of the codeContext attribute;
[MIKE] - I'm not sure what you mean here.  In [Test], we specifically
indicate a "Validate_Received_Message_Filter" in our abstract test case for
the Test Assertion.  This agrees with the <ValidateContent> operation in the
<TestAssertion> in [ExeT].   We are not just verifying exstence, but are
providing an XPath to the content ( a URI ) that we wish to examine to
determine if it is in fact a valid URI.

 in [ExeT] the
operation is right while the Message Expression is wrong because is the
same of the previous assertion.

[MIKE]- The message expression is just an XPath to be used by the Test
Driver to retrieve the content at that node in the tree, not to do an
evaluation.  That must be done internally by the Test Driver.


5) In the [ExeT] TestCase_61, tha Assertion is wrong bercause the test
has to check if the location attribute is an XPointer to the erroneus
element, while here the Assertion verify that Error codeContext
attribute is returned.
In [Test], the Assertion verify that Error location attribute is
returned,but not if it is an XPath.

[MIKE] - You are right.  This test does not validate that there is a correct
XPointer in the "codeContext" attribute of the Error element.   I think
that, at best we may be able to validate that the content at this location
is in fact  an XPointer.  I think that it may be difficult to verify that a
returned XPointer is semantically correct, because the ebXML MS 2.0
specification is very vague in defining what an XPointer to any location in
the message should look like.  I have modified the test so that it does a
<ValidateContent> on the codeContext with a type of "XPointer"... meaning a
Test Driver must be able to parse an XPointer and determine that it is
syntactically correct ( not necessarily semantically ).


6) In TestCase_64, the precondition says "when a MSH detects an error in
a message and - the Error Reporting Location to which the message
reporting the error should be sent can be determined and the message in
error doesn't have an ErrorList element with highestSeverity set to
Error" then the error is reported to MSH that sent the original message
in error. Like the other tests, this test case is only partial covered but
the assertion only verifies the presence of an error. I think that this
check is too weak,
comparing to specification (ebXMLTestFramework V1.0 #4.2.4.1)?
(I think it's the same for TestCase_66 and TestCase_67).
[MIKE] - Actually, after re-reading the specification, I think that this
test can only be written in a generalized way.  The additional constraint
that the "offending" message MUST NOT contain an ErrorList with a
"highestSeverity" of "Error" is important, and the originating message in
the <TestCase> does not contain an Errorlist, so that is acknowledged.  The
"PreCondition" is that the candidate MSH must be able to recognize a
particular error and generate an ErrorList.  This is not a particularly
strong test,  because  the presence of ANY ErrorList would signify success,
since it would indicate that the candidate MSH is sending its Errors back to
the originating MSH ( as it should ).   A more rigorous test in which many
types of Error scenarios are introduced would give a better indication if
the MSH was conformant or not.  I have introduced a few more "errors" into
the originating message to try to strengthen the test and generate a
returned ErrorList. This includes adding an unknown "Action" and an unknown
"ConversationId".
Test Case 66 is virtually identical to Test Case 64, except that the Error
Reporting Location should be derivable from the CPA.. so this scenario is
given a valid CPAId , and an an invalid From/PartyId ( along with an invalid
Action and ConversationId just to generate an error ).
Test Case 67 is different in that the CPA does not specify the Error
Reporting Location.   In this case, the candidate MSH must derive it from
the incoming message From/PartyId, so a valid From/PartyId is provided in
the originating message, and the CPAId points to a "special" CPA that does
not specify the error reporting location.  Additional errors are added (bad
ConversationId, bad Action value).


7) In the [Test] TestCase_68, Test Step 2, when Test Driver filters a
message received from MSH, the GetMessage operation checks the
eb:Service, but I think this is wrong because eb:Service must be verified
in the assertion. I agree with [ExeT] where the check is " eb:Action!=Mute
".

[MIKE] - I modified these tests in [ExeT} to filter on the expected Service
and Action names, then simply count that at least 1 message matches that
filter ( i.e. is an "independent"  ErrorList message).  I'll post this
updated test suite to the IIC site early next week.  I'll also modify [Test]
accordingly.


I hope that you will find the descriptions above accurate enough to give
you the opportunity to make right checks without losing time.

[MIKE] - Your input is very valuable.   We are also getting input from Drake
Certivo now also, to futher strengthen the test suite.  You will see their
comments as soon as I review them.  I will continue to forward your comments
and my responses back to the IIC, and encourage you to look into joining the
IIC ( talk to Jacques Durand ).


Thank you for your attention. I would appreciate every kind of comments
on discovered problems, so that I can continue my work.

Laura Bianchini
Undegraduated student of computer science (Bologna-Italy)