OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

ebxml-iic message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: RE: [ebxml-iic] test case outcomes: semantics

Title: RE: [ebxml-iic] test case outcomes: semantics

inline <JD>

-----Original Message-----
From: Michael Kass [mailto:michael.kass@nist.gov]
Sent: Tuesday, August 17, 2004 3:06 PM
To: Monica Martin; Jacques Durand
Cc: ebXML IIC - main list (E-mail) (E-mail); dmoberg@cyclonecommerce.com
Subject: Re: [ebxml-iic] test case outcomes: semantics

----- Original Message -----
From: "Monica Martin" <Monica.Martin@Sun.COM>
To: "Jacques Durand" <JDurand@us.fujitsu.com>
Cc: "ebXML IIC - main list (E-mail) (E-mail)"
<ebxml-iic@lists.oasis-open.org>; <dmoberg@cyclonecommerce.com>
Sent: Tuesday, August 17, 2004 3:16 PM
Subject: Re: [ebxml-iic] test case outcomes: semantics

> Jacques,
> Do you think it is clear enough between Undetermined and System Exception
> The former is for a test case, for example, cannot fulfill its purpose
(i.e. material issue) vs. the system exception is specific to a test bed
condition (i.e. the infrastructure has failed to support the test).

[MIKE] - That's the way I see it too.  We need a clear distinction between
failure to make a "pass"/"fail" call based upon testing logic, and simple
"system failure" due to an infrastructure problem.  I think we need
4 final states of :  pass,  fail,  undetermined and exception

<JD> I think Monica was trying to say that if we keep this distinction,
it may not be clearcut in every case. For example, it could be that non-relevant material
could ALSO cause a system exception. For example, a test case is about received messages under
size N, and does not have control on the actual size of messages it receives (so it should
just ignore cases where size > N, where the outcome is "undetermined").
But some messages over N+... may cause tesbed exception.
But that's OK: that is also true for the 2 other "logical" outcomes (pass, fail) these
could be override by a system exception at any time.

> Perhaps my example could be helpful.
> On a secondary note, comments on your Section 7.1 revisions yesterday:
> 1. TC #1: Some confusion may still exist about message and business
acknowledgments. At the business level, we have to deal with syntax (receipt
ack), intellibility (receipt ack QOS) and semantic validation (can be
processed by an application) [acceptance ack].

[MIKE] - We can test for syntax and intelligibility (via schema validation).
Semantic validation would be done via XPath or Schematron evaluation of
content, and based upon such a result, a "canned" response to further test
BSI behavior would take place (since the Test Driver is not a reference
implementation of a BSI, test logic must follow predetermined paths based
upon received message content semantics).

<JD> I was about to do the same comment as Monica: We should leave out the ebXML MS Acknowledgment message for this use case: it complicates the example and may confuse people. However, we can add a note that says, in case an Ack is generated - or any other non relevant message - by the implementation under test, then our test case would simply ignore it, by NOT selecting it in GetMessage filters)

> 2. TC#2: The response is completely separate; there has been discussion
for example in UBL to potentially separate 'response' to an 'acceptance' or
'rejection'. It has been pointed out to me these are not the same
(structurally perhaps, semantically not).   Note: The business process
delegates to and depends (too) on the reliable messaging infrastructure, but
msg ack!=bus ack.

[MIKE] - Understood.. We'll modify this Test Case to reflect the difference.

> On above points, I would err on being explicit about the separation as
much as possible. When we get to ebBP testing, we will have to discuss how
to represent the QoS constraints usage (for example,
isIntelligibleCheckRequired). Dale might have more to add (cc: him herein).

<JD> Just let us know what kind of test condition on the message material should
verify this.

> Jacques, you can get an example of the ebXML case studies at:
> Thanks.
> ----- Original Message -----
> From: Jacques Durand <JDurand@us.fujitsu.com>
> Date: Tuesday, August 17, 2004 12:33 pm
> Subject: [ebxml-iic] test case outcomes: semantics
> > Miekand all:
> >
> > here is a proposed insert on the meaning of test case outcomes:
> > For review
> > (note: there may be a blurred distinction between som cases of
> > "Undetermined" and "system exception")
> >
> > Jacques
> >
> >
> > Pass:
> > -----
> >
> > Meaning:
> > The material under test (e.g. messages from the implementation
> > under test)
> > is satisfying
> > the test requirement associated with the test case.
> >
> > Fail:
> > -----
> >
> > Meaning:
> > The material under test (e.g. messages from the implementation
> > under test)
> > does not satisfy the test requirement associated with the test
> > case.
> >
> > Action:
> > The expected remedial actions include fixing the implementation
> > under test,
> > as this is a failure to conform to the specification as being
> > verified by
> > this test suite.
> >
> >
> > Undetermined:
> > -------------
> >
> > Meaning:
> > The test case execution could not conclude by either a Fail or a
> > Pass,
> > because of the irrelevance of the material under test, or because
> > the right
> > context for this test case could not
> > be created. These reasons could be:
> > (a)- the material under test is simply not relevant to the test
> > requirementbeing tested, and a preliminary test assertion (acting
> > as a pre-requisite or
> > pre-condition) filtered this material out. (e.g. test case on
> > conformance to
> > "simple SOAP", will not be relevant to messages generated as SWA,
> > and the
> > test case may not be able to "force" the implementation under test to
> > generate the former instgead of the latter.)
> > (b)- The environment of the test case other than the testbed
> > itself (e.g.
> > expected events from other parties assisting the test case, expected
> > resources external to the testbed such as an error log site)
> > cannot be
> > obtained, or failed during execution, so that the outcome would
> > not be
> > meaningful.
> >
> > Action:
> > The expected remedial actions include, for (a) possibly replay the
> > test case
> > (in case there are some chances the proper material can be
> > obtained), and
> > for (b) fixing the test case environment, possibly the test case
> > itself, so
> > that the test case execution can proceed.
> >
> > System Exception:
> > ----------------
> >
> > Meaning:
> > The test case execution failed due to the inability of the testbed to
> > complete its execution
> > because of problems related to the testbed itself. E.g., a connection
> > between testbed components failed.
> > Or, material being manipulated (e.g. message data) is too large to be
> > handled by this version of the tsetbed.
> >
> > Action:
> > The expected remedial actions includes either upgrading / fixing the
> > testbed, or modifying the testcases so that the material being
> > handled can
> > be processed by the testbed.
> >
> >
> >
> To unsubscribe from this mailing list (and be removed from the roster of
the OASIS TC), go to

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]