Re: [ebxml-iic] test case outcomes: semantics

[Monica Martin <Monica.Martin@Sun.COM>]

Jacques,
Do you think it is clear enough between Undetermined and System Exception where:
The former is for a test case, for example, cannot fulfill its purpose (i.e. material issue) vs. the system exception is specific to a test bed condition (i.e. the infrastructure has failed to support the test).  

Perhaps my example could be helpful.

On a secondary note, comments on your Section 7.1 revisions yesterday:
1. TC #1: Some confusion may still exist about message and business acknowledgments. At the business level, we have to deal with syntax (receipt ack), intellibility (receipt ack QOS) and semantic validation (can be processed by an application) [acceptance ack]. 
2. TC#2: The response is completely separate; there has been discussion for example in UBL to potentially separate 'response' to an 'acceptance' or 'rejection'. It has been pointed out to me these are not the same (structurally perhaps, semantically not).   Note: The business process delegates to and depends (too) on the reliable messaging infrastructure, but msg ack!=bus ack. 

On above points, I would err on being explicit about the separation as much as possible. When we get to ebBP testing, we will have to discuss how to represent the QoS constraints usage (for example, isIntelligibleCheckRequired). Dale might have more to add (cc: him herein).

Jacques, you can get an example of the ebXML case studies at: http://www.ebxml.org/case_studies/index.htm

Thanks. 

----- Original Message -----
From: Jacques Durand <JDurand@us.fujitsu.com>
Date: Tuesday, August 17, 2004 12:33 pm
Subject: [ebxml-iic] test case outcomes: semantics

> Miekand all:
> 
> here is a proposed insert on the meaning of test case outcomes:
> For review 
> (note: there may be a blurred distinction between som cases of
> "Undetermined" and "system exception")
> 
> Jacques
> 
> 
> Pass: 
> -----
> 
> Meaning:
> The material under test (e.g. messages from the implementation 
> under test)
> is satisfying
> the test requirement associated with the test case.
> 
> Fail:
> -----
> 
> Meaning:
> The material under test (e.g. messages from the implementation 
> under test)
> does not satisfy the test requirement associated with the test 
> case. 
> 
> Action:
> The expected remedial actions include fixing the implementation 
> under test, 
> as this is a failure to conform to the specification as being 
> verified by
> this test suite.
> 
> 
> Undetermined:
> -------------
> 
> Meaning:
> The test case execution could not conclude by either a Fail or a 
> Pass, 
> because of the irrelevance of the material under test, or because 
> the right
> context for this test case could not
> be created. These reasons could be:
> (a)- the material under test is simply not relevant to the test 
> requirementbeing tested, and a preliminary test assertion (acting 
> as a pre-requisite or
> pre-condition) filtered this material out. (e.g. test case on 
> conformance to
> "simple SOAP", will not be relevant to messages generated as SWA, 
> and the
> test case may not be able to "force" the implementation under test to
> generate the former instgead of the latter.)
> (b)- The environment of the test case other than the testbed 
> itself (e.g.
> expected events from other parties assisting the test case, expected
> resources external to the testbed such as an error log site) 
> cannot be
> obtained, or failed during execution, so that the outcome would 
> not be
> meaningful.
> 
> Action:
> The expected remedial actions include, for (a) possibly replay the 
> test case
> (in case there are some chances the proper material can be 
> obtained), and
> for (b) fixing the test case environment, possibly the test case 
> itself, so
> that the test case execution can proceed.
> 
> System Exception:
> ----------------
> 
> Meaning:
> The test case execution failed due to the inability of the testbed to
> complete its execution
> because of problems related to the testbed itself. E.g., a connection
> between testbed components failed.
> Or, material being manipulated (e.g. message data) is too large to be
> handled by this version of the tsetbed. 
> 
> Action:
> The expected remedial actions includes either upgrading / fixing the
> testbed, or modifying the testcases so that the material being 
> handled can
> be processed by the testbed.
> 
> 
>