definitions

[Jacques Durand <JDurand@fsw.fujitsu.com>]

Title: definitions

OK, here is attached an update version based on Mike/Monica comments.
If we agree, we should insert into the Conf spec, and also candidate for the next TestFramework update.

>[MIKE] - Following Monica's thread, I think that we should identify
>assertions in the conformance testing requirements
>that we recognize as "implementation guidelines", since in fact the ebMS
>specification
>is full of them.   For example, "storing a message in its entirety in
>persistent store" is one
>assertion that falls in the implemtation guideline category.  I believe that
>any test requirement identified as an
>implementation guideline should NOT have an abstract test case.

Definitely. But I can't think of many such cases in ebMS.
maybe some SHOULD fall into this class...

>If, however we recognize
>a test requirement as being a valid conformance test requirement, but we
>cannot write a test case for it because of constraints
>on our test framework, then we SHOULD write an abstract test case for it
>(e.g. MSH interrupts and
>restarts ).  I think that this approach would give us a consistent test
>requirements (and test suite) document.

Absolutely. Our definitions of abstract / executable show this is possible.

Jacques
<<Test_Req-Abstract-Concrete.txt>>

1. Test Requirements (also called Test Assertions in literature)
-----------------------------------------------------------------------------------------
(See Test Framework Section 5.)
A statement that rewords a requirements of the original specification, in a testable way.
The Test Requirement is more structured than the specification narrative:
it identifies more precisely the material to be tested (e.g. the message content, the state of the 
implentation under test, or some other specific behavior of the impl under test: 
message sending, timing.
It also identifies formally under which conditions the requirement applies.
The test requirement is to a large extent independent from any specific test framework.
Test requirements can be developed without explicit knowledge of a particular test infrastructure 
or environment (i.e. an implementation of the test environment designed to support the framework)
and should not even make any assumption on teh test methodology (e.g. black box).


2. Abstract test Cases:
---------------------------------
An abstract test case more precisely verifies a test requirement, or parts of it 
(e.g. by defining a subset of  situations where the test requirement applies.).
So it generally does not cover all the situations relevant to a test requirement,
but is representative of the most common situations.
It defines detailed test material (e.g. message content, context of use), defines test steps.
It is precise enough so that test cases can be created that will yield a correct and repeatable 
test result.
It is more restricted than a test requirement, because focusing on the input, output and controls
that have been described in the specification (and therefore can be assumed to be supported by
all implementations), or that are implicitly assumed to be supported (e.g. manual shutdown and 
restart). In other words, each step of the abstract test case is either:
(1) capturing and/or analyzing the observable behavior (generally black-box), 
(2) manipulating available controls for the implementation under test. 
For example, for ebMS, (1) includes the messages generated on the wire, 
the message data passed to the application, plus errors and notifications to application. 
(2) is limited to the following material and actions: input of (received) messages,
configuration with CPA data, MSH shutting down / starting. 
Any test requirement involving some state or material for which the original specification 
does not describe any standard input/output, interface or control, cannot be expressed as is 
by an abstract test case, as it would be at the discretion of each implementation. 
This is the case about requirements about the state of the persistence store that is supposed 
to be part of an MSH implementation: persistence of a message cannot be directly observed, 
only its effect on the overall behavior of the MSH (e.g. resending the messsage after a 
shutdown/recovery) can be observed in an abstract test case.


3. Executable test cases:
---------------------------------
An executable test case uses and assumes an instance of the Test Framework, i.e. a 
specific test harness, which provides both the execution environment and the (executable)
definition syntax for the test case.
In the case of ebXML, the executable test case is a translation of an abstract test case 
into the material specified by a particular version of the Test Framework.
Therefore, depending on the version of the Test Framework that is referred to, the same 
abstract test case can have different executable expressions. 
Some test harness restrictions may cause an abstract test case to not be translatable
into an executable test case: e.g. for ebMS and V1.0 of the Test Framework, inability 
to generate some kind of application input, 
and inability to carry out shutdown and restart in an automated way.
For these reasons, it may happen that - for a particular test framework - an abstract test case 
(and the test requirement it verifies) does not have an executable equivalent, or is only
partially verified by an executable test case.