RE: Issues list: revised with new spec references

[Jacques Durand <JDurand@us.fujitsu.com>]

Title: Minutes and next call April 5th

Thx Mike,

 

Here is a first cut at more accurate test case outcome definitions (section 7.1):

 

A Test Case has a final state of "fail" if:

Either:

- Any TestAssertion operation in the Test Case workflow evaluates to a result of "false" (fail), and this result is not

"interpreted" otherwise (need more precise explanation here.)

- or an "exitCondition" attribute was set to.... (expand here)

 

A Test Case has a final state of "undetermined" if:

The Test case workflow has terminated all its threads, and the last test step that did a verification had a TestPreCondition result of value  "false".

In case the test case was running several threads concurrently,....? (we can't just say that first thread

to generate "undetermined" causes the entire test case to be so: we could argue that

 

A Test Case has a final state of "pass" if:

The Test case workflow has terminated all its threads, and has not "failed", nor satisfied the conditions for "undetermined outcome".

 

It still appears to me that due to concurrent paths in the test case, and the possibility to "interpret" TA results,

the above definitions can be tricky, and confusing to users.

The point I was trying to make with item #2 previous mails, is that we need provide means to decouple the

result of a TestAssertion with the general outcome of the test case (e.g. T.A. = "true" could mean

failure of the Test Case, the opposite could be true as well.) I believe we can do that today, but it is

not explicit in the spec.

 

Also, assume the following test case:

(1) putMessage M1

(2) getMessage M2

(3) getMessage M3

 

We may want that TA(M2)=true means "Fail" for teh test case.

Yet we may also want that any other result for M2 (precond = "false", TA = "false") to be inconsequential,

and let step (3) proceed, which will decide of the test case outcome. How can we script this?

 

One suggestion:

- we could add this mostly editorial refinement about  the outcome of a TA:  there are 3 possible TA outcomes:

 "true", "false" and  "non-applicable" in case the precond is "false".

- We could tag each TA with an exitCondition attr of 4 possible values:

o "Fail" (on TA value = true or false or N/A),

o "Pass" (on value = true or false or N/A), 

o "Undetermined" (on value = true or false or N/A), 

o "Continue" (on value = true or false or N/A) (no actual exit, meaning, we proceed to next test step).

- default is "Fail" for TA=false, "Continue" for TA=true, "Undetermined" for TA=N/A.

 

So, unless we use explicit exit tags to TAs, the following default behavior will apply on a very simple basis:

- first TA evaluating to "false" or to "N/A" will exit the test case respectively on "Fail" and "Undetermined".

- if the entire workflow completes naturally according to its logic (reaches the "final" state),

it means the test case outocme is "Pass".

 

So these exit tags above are not so much "go to" than exceptions:

The overall exec model is more like:

try {

...

- test case workflow, that can raise exceptions "Fail" "Pass" "Undetermined"

...

- execute Pass termination logic for the test case.

}

catch {

exit="Fail": {...execute Fail termination logic for the test case...}

exit="Pass": {...execute Pass termination logic for the test case...}

exit="Undetermined": {...execute Undetermined logic ...}

}

 

Opinion?

 

Jacques

-----Original Message-----
From: Michael Kass [mailto:michael.kass@nist.gov]
Sent: Monday, April 12, 2004 1:34 PM
To: Michael Kass; Jacques Durand; ebXML IIC - main list (E-mail) (E-mail)
Subject: Issues list: revised with new spec references

Jacques and all,

 

        I've revised the Test Framework issues list to reflect current locations in spec addressing particular issues.

 

Regards,

Mike

\

 

A- Script extensions
- filter-results notation.  -  Section  7.1.4.1  and Appendix D (schemas )
- split / join ops - Defined in section 7.1  table 10, section  7.1.1, table 11
- conditional ops, where used. - Section 7.1.8
- exception catching best practice - All tables in the spec
- timing of test steps, timeout vs "sleep". - Two different things completely , described as "stepDuration" section 7.1.2.2  table 12 and "Sleep", tables 10, 11, 33

B- Message store and getMessage semantics - Section 7.1.3.1 and 7.1.9.1
- masking / unmasking the events that are selected. - Section 7.1.3.1
- scope of a store. - 7.1.3.1 .. first paragraph
- time limit for a getMessage op - Defined as "stepDuration", section 7.1.2.2, table 12

C-Parameters / variables
- how are they set.. Section 4.2.2.1
- pre-processing model, especially in filters - Section 7.1.2.3.2

D- Interfaces
- assumed communication model. - Section 3.2.5, assumes an RPC communication model
- how ebMS-independent - Section 3.2.5 describes RPC communication model, ebMS independent..with SOAP binding example
- schema for message data. - Appendix F.. and section 7.3

 

also need to break apart RPC messages from Test Service ebXML messages ( i.e. make 2 separate schemas )

E- General execution model
- how do we execute an entire test suite. -  Section 3.3.1
- possible outcomes for a test case, meaning for the test suite. - Section 7.1
- when a test has threads that don't join. - Section 7.1.1
- best practices for Exception threads? - Method "how" described.. but no example - Section 7.1.1
(e.g. verify we get either an Acceptance or a Reject, but not both?) No examples in spec.  Suggest  creating sample Test Case performing "if", "then" , "else" to accomplish illustrate handling this particular test case ( since there is no "orJoin")