Jacques,
My comments below. In a nutshell, I agree with your
"exception" notion as a way to set the ultimate result state of the Test Case,
and
Believe
that using such a method to establish the ultimate state of the Test Case
(pass/fail/undetermined) is
crucial.
I
believe, however, that we can define a deterministic execution model where we
do not need to manually "override" the default
behavior
of the
Test Driver (setting exitConditions to our own desired value). I believe we
should avoid this if at all possible, since this puts a burden on the test
writer
to
have to "tweak" their test to get their desired results. It would be more desirable to have a
totally deterministic and explainable test execution
model.
I know
that setting the exitConditions is "explainable" and "deterministic".. but it
requires human intervention where I don't believe we need
it.
I also do not see the need to "flip" or
"negate" or "exitOnTrue/exitOnFalse" a <TestAssertion>. I believe that a test writer can
define (using XPath)
both
the positive and negative of an Assertion adequately ( as I will illustrate below ). You don't explicitly mention
"negation", but that
is how
I am interpreting what you said in item #2 of your previous
post:
<Quote> "whenever a Test Step executes a
TestAssertion, an Exit statement may be associated with the TestAssertion
result (either True or False)."
</Quote>
I believe that a "false" result of a
<TestAssertion> should be sufficient to trigger an appropriate
exitCondition ( or perhaps we should call it an exitException ). I
think
That
muddying the waters with "true" or "false" triggering an exception just makes
things more confusing than is necessary in my
opinion.
Anyway, below is my proposed wording
for describing the "final state" of a Test
Case.
One more thing of note.. which was
really the most difficult to surround.. was the notion of a single,
asynchronous Thread, and how to treat it in the execution
model.
Sorry
it took so long to respond, but I needed to totally be clear on what the
execution model should be.
Comments
needed, as I will begin modifying based on our
decision.
Mike
-----Original
Message-----
From: Jacques
Durand [mailto:JDurand@us.fujitsu.com]
Sent: Monday, April 12, 2004 8:04
PM
To: 'Michael Kass';
Jacques Durand; ebXML IIC - main list (E-mail) (E-mail)
Cc: 'tsakach@certivo.net'
Subject: RE: Issues list: revised with
new spec references
Thx
Mike,
Here is
a first cut at more accurate test case outcome definitions (section
7.1):
A Test Case has a
final state of "fail" if:
Either:
- Any TestAssertion
operation in the Test Case workflow evaluates to a result of "false"
(fail), and this result is not
"interpreted"
otherwise (need more precise explanation here.)
-
or an
"exitCondition" attribute was set to.... (expand here)
[MIKE]
- EXPANDED THIS TO:
A Test
Case has a final state of "fail" if:
Any
TestAssertion that evaluates to a result of "false", and from which no further workflow execution can
occur (i.e. no branching is possible) based upon its boolean result
causes the Test Driver to cease execution of the Test Case, and report a final
result of the Test Case as "fail". In the case of a "false" result in an
asynchronous Thread that contains that TestPreCondition, the execution of all
other concurrent Threads belonging to that Split MUST also complete so that it
can be determined if workflow execution may continue based upon the associated
Join operation. . If the
Thread in question is not subsequently Joined in the workflow, then the Test
Case execution ceases with a final result of
"undetermined".
(I believe that an "
exitCondition" and "exitOnTrue" and "exitOnFalse" are not needed, if we have a
deterministic execution model. I
believe that we do. What the statement above says is (reading
between the lines):
If a TestAssertion evaluates to "false", and the test script does not
"branch" anywhere based upon that
"false" result of the
TestAssetion, then the final state of the TestCase is "fail", because this was
the last TestAssertion executed, and its "exitCondition" (using your
terminology, but not the actual attribute) is "fail" ( i.e. we include the
default behavior you describe below for a "false"
TestAssertion.
The Test Driver is intelligent enough recognize that it has no further
workflow path, and must make a call on the final result of the Test Case based
upon its last executed TestAsssertion.
Note that
this model is more complex than Test Framework V1.0, because V1.0 had a
totally serial execution of TestSteps, and did not have Threads (i.e no
"nesting"). In V1.0, anywhere
where a TestAssertion result ="false" occurred was an automatic TestCase "fail", because there was no
"branching" in its syntax, and every TestAssertoin had to evaluate to
"true". It was an easy call for
the Test Driver to make.
Branching permits
options for the Test Driver to recognize a failed TestAssertion, but still
continue execution, if the script permits it ( via
<If><Then><Else>
or via <Split><Join> )
In
V2.0, I can fail a TestAssertion nested deep inside of a Thread, but if the
workflow (boolean) logic permits a branch based upon the result of the
TestAssertion, OR based upon the boolean result of the Thread that CONTAINS
the TestAssertion (i.e. the boolean "false" result "bubbles up" through the
object hierarchy), then the "false" result of the TestCase DOES NOT result in
an automatic TestCase state of "fail".
That would ultimately be determined by the resulting workflow branch
and what occurs there.
Now, lastly, the
issue of "asynchronous threads". If we assume the logic described above,
than a "lone asynchronous Thread" that returns a value of "false" would create
a "fail" exitCondition if workflow execution could not continue if its result
is "false". On the other hand.. if there is a Join
operation that includes this thread, and in particular it is an ORJOIN... and
another concurrent thread results in "true".. then workflow execution can
continue.. and the Test Case would not fail.
This is my interpretation of how
we can have a deterministic execution model that will not require intervention
by the Test Writer to handle ambiguity.. because there is no ambiguity.
Comments?
A Test Case has a
final state of "undetermined" if:
The Test case
workflow has terminated all its threads, and the last test step that did a
verification had a TestPreCondition result of value "false".
In case the test
case was running several threads concurrently,....? (we can't just say
that first thread
to generate
"undetermined" causes the entire test case to be so: we could argue
that
[MIKE]
- EXPANDED THIS TO:
A Test
Case has a final state of "undetermiend" if:
Any
TestPreCondition operation that evaluates to a result of "false", and from which no further workflow execution can
occur (i.e. no branching is possible) based upon its boolean result causes
the Test Driver to cease execution of the Test Case, and report a final result
of the Test Case as "undetermined".
In the case of a "false" TestPreCondition result within an asynchronous
Thread, the execution of all other concurrent Threads belonging to that Split
MUST also complete so that it can be determined if workflow execution may
continue based upon the associated Join operation. If the Thread in question is not
subsequently Joined in the workflow, then the Test Case execution ceases with
a final result of "undetermined".
A Test Case has a
final state of "pass" if:
The Test case
workflow has terminated all its threads, and has not "failed", nor satisfied
the conditions for "undetermined outcome".
[MIKE]
- This works for me.
It still
appears to me that due to concurrent paths in the test case, and the
possibility to "interpret" TA results,
the
above definitions can be tricky, and confusing to users.
The
point I was trying to make with item #2 previous mails, is that we need
provide means to decouple the
result
of a TestAssertion with the general outcome of the test case (e.g. T.A. =
"true" could mean
failure
of the Test Case, the opposite could be true as well.)
[MIKE]
- I
argue that a test writer can understand that if their assertion results in
"true" its a "pass", and if their assertion results in a "false" it "fails"..
and that we do not need any additional operators at all defined for
TestAssertion (i.e. <TestAssertion exitOn =
"true"/> I think this is
what you are suggesting.
A
Test Writer can write XPath queries without the need to "negate" their TestAssertion result externally.. If
they wish to "negate" a TestAssertion, they can do it within XPath with the
XPath "not" operator. For
example:
<TestAssertion>/FilterResult//eb:ErrorList</TestAssertion>
(test for the presence of the ErrorList element in FilterResult, return
"true" if
found)
<TestAssertion>/FilterResult//*[not(eb:ErrorList)]</TestAssertion>
(test for presence of the ErrorList element in FilterResult, return
"true" if NOT
found)
Adding
an additional exitOn attribute would only make understanding of the query more
complex. I suggest letting the
XPath query say what it means without "externally" negating an already complex
syntax.
I
believe we can do that today, but it is
not
explicit in the spec.
[MIKE]
- It is not in the spec, and perhaps should be part of our
vote.
Also,
assume the following test case:
(1)
putMessage M1
(2)
getMessage M2
(3)
getMessage M3
We
may want that TA(M2)=true means "Fail" for teh test case.
Yet
we may also want that any other result for M2 (precond = "false", TA =
"false") to be inconsequential,
and
let step (3) proceed, which will decide of the test case outcome. How can we
script this?
[MIKE]
- Based on my definitions above, precond="false", TA = "false" will be
inconsequential IF the test writer allows the Test
Driver
to
"branch" from this result (using <If><Then><Else>). So there is no need for them to "set"
any attributes anywhere, they just have
to
script their logic appropriately.
One
suggestion:
-
we could add this mostly editorial refinement about the outcome of
a TA: there are 3 possible TA outcomes:
"true",
"false" and "non-applicable" in case the precond is
"false".
-
We could tag each TA with an exitCondition attr of 4 possible
values:
o
"Fail" (on TA value = true or false or N/A),
o
"Pass" (on value = true or false or N/A),
o
"Undetermined" (on value = true or false or N/A),
o
"Continue" (on value = true or false or N/A) (no actual exit, meaning, we
proceed to next test step).
-
default
is "Fail" for TA=false, "Continue" for TA=true, "Undetermined" for
TA=N/A.
[MIKE]
- I agree with the concept of "exception" conditions of "pass", "fail",
"undetermined", "continue". I
believe that all of these
can be
internally defined by the Test Drive, based up the deterministic model
described above.
I am
not in favor of letting a test writer arbitrarily redefine the "exitCondition"
of a <TestAssertion>. All
the rules I state in my definitions
Of
Test Case pass/fail are meaningless ,
including the meaning of a TestPreCondition and a
TestAssertion.
So,
unless we use explicit exit tags to TAs, the following default behavior will
apply on a very simple basis:
-
first
TA evaluating to "false" or to "N/A" will exit the test case respectively on
"Fail" and "Undetermined".
[MIKE]
- Agreed, if there is NO BRANCHING possible that would allow the workflow to
continue
-
if the entire workflow completes naturally according to its logic (reaches the
"final" state),
it
means the test case outocme is "Pass".
[MIKE]
- That would be true.
So
these exit tags above are not so much "go to" than
exceptions:
[MIKE] - I agree with that, they are settting
the state, not directing the Test Driver to stop
execution
The
overall exec model is more like:
try
{
...
-
test case workflow, that can raise exceptions "Fail" "Pass"
"Undetermined"
...
-
execute Pass termination logic for the test case.
}
catch
{
exit="Fail":
{...execute Fail termination logic for the test case...}
exit="Pass":
{...execute Pass termination logic for the test case...}
exit="Undetermined":
{...execute Undetermined logic ....}
}
Opinion?
Jacques
-----Original
Message-----
From: Michael
Kass [mailto:michael.kass@nist.gov]
Sent: Monday, April 12, 2004 1:34
PM
To: Michael Kass; Jacques
Durand; ebXML IIC - main list (E-mail) (E-mail)
Subject: Issues list: revised with new
spec references
Jacques
and all,
I've revised the Test Framework issues list to reflect current
locations in spec addressing particular
issues.
Regards,
Mike
\
A- Script
extensions
- filter-results notation.
- Section
7.1.4.1 and Appendix D
(schemas )
- split / join ops - Defined in section 7.1 table
10, section 7.1.1, table
11
- conditional
ops, where used. - Section
7.1.8
- exception catching best practice - All tables in the spec
- timing of
test steps, timeout vs "sleep". - Two
different things completely , described as "stepDuration" section
7.1.2.2 table 12 and
"Sleep", tables 10, 11,
33
B- Message store
and getMessage semantics - Section 7.1.3.1 and
7.1.9.1
- masking / unmasking the events that
are selected. - Section
7.1.3.1
- scope of a
store. - 7.1.3.1 ... first
paragraph
- time limit for a getMessage op -
Defined as "stepDuration", section
7.1.2.2, table
12
C-Parameters / variables
- how
are they set.. Section
4.2.2.1
-
pre-processing model, especially in filters - Section 7.1.2.3.2
D- Interfaces
- assumed
communication model. - Section 3.2.5,
assumes an RPC communication model
- how ebMS-independent -
Section 3.2.5 describes RPC communication
model, ebMS independent..with SOAP binding example
- schema for
message data. - Appendix F..
and section
7.3
also need to break
apart RPC messages from Test Service ebXML messages ( i.e. make 2 separate
schemas )
E- General execution model
-
how do we execute an entire test suite. - Section
3.3.1
- possible
outcomes for a test case, meaning for the test suite. - Section 7.1
- when a test has threads
that don't join. - Section
7.1.1
- best practices for Exception
threads? - Method "how" described.. but no example - Section
7.1.1
(e.g. verify we get either an
Acceptance or a Reject, but not both?) No
examples in spec. Suggest creating sample Test Case performing
"if", "then" , "else" to accomplish illustrate handling this particular test
case ( since there is no "orJoin")