[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: Issues list: revised with new spec references
|
Jacques, My comments below. In a nutshell, I agree with your “exception” notion as a way
to set the ultimate result state of the Test Case, and Believe that using such a method to establish the ultimate state of the
Test Case (pass/fail/undetermined) is crucial. I believe, however, that we can define a deterministic execution model
where we do not need to manually “override” the default behavior of the Test Driver (setting exitConditions to our own desired value). I believe
we should avoid this if at all possible, since this puts a burden on the test
writer to have to “tweak” their test to get their desired results. It would be more desirable to have a
totally deterministic and explainable test execution model. I know that setting the exitConditions is “explainable” and “deterministic”..
but it requires human intervention where I don’t believe we need it. I also do not see the need
to “flip” or “negate” or “exitOnTrue/exitOnFalse” a <TestAssertion>. I believe that a test writer can define
(using XPath) both the positive and negative of an Assertion adequately ( as I will illustrate below ). You don’t explicitly mention “negation”,
but that is how I am interpreting what you said in item #2 of your previous post: <Quote> "whenever
a Test Step executes a TestAssertion, an Exit statement may be associated with
the TestAssertion result (either True or False).” </Quote> I believe that a “false”
result of a <TestAssertion> should be sufficient to trigger an
appropriate exitCondition ( or perhaps we should call it an exitException
). I think That muddying the waters with “true” or “false” triggering an exception
just makes things more confusing than is necessary in my opinion. Anyway, below is my
proposed wording for describing the “final state” of a Test Case. One more thing of note..
which was really the most difficult to surround.. was the notion of a single,
asynchronous Thread, and how to treat it in the execution model. Sorry it took so long to respond, but I needed to totally be clear on
what the execution model should be. Comments needed, as I will begin modifying based on our decision. Mike -----Original
Message----- Thx Mike, Here is a first cut at
more accurate test case outcome definitions (section 7.1): A Test
Case has a final state of "fail" if: Either: - Any TestAssertion
operation in the Test Case workflow evaluates to a result of "false"
(fail), and this result is not "interpreted"
otherwise (need more precise explanation here.) -
or an
"exitCondition" attribute was set to.... (expand here) [MIKE] – EXPANDED THIS
TO: A Test Case has a final state of "fail" if: Any
TestAssertion that evaluates to a result of "false", and from which no further workflow execution can occur
(i.e. no branching is possible) based upon its boolean result causes the Test
Driver to cease execution of the Test Case, and report a final result of the
Test Case as “fail”. In the case of a “false”
result in an asynchronous Thread that contains that TestPreCondition, the
execution of all other concurrent Threads belonging to that Split MUST also
complete so that it can be determined if workflow execution may continue based
upon the associated Join operation. . If the Thread in question is not subsequently Joined
in the workflow, then the Test Case execution ceases with a final result of “undetermined”. (I believe that an “
exitCondition” and “exitOnTrue” and “exitOnFalse” are not needed, if we have a
deterministic execution model. I
believe that we do. What the statement
above says is (reading between the lines): If
a TestAssertion evaluates to “false”, and the test script does not “branch” anywhere
based upon that “false” result of the TestAssetion, then
the final state of the TestCase is “fail”, because this was the last
TestAssertion executed, and its “exitCondition” (using your terminology, but
not the actual attribute) is “fail” ( i.e. we include the default behavior you
describe below for a “false” TestAssertion. The Test
Driver is intelligent enough recognize that it has no further workflow path,
and must make a call on the final result of the Test Case based upon its last
executed TestAsssertion. Note that this model is more
complex than Test Framework V1.0, because V1.0 had a totally serial execution
of TestSteps, and did not have Threads (i.e no “nesting”). In V1.0, anywhere where a TestAssertion
result =”false” occurred was an automatic TestCase “fail”, because there was no “branching” in its
syntax, and every TestAssertoin had to evaluate to “true”. It was an easy call for the Test Driver
to make. Branching permits
options for the Test Driver to recognize a failed TestAssertion, but still
continue execution, if the script permits it ( via
<If><Then><Else>
or via <Split><Join> ) In
V2.0, I can fail a TestAssertion nested deep inside of a Thread, but if the
workflow (boolean) logic permits a branch based upon the result of the TestAssertion,
OR based upon the boolean result of the Thread that CONTAINS the TestAssertion
(i.e. the boolean “false” result “bubbles up” through the object hierarchy),
then the “false” result of the TestCase DOES NOT result in an automatic
TestCase state of “fail”. That
would ultimately be determined by the resulting workflow branch and what occurs
there. Now, lastly, the
issue of “asynchronous threads”. If we assume the logic described above, than a “lone asynchronous
Thread” that returns a value of “false” would create a “fail” exitCondition if
workflow execution could not continue if its result is “false”. On the other hand.. if there is a Join operation that
includes this thread, and in particular it is an ORJOIN… and another concurrent
thread results in “true”.. then workflow execution can continue.. and the Test
Case would not fail. This is my interpretation of how
we can have a deterministic execution model that will not require intervention
by the Test Writer to handle ambiguity.. because there is no ambiguity. Comments? A Test
Case has a final state of "undetermined" if: The Test case workflow
has terminated all its threads, and the last test step that did a verification
had a TestPreCondition result of value "false". In case the test case
was running several threads concurrently,....? (we can't just say that
first thread to generate
"undetermined" causes the entire test case to be so: we could argue
that [MIKE] – EXPANDED THIS
TO: A Test Case has a final state of "undetermiend" if: Any
TestPreCondition operation that evaluates to a result of "false", and from which no further workflow execution can occur
(i.e. no branching is possible) based upon its boolean result causes the
Test Driver to cease execution of the Test Case, and report a final result of
the Test Case as “undetermined”.
In the case of a “false” TestPreCondition result within an asynchronous Thread,
the execution of all other concurrent Threads belonging to that Split MUST also
complete so that it can be determined if workflow execution may continue based
upon the associated Join operation.
If the Thread in question is not subsequently Joined in the workflow,
then the Test Case execution ceases with a final result of “undetermined”. A Test
Case has a final state of "pass" if: The Test case workflow
has terminated all its threads, and has not "failed", nor satisfied
the conditions for "undetermined outcome". [MIKE] – This works for
me. It still appears to me
that due to concurrent paths in the test case, and the possibility to
"interpret" TA results, the above definitions can
be tricky, and confusing to users. The
point I was trying to make with item #2 previous mails, is that we need provide
means to decouple the result
of a TestAssertion with the general outcome of the test case (e.g. T.A. =
"true" could mean failure
of the Test Case, the opposite could be true as well.) [MIKE] – I argue that a
test writer can understand that if their assertion results in “true” its a
“pass”, and if their assertion results in a “false” it “fails”.. and that we do
not need any additional operators at all defined for TestAssertion (i.e. <TestAssertion exitOn = “true”/> I think this is what you are
suggesting. A Test Writer can write XPath queries without the need to “negate”
their TestAssertion result
externally.. If they wish to “negate” a TestAssertion, they can do it within
XPath with the XPath “not” operator.
For example: <TestAssertion>/FilterResult//eb:ErrorList</TestAssertion>
(test for the presence of the ErrorList element in FilterResult, return
“true” if found) <TestAssertion>/FilterResult//*[not(eb:ErrorList)]</TestAssertion>
(test for presence of the ErrorList element in FilterResult, return
“true” if NOT found) Adding an additional exitOn attribute would only make
understanding of the query more complex.
I suggest letting the XPath query say what it means without “externally”
negating an already complex syntax. I
believe we can do that today, but it is not
explicit in the spec. [MIKE] – It is not in
the spec, and perhaps should be part of our vote. Also,
assume the following test case: (1)
putMessage M1 (2)
getMessage M2 (3)
getMessage M3 We
may want that TA(M2)=true means "Fail" for teh test case. Yet
we may also want that any other result for M2 (precond = "false", TA
= "false") to be inconsequential, and
let step (3) proceed, which will decide of the test case outcome. How can we
script this? [MIKE] – Based on my definitions above, precond=”false”, TA = “false” will
be inconsequential IF the test writer allows the Test Driver to “branch” from this result (using
<If><Then><Else>).
So there is no need for them to “set” any attributes anywhere, they just
have to script their logic appropriately. One
suggestion: -
we could add this mostly editorial refinement about the outcome of a
TA: there are 3 possible TA outcomes: "true",
"false" and "non-applicable" in case the precond
is "false". -
We could tag each TA with an exitCondition attr of 4 possible values: o
"Fail" (on TA value = true or false or N/A), o
"Pass" (on value = true or false or N/A), o
"Undetermined" (on value = true or false or N/A), o
"Continue" (on value = true or false or N/A) (no actual exit,
meaning, we proceed to next test step). -
default
is "Fail" for TA=false, "Continue" for TA=true,
"Undetermined" for TA=N/A. [MIKE] – I agree with the concept of “exception” conditions of “pass”, “fail”,
“undetermined”, “continue”. I
believe that all of these can be internally defined by the Test Drive, based up the deterministic
model described above. I am not in favor of letting a test writer arbitrarily redefine the “exitCondition”
of a <TestAssertion>. All
the rules I state in my definitions Of Test Case pass/fail are meaningless , including the meaning of a TestPreCondition and a
TestAssertion. So,
unless we use explicit exit tags to TAs, the following default behavior will
apply on a very simple basis: -
first
TA evaluating to "false" or to "N/A" will exit the test
case respectively on "Fail" and "Undetermined". [MIKE] – Agreed, if there is NO BRANCHING possible that would allow the
workflow to continue -
if the entire workflow completes naturally according to its logic (reaches the
"final" state), it
means the test case outocme is "Pass". [MIKE] – That would be true. So
these exit tags above are not so much "go to" than exceptions: [MIKE] - I agree with that,
they are settting the state, not directing the Test Driver to stop execution The
overall exec model is more like: try
{ ... -
test case workflow, that can raise exceptions "Fail" "Pass"
"Undetermined" ... -
execute Pass termination logic for the test case. } catch
{ exit="Fail":
{...execute Fail termination logic for the test case...} exit="Pass":
{...execute Pass termination logic for the test case...} exit="Undetermined":
{...execute Undetermined logic ....} } Opinion? Jacques -----Original Message----- Jacques and all, I've revised the Test
Framework issues list to reflect current locations in spec addressing
particular issues. Regards, Mike \
B- Message
store and getMessage semantics - Section
7.1.3.1 and 7.1.9.1 also need to break apart RPC messages from Test Service ebXML messages (
i.e. make 2 separate schemas ) |
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]