[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: Issues list: revised with new spec references
Jacques and all, My comments below: -----Original
Message----- Mike: Mostly
two points to discuss: >Any TestAssertion that evaluates to a result of
"false", and from which no
further > >workflow execution can occur (i.e. no branching is possible) based upon its boolean >result causes the Test Driver to cease execution of the
Test Case, and report a final result >of the Test Case as "fail". Even
if we don't add explicit exits statements - agree we should avoid them if
we can -, I find the definition above a little shaky, and we should try to make
it better if we can: -
what bothered me
in this definition, is that it is context-dependent: the interpretation of this
step in a thread depends on what is "around" the thread (here, what's
after). [MIKE] – This is true.
It will always be “context dependent” upon the surrounding boolean logic. So not only does the
<TestAssertion> determine flow, but how that result “bubbles up” through
the logic tree will also ultimately determine whether a TestCase passes or
fails. Perhaps a little more
definition can be added here. -
Also, what if we
always or-join all threads at the very end of a test case, as a best scripting
practice, with no other step behind? We need to better define the
"end" of a thread, or what "workflow execution" means. [MIKE] – What about an “and-join”?
Wouldn’t that be preferable? I agree that a best practice for “hanging
threads” would be desirable. A Test Writer can write XPath queries
without the need to "negate" their TestAssertion result externally.. If they wish to
"negate" a TestAssertion, they can do it within XPath with the XPath
"not" operator. For
example: <TestAssertion>/FilterResult//eb:ErrorList</TestAssertion>
(test for the presence of the ErrorList element in FilterResult, return
"true" if found) <TestAssertion>/FilterResult//*[not(eb:ErrorList)]</TestAssertion>
(test for presence of the ErrorList element in FilterResult, return
"true" if NOT found) Users
will have to be very cautious using negation when cardinality is involved, i.e.
when the filterResult contains several selected messages. This is a general
issue with negation. In
your example above, Case
(a): first Assertion would succeed if there is at least 1 message in the
selection that has an ErrorList. Case (b) With the second Assertion, it would
succeed if there is AT LEAST one message in teh selection with NO ErrorList. [MIKE] – This is true, that users will have to be cautions, and
construct their Filters.. and their TestAssertions very selectively. But that will always be the case,
regardless of whether negation occurs “inside” of XPath.. or externally. I see your point with Case (b).. A better way to write the TestAssertion
is: <TestAssertion>/[not(FilterResult//ErrorList)]</TestAssertion> (which
says this Assertion succeeds only if there are NO ErrorLists anywhere in the
FilterResult) Case
(c) Now, what if I want the test case to FAIL in case there is AT LEAST one
message in teh selection with NO ErrorList? That means, we would have to write
a test Assertion that is true if ALL the filter-selected messages have each an
ErrorList. That becomes tedious and error prone: don't you think it would be
easier to check for at least one without errorlist (like in your case (b)
above), then just flip the exit interpretation to "fail"? [MIKE] – I would write it with the “double negation”.. <TestAssertion>/[not(FilterResult/Message//*[not(eb:ErrorList)])]</TestAssertion> Again, if you didn’t use XPath to do it, you would still have to do the same thing if we externally “negated”
the XPath assertion <TestAssertion negate=”true”>/FilterResult/Message//*[not(eb:ErrorList)]</TestAssertion> My feeling is, why add something that already exists in XPath? And I don’t think that it makes it any
more readable or understandable. Jacques -----Original Message----- Jacques, My
comments below. In a
nutshell, I agree with your "exception" notion as a way to set the
ultimate result state of the Test Case, and Believe that using such a method to establish the
ultimate state of the Test Case (pass/fail/undetermined) is crucial. I believe, however, that we can define a
deterministic execution model where we do not need to manually
"override" the default behavior of the Test Driver (setting exitConditions to our own
desired value). I believe we should avoid this if at all possible, since this
puts a burden on the test writer to have to "tweak" their test to get their
desired results. It would be more
desirable to have a totally deterministic and explainable test execution model. I know that setting the exitConditions is
"explainable" and "deterministic".. but it requires human
intervention where I don't believe we need it. I also
do not see the need to "flip" or "negate" or
"exitOnTrue/exitOnFalse" a <TestAssertion>. I believe that a test writer can define
(using XPath) both the positive and negative of an Assertion
adequately ( as I will illustrate
below ). You don't explicitly
mention "negation", but that is how I am interpreting what you said in item #2 of
your previous post: <Quote>
"whenever a Test Step executes a TestAssertion, an Exit statement
may be associated with the TestAssertion result (either True or False)."
</Quote> I
believe that a "false" result of a <TestAssertion> should be
sufficient to trigger an appropriate exitCondition ( or perhaps we should call
it an exitException ). I think That muddying the waters with "true" or
"false" triggering an exception just makes things more confusing than
is necessary in my opinion. Anyway,
below is my proposed wording for describing the "final state" of a
Test Case. One
more thing of note.. which was really the most difficult to surround.. was the
notion of a single, asynchronous Thread, and how to treat it in the execution
model. Sorry it took so long to respond, but I needed to
totally be clear on what the execution model should be. Comments needed, as I will begin modifying based on
our decision. Mike -----Original
Message----- Thx
Mike, Here is
a first cut at more accurate test case outcome definitions (section 7.1): A Test
Case has a final state of "fail" if: Either: - Any TestAssertion
operation in the Test Case workflow evaluates to a result of "false"
(fail), and this result is not "interpreted"
otherwise (need more precise explanation here.) - or an "exitCondition" attribute was set to.... (expand
here) [MIKE] -
EXPANDED THIS TO: A Test Case has a final state of "fail" if: Any
TestAssertion that evaluates to a result of "false", and from which no further workflow execution can occur
(i.e. no branching is possible) based upon its boolean result causes the Test
Driver to cease execution of the Test Case, and report a final result of the
Test Case as "fail". In the case of a "false"
result in an asynchronous Thread that contains that TestPreCondition, the
execution of all other concurrent Threads belonging to that Split MUST also
complete so that it can be determined if workflow execution may continue based
upon the associated Join operation. .
If the Thread in question is not subsequently Joined in the workflow,
then the Test Case execution ceases with a final result of
"undetermined". (I believe that an
" exitCondition" and "exitOnTrue" and
"exitOnFalse" are not needed, if we have a deterministic execution
model. I believe that we do. What
the statement above says is
(reading between the lines): If
a TestAssertion evaluates to "false", and the test script does not
"branch" anywhere based
upon that "false" result
of the TestAssetion, then the final state of the TestCase is "fail",
because this was the last TestAssertion executed, and its
"exitCondition" (using your terminology, but not the actual
attribute) is "fail" ( i.e. we include the default behavior you
describe below for a "false" TestAssertion. The Test
Driver is intelligent enough recognize that it has no further workflow path,
and must make a call on the final result of the Test Case based upon its last
executed TestAsssertion.
Note that this model is more complex than Test Framework V1.0, because
V1.0 had a totally serial execution of TestSteps, and did not have Threads (i.e
no "nesting"). In V1.0,
anywhere where a TestAssertion result ="false" occurred was an
automatic TestCase
"fail", because there was no "branching" in its syntax, and
every TestAssertoin had to evaluate to "true". It was an easy call for the Test Driver
to make. Branching permits
options for the Test Driver to recognize a failed TestAssertion, but still
continue execution, if the script permits it ( via
<If><Then><Else>
or via <Split><Join> ) In
V2.0, I can fail a TestAssertion nested deep inside of a Thread, but if the
workflow (boolean) logic permits a branch based upon the result of the
TestAssertion, OR based upon the boolean result of the Thread that CONTAINS the
TestAssertion (i.e. the boolean "false" result "bubbles up"
through the object hierarchy), then the "false" result of the
TestCase DOES NOT result in an automatic TestCase state of "fail". That would ultimately be determined by
the resulting workflow branch and what occurs there. Now, lastly, the
issue of "asynchronous threads". If we assume the logic described above, than a "lone
asynchronous Thread" that returns a value of "false" would
create a "fail" exitCondition if workflow execution could not
continue if its result is "false". On the other hand.. if there is a Join operation that
includes this thread, and in particular it is an ORJOIN... and another
concurrent thread results in "true".. then workflow execution can
continue.. and the Test Case would not fail. This is my interpretation of how
we can have a deterministic execution model that will not require intervention
by the Test Writer to handle ambiguity.. because there is no ambiguity. Comments? A Test
Case has a final state of "undetermined" if: The Test case workflow
has terminated all its threads, and the last test step that did a verification
had a TestPreCondition result of value "false". In case the test case
was running several threads concurrently,....? (we can't just say that
first thread to generate
"undetermined" causes the entire test case to be so: we could argue
that [MIKE] -
EXPANDED THIS TO: A Test Case has a final state of "undetermiend" if: Any
TestPreCondition operation that evaluates to a result of "false", and from which no further workflow execution can occur
(i.e. no branching is possible)
based upon its boolean result causes the Test Driver to cease execution
of the Test Case, and report a final result of the Test Case as
"undetermined". In the
case of a "false" TestPreCondition result within an asynchronous
Thread, the execution of all other concurrent Threads belonging to that Split
MUST also complete so that it can be determined if workflow execution may
continue based upon the associated Join operation. If the Thread in question is not subsequently Joined
in the workflow, then the Test Case execution ceases with a final result of
"undetermined". A Test
Case has a final state of "pass" if: The Test case workflow
has terminated all its threads, and has not "failed", nor satisfied
the conditions for "undetermined outcome". [MIKE] - This
works for me. It still
appears to me that due to concurrent paths in the test case, and the
possibility to "interpret" TA results, the
above definitions can be tricky, and confusing to users. The point I was trying to make with item #2 previous mails,
is that we need provide means to decouple the result of a TestAssertion with the general outcome of the
test case (e.g. T.A. = "true" could mean failure of the Test Case, the opposite could be true as
well.) [MIKE]
- I argue that a
test writer can understand that if their assertion results in "true"
its a "pass", and if their assertion results in a "false"
it "fails".. and that we do not need any additional operators at all
defined for TestAssertion
(i.e.
<TestAssertion exitOn = "true"/> I think this is what you are
suggesting. A Test Writer can write XPath queries
without the need to "negate" their TestAssertion result externally.. If they wish to
"negate" a TestAssertion, they can do it within XPath with the XPath
"not" operator. For
example: <TestAssertion>/FilterResult//eb:ErrorList</TestAssertion>
(test for the presence of the ErrorList element in FilterResult, return
"true" if found) <TestAssertion>/FilterResult//*[not(eb:ErrorList)]</TestAssertion>
(test for presence of the ErrorList element in FilterResult, return
"true" if NOT found) Adding an additional exitOn attribute
would only make understanding of the query more complex. I suggest letting the XPath query say
what it means without "externally" negating an already complex
syntax. I believe we can do that today, but it is not explicit in the spec. [MIKE] - It is not in
the spec, and perhaps should be part of our vote. Also, assume the following test case: (1) putMessage M1 (2) getMessage M2 (3) getMessage M3 We may want that TA(M2)=true means "Fail" for teh
test case. Yet we may also want that any other result for M2 (precond =
"false", TA = "false") to be inconsequential, and let step (3) proceed, which will decide of the test case
outcome. How can we script this? [MIKE] - Based on my definitions above,
precond="false", TA = "false" will be inconsequential IF
the test writer allows the Test Driver to "branch" from this result (using
<If><Then><Else>).
So there is no need for them to "set" any attributes anywhere,
they just have to script their logic appropriately. One suggestion: - we could add this mostly editorial refinement about
the outcome of a TA: there are 3 possible TA outcomes: "true", "false" and
"non-applicable" in case the precond is "false". - We could tag each TA with an exitCondition attr of 4
possible values: o "Fail" (on TA value = true or false or N/A), o "Pass" (on value = true or false or N/A), o "Undetermined" (on value = true or false or
N/A), o "Continue" (on value = true or false or N/A) (no
actual exit, meaning, we proceed to next test step). -
default is
"Fail" for TA=false, "Continue" for TA=true,
"Undetermined" for TA=N/A. [MIKE] - I agree with the concept of
"exception" conditions of "pass", "fail",
"undetermined", "continue". I believe that all of these can be internally defined by the Test Drive, based up
the deterministic model described above. I am not in favor of letting a test writer
arbitrarily redefine the "exitCondition" of a
<TestAssertion>. All the
rules I state in my definitions Of Test Case pass/fail are meaningless , including the meaning of a
TestPreCondition and a TestAssertion. So, unless we use explicit exit tags to TAs, the following
default behavior will apply on a very simple basis: -
first TA evaluating to
"false" or to "N/A" will exit the test case respectively on
"Fail" and "Undetermined". [MIKE] - Agreed, if there is NO BRANCHING possible
that would allow the workflow to continue - if the entire workflow completes naturally according to its
logic (reaches the "final" state), it means the test case outocme is "Pass". [MIKE] - That would be true. So these exit tags above are not so much "go to"
than exceptions: [MIKE] -
I agree with that, they are settting the state, not directing the Test Driver
to stop execution The overall exec model is more like: try { ... - test case workflow, that can raise exceptions
"Fail" "Pass" "Undetermined" ... - execute Pass termination logic for the test case. } catch { exit="Fail": {...execute Fail termination logic for
the test case...} exit="Pass": {...execute Pass termination logic for
the test case...} exit="Undetermined": {...execute Undetermined logic
....} } Opinion? Jacques -----Original Message----- Jacques and all, I've revised the Test
Framework issues list to reflect current locations in spec addressing
particular issues. Regards, Mike \
B- Message
store and getMessage semantics - Section
7.1.3.1 and 7.1.9.1 also need to break apart RPC messages from Test Service ebXML messages (
i.e. make 2 separate schemas ) |
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]