RE: [tag] Notes from 11/7 meeting - Anatomy Version 06

["Durand, Jacques R." <JDurand@us.fujitsu.com>]

Adding some inline <JD> comments addressing some of the notes from Paul - last meeting schedule A .

(this NOT a correction of Paul's note  !  but further discussion on these)

 

Jacques

---

From: Paul.Rank@Sun.COM [mailto:Paul.Rank@Sun.COM]
Sent: Friday, November 09, 2007 2:11 PM
To: tag@lists.oasis-open.org
Subject: [tag] Notes from 11/7 meeting - Anatomy Version 06

The following are the notes I took during our review of the "Anatomy of a Test Assertion" section (V06) during the 11/7 meeting.  I've inserted the notes I took in blue italics. 
Paul

=========================================

Anatomy of a Test Assertion:

What is in a Test Assertion:

A test assertion (or TA) must always:

• Be uniquely identified by a test assertion identifier (TA id).

• Refer to the specification requirement(s) that it addresses.

• Identify a class of items under test (or IUT) so that a statement can be made on these IUTs regarding fulfillment of the addressed specification requirements.

Why is this (IUT) singled out and level/profile/module are not singled out?
Example "test assertion can apply at level 3 and above."

Should IUT be in the conformance clause?
-Jacques disagrees – see e-mail from 11/??

< JD> I think there are good reasons to keep the IUT in TA, as mentioned in the meeting: as a "fine-grained" conformance target, it often does not match the target of a conformance clause which usually is a "product" , or a process, or a service (W3C QA framework).

For example, a TA may target a "message", or a "signature", or a UDD entry, like in WS-I profiles. But the conformance clause will target the Web service instance that generates this message, or the SOAP stack, or the entire UDDI instance.

As for levels of conformance, etc: I believe these have nothing to do with the basic model of a TA. They can always be added in the context of "how this TA must be used". E.g. a conformance clause will define conformance levels, and may always say: "TA 123 applies level 3 and above". This does not affect the definition of the TA itself, w/r to the addressed spec requirement. </JD>

 

• Define a Test Outcome.

The test outcome is defined by stating some condition(s) under which an IUT can be declared as adhering to (or fulfilling, or consistent with) the addressed specification requirement, or as incompatible (inconsistent) with it.

There are two types of test assertion outcomes (or test outcomes):

• Pass: the condition for adherence is satisfied (or is “true”)

• Fail: either the condition for adherence is “false”, or a condition defined for incompatibility only is satisfied (or “true”)

Do we need to include the case where a test assertion does not apply?  Should we say “where” it applies?  – Jacques is fine with the change. 

<JD> "Pass: the condition for adherence is satisfied (or is “true”), assuming this test assertion applies.

"Fail: either....or a condition for incompatibility is satisfied (or is “true”), assuming this test assertion applies.

("Applying"  having to be defined earlier as the pre-condition being satisfied (if any) as well as pre-requisites.) </JD> 

Notes:

• A test assertion may specify condition(s) for Pass alone, or for Fail alone, or for both.

• If the adherence condition associated with a pass outcome is not satisfied for an IUT (i.e. it evaluates to “false”) this usually causes by default a Fail outcome, but it is recommended to explicitly state so in the TA.

• Some TAs are intended for “negative testing” only, i.e. they only detect incompatibility of an IUT to the specification requirement, not its adherence. Such a TA – called a “negative TA” - would only define a Fail outcome, along with an incompatibility condition. If this incompatibility condition is not satisfied (i.e. it evaluates to “false”) one cannot conclude to a Pass outcome.

Negative testing needs more clarification.

• It could be that for some IUT, the test assertion is inconclusive. This simply means that neither Pass nor Fail can be asserted for this IUT. No specific condition needs be associated with an inconclusive outcome.

Change "inconclusive" to "inconclusive output"

In addition, a TA may also:

• Define a Test Flow

Everything needed to evaluate test outcome

This seems more applicable to a test not a test assertion. Jacques agrees that test flow can be removed.

What about test outcome?  Lynn/Paul/Patrick/others think this is too close to the test itself. 

<JD>  In all the TAs I have seen used for OASIS specs, WS-I specs and others, there are always clear indications in the TA on what condition (from a testing perspective) must be met by the IUT to be considered as fulfilling the spec requirement, i.e. what "pass"  means (and more rarely, what specific conditions must be met for the IUT to be considered as failing to comply). It seems to me that people expect this, and especially the folks who write the test cases. If there is something that could be confused with the test itself, this is rather the "test flow" which describes the test process even if abstractly.

The "test outcome" does not have to be expressed using detailed test metadata. Now, if we find it difficult to express a test outcome at a level more abstract than test case, this could be an indication that in fact, there is no need to write a TA for this spec requirement ! (TAs may not be useful in all cases) </JD>

In some cases, the conditional expressions associated with the Test Outcome would only make sense within a particular context of actions or events involving the IUT, that need to occur as part of the “testing”. This sequence of actions or events only needs to be described in an abstract way, with only the level of detail necessary to express the Test Outcome conditions.

Example:

A messaging specification requires that: “…an Error message caused by the reception of a message that is not well-formed always refers to the faulty message using its MessageID.”

A test assertion addressing this requirement may be defined as:

• Test Assertion Id: ErrorTA-123

• Specification Reference: section 3.2.1.2

• Item Under Test: Message service handler (or MSH)

• Test Outcome:

  • Pass if: (An Error message is send out by MSH in response to receiving an ill-formed message ‘M’) AND (The ReferenceId field of the Error message has same value as the MessageId of M).

  • Fail if: (An Error message is send out by MSH in response to receiving an ill-formed message ‘M’) AND (The ReferenceId field of the Error message does NOT match the MessageId of M).

Notes:

• The ITU is here defined as the Message Service Handler, not the Error message itself. This is an arbitrary choice that reveals some intent on the future use of this test assertion. In the present case, the conformance target will be the MSH: the TA will be used to derive a test case intended to verify the conformance of a Message Handler to the specification.

• This TA is only supposed to address the correctness of the Error message w/r to referencing the faulty message. It is NOT supposed to test if an Error message is generated in the first place – this is another specification requirement. Therefore, the Test Outcome is worded so that if an Error message is not generated, the TA outcome is NOT Fail, but is just inconclusive (neither Pass or Fail).

The above TA does not distinguish very well what actions or events need to occur prior to evaluating the Test Outcome. Some parts of the Test Outcome expression are about the test logistics (ensuring the proper message flow occurs), and some parts are about verifying the specification requirement.

It would be better to separate both. This would give to the test expert in charge of deriving a test case from this TA a clearer indication of what actions need be initiated by the test environment in order to obtain a meaningful outcome. In our example, it is clear that if nothing is done to get an “ill-formed message M” to be received by the MSH, the Test Outcome will be meaningless (actually neither Pass nor Fail).

This sequence of actions may be specified as the Test Flow that specifies the context for the evaluation of the Test Outcome, now simplified:

• Test Flow:

  • A message M with an envelope header referring to a wrong namespace URL is sent to the MSH under test.

  • An Error message is generated in response.

What does the test flow add? Doesn't it just say how to test? Also, is the test flow “limiting” the assertion.

Spec text can break out into 2 assertions - “derived” assertions.

Disagreement concerning the inclusion of test flow in the assertion. 

<JD> Test flow is introduced as "optional" - does not need to be used. But it could be seen as the behavior / action part, in the "statement of behavior, action, or condition that can be measured or tested" as in the W3C definition of a TA. E.g., when describing an expected behavior / action / condition for an item under test, this often is conditional to some prior behavior / action of the test environment. For example in our present case, regarding the requirement for generating an Error message when receiving a faulty message, the TA will state the expected behavior of generating an Error, but must also describe under which condition (here reception of a bad message). That certainly suggests a sequence of actions,  whether we call it "test flow" or not. </JD>

 

 Test Outcome:

• • Pass if: (The ReferenceId field of the Error message has same value as the MessageId of M).

  • Fail if: (Pass condition is false).

Notes:

• Details on how the behavior or condition described in the Test Flow may be met is out of scope of the TA definition – e.g. in the above example a test environment may take action to generate the faulty message, or on the contrary a monitoring device may just wait for such a message to be exchanged. These are left for the test harness or test suite implementation to decide.

• The Test Flow may help narrow the scope of the test cases that will be derived from the test assertion, which in turn will reduce variability in the outcome of different test suites derived from the same test assertions. In our example, the kind of fault used is specified: wrong namespace.

The above TA (ErrorTA-123) will still have an inconclusive outcome if no Error message is generated: the requirements of the Test Flow would not be met.

Patrick views this as a failure.

Confusion over "inconclusiveness". 

<JD> Agree that this should not be seen as inconclusive outcome for ErrorTA-123. If the generation of an error message was a requirement, either it is covered by TA-123 (in which case no Error means a fail outcome) or there should be another TA (say 234)  for it - in which case 234 would be a pre-requisite to 123, and if 234 fails then 123 does not even apply. </JD>

In order to avoid such an outcome, we could add a pre-requisite to this TA, which would be another TA (say ErrorTA-000) addressing the requirement that an Error message must be generated when a faulty message is received by an MSH. In that case, ErrorTA-123 would only be exercised if its pre-requisite (ErrorTA-000) had a positive outcome. Concretely, this will help execute the test cases related to ErrorTA-123 and ErrorTA-000 in the right order, so that ErrorTA-123 is never exercised uselessly in case the test case for ErrorTA-000 fails on the same IUT.

<snip>