[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Notes from 11/7 meeting - Anatomy Version 06
The following are the notes I took during our review of the "Anatomy of
a Test Assertion" section (V06) during the 11/7 meeting. I've inserted
the notes I took in blue italics.
Paul
Anatomy of a Test Assertion: What is in a Test Assertion: A test assertion (or TA) must always:
Why is this
(IUT) singled out and level/profile/module are not singled out? Should IUT
be in the conformance
clause?
The test outcome is defined by stating some condition(s) under which an IUT can be declared as adhering to (or fulfilling, or consistent with) the addressed specification requirement, or as incompatible (inconsistent) with it. There are two types of test assertion outcomes (or test outcomes):
Do we need to include the case where a test assertion does not apply? Should we say “where” it applies? – Jacques is fine with the change. Notes:
Negative testing needs more clarification.
Change "inconclusive" to "inconclusive output" In addition, a TA may also:
Everything needed to evaluate test outcome This seems more applicable to a test not a test assertion. Jacques agrees that test flow can be removed. What about test outcome? Lynn/Paul/Patrick/others think this is too close to the test itself. In some cases, the conditional expressions associated with the Test Outcome would only make sense within a particular context of actions or events involving the IUT, that need to occur as part of the “testing”. This sequence of actions or events only needs to be described in an abstract way, with only the level of detail necessary to express the Test Outcome conditions.
Example: A messaging specification requires that: “…an Error message caused by the reception of a message that is not well-formed always refers to the faulty message using its MessageID.” A test assertion addressing this requirement may be defined as:
Notes:
The above TA does not distinguish very well what actions or events need to occur prior to evaluating the Test Outcome. Some parts of the Test Outcome expression are about the test logistics (ensuring the proper message flow occurs), and some parts are about verifying the specification requirement. It would be better to separate both. This would give to the test expert in charge of deriving a test case from this TA a clearer indication of what actions need be initiated by the test environment in order to obtain a meaningful outcome. In our example, it is clear that if nothing is done to get an “ill-formed message M” to be received by the MSH, the Test Outcome will be meaningless (actually neither Pass nor Fail). This sequence of actions may be specified as the Test Flow that specifies the context for the evaluation of the Test Outcome, now simplified:
What does the test flow add? Doesn't it just say how to test? Also, is the test flow “limiting” the assertion. Spec text can break out into 2 assertions - “derived” assertions. Disagreement
concerning the inclusion of test flow in the assertion.
Notes:
The above TA (ErrorTA-123) will still have an inconclusive outcome if no Error message is generated: the requirements of the Test Flow would not be met. Patrick views this as a failure. Confusion over "inconclusiveness". In order to avoid such an outcome, we could add a pre-requisite to this TA, which would be another TA (say ErrorTA-000) addressing the requirement that an Error message must be generated when a faulty message is received by an MSH. In that case, ErrorTA-123 would only be exercised if its pre-requisite (ErrorTA-000) had a positive outcome. Concretely, this will help execute the test cases related to ErrorTA-123 and ErrorTA-000 in the right order, so that ErrorTA-123 is never exercised uselessly in case the test case for ErrorTA-000 fails on the same IUT. <snip> |
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]