[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: tag-wiki-2008-09-17
TA Guidelines - wiki with comments below/attached Best regards -- Stephen D. Green Partner SystML, http://www.systml.co.uk Tel: +44 (0) 117 9541606 Associate Director Document Engineering Services http://www.documentengineeringservices.com http://www.biblegateway.com/passage/?search=matthew+22:37 .. and voice
= Test Assertion Guidelines =
'''''Latest output document (draft)'':'''[http://www.oasis-open.org/apps/group_public/download.php/29080/TestAssertionsGuidelines-draft-0-9-9.1.doc Test Assertions Guidelines]
__'''Ongoing Editorial and TC Group Updates (underlined):'''__
[[OASIS Logo]]
'''Test Assertions Guidelines'''
'''Draft 0.x'''
'''2008'''
'''This Version:'''
'''Previous Version:'''
'''Latest Version:'''
'''Latest Approved Version:'''
N/A
'''Technical Committee:'''
OASIS Test Assertions Guidelines (TAG) TC
'''Chair(s):'''
Patrick Curran, Sun Microsystems
Jacques Durand, Fujitsu
'''Editor(s):'''
Stephen Green, SystML
'''Contributor(s):'''
David Marston, IBM Research
David Pawson, Royal National Institute for the Blind
Hyunbo Cho, Pohang University
Kevin Looney, Sun Microsystems
Kyoung-Rog Yi, KIEC
Lynne Rosenthal, NIST
Paul Rank, Sun Microsystems
Serm Kulvatunyou, NIST
Tim Boland, NIST
Victor Rudometov, Sun Microsystems
Youngkon Lee, Korea TAG forum
'''Abstract:'''
This documents provides guidelines and best practices for writing test assertions along with mandatory and optional components of a test assertion model.
'''Status:'''
This document was last revised or approved by the Test Assertions Guidelines Technical Committee on the above date. The level of approval is also listed above. Check the "Latest Version" or "Latest Approved Version" location noted above for possible later revisions of this document.
Technical Committee members should send comments on this document to the Technical Committee’s email list. Others should send comments to the Technical Committee by using the “Send A Comment” button on the Technical Committee’s web page at [http://www.oasis-open.org/committeees/tag/ http://www.oasis-open.org/committees/tag/].
For information on whether any patents have been disclosed that may be essential to implementing this document, and any offers of patent licensing terms, please refer to the Intellectual Property Rights section of the Technical Committee web page (http://www.oasis-open.org/committees/tag/ipr.php).
The non-normative errata page for this document is located at http://www.oasis-open.org/committees/tag/.
.
'''Notices'''
Copyright © OASIS® 2007-2008. All Rights Reserved.
All capitalized terms in the following text have the meanings assigned to them in the OASIS Intellectual Property Rights Policy (the "OASIS IPR Policy"). The full Policy may be found at the OASIS website.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published, and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this section are included on all such copies and derivative works. However, this document itself may not be modified in any way, including by removing the copyright notice or references to OASIS, except as needed for the purpose of developing any document or deliverable produced by an OASIS Technical Committee (in which case the rules applicable to copyrights, as set forth in the OASIS IPR Policy, must be followed) or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by OASIS or its successors or assigns.
This document and the information contained herein is provided on an "AS IS" basis and OASIS DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY OWNERSHIP RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
= Introduction =
This document is a guide to test assertions. Its purpose is to help the reader understand what test assertions are, their benefits, and most importantly, how they are created. As you will discover, test assertions can be an important and useful tool in promoting quality of specifications, test suites and implementations of specifications. You will learn that there are many ways to create test assertions.
By following the guidance in this document, you are more likely to develop well-defined test assertions that can have many useful purposes and applications (for example, as the starting point for a conformance test suite for a specification). Experiences in developing test assertions will be shared, along with lessons learned, helpful tricks and tools, hazards to avoid, and other snippets of knowledge that may be helpful in crafting test assertions.
Organization of the document:
'''Section 1''' the rationale for test assertions
'''Section 2''' describes basic design principles
'''Section 3''' advanced features related to test assertions
Appendices provide a glossary of important terms, references and a listing of prior art.
The document is structured to accommodate different levels of interest:
Basic understanding of test assertions, '''Sections 1 and 2'''
Developers and users of test assertions should read '''Section 3'''.
== Scope ==
These guidelines are intended to apply to any technology or business field but some of the text may apply more to the technical field of software engineering which is the primary focus. The examples describe an arbitrary mechanical device so as to ensure a general understanding whatever the background of the reader. This document is limited to the essentials of test assertions with an expectation that a further document will follow to cover matters in greater depth and detail.
= 1 Rationale =
== 1.1 What is a Test Assertion? ==
A test assertion is a statement of behavior, action or condition that
can be measured or tested.
* [jacques] ''I believe the two following sentences are making a point that is hardly understandable by the reader so early in the guideline. This point should be stated later on, when detailing the Predicate and considerations about its testability.''
Judging whether the test assertion is testable may require some knowledge about testing capabilities and
resource constraints. Sometimes the primary factor is conformance
testing, in which case closeness of adherence of the test assertions
to corresponding specification statements may be preferred over
judgements about testability. In some cases the focus is on supporting
a known test harness and testing technology and here the imprtance of
testability may be the uppermost concern.
In either case though there
is always a need to make explicit the relationship between a test
assertion and the precise part of the specification to which it
applies.
The specification will often have a clause called a conformance clause
which identifies those parts of the specification to which adherence
is required for an implementation to be said to be conformant. Testing
such conformance entails matching test results to specification
statements for which there is directly or indirectly a conformance
requirement. Test assertions sit between the specification and
conformance clauses and any tests to be conducted to determine
conformance or otherwise. The test assertion is not the same as a
conformance clause as such. Test assertions are sometimes defined
prior to the completion of the specification and may even then be
refered to in the wording of the conformance clause to make it clear
exactly what conformance will entail. Sometimes test assertions are
authored after both specification and conformance clauses have been
finalised.
Test cases may be accompanied by metadata (see Glossary, Appendix A).
The metadata for each case could tie its case to either specification
statements or test assertions. When a suite has both test assertions
and test case metadata, the assertions can contain some information
(such as specification references) that would otherwise have to be in
the metadata.
== 1.2 Benefits of Test Assertions ==
=== Improving the Specification ===
Test assertions may help to provide for a tighter specification: Any ambiguities, contradictions and statements which require unacceptable resources for testing would be noted as they become apparent during test assertion creation. If there is still opportunity to correct or improve the specification, these notes can be the basis of comments to the specification authors. If not developed by the specification authors, test assertions should be reviewed and approved by them. This improves quality and time-to-deployment of the specification so best results are achieved when assertions are developed in parallel with the specification.
=== Facilitating Conformance Testing ===
* [jacques] ''Often, Conformance testing is contrasted with Interoperability testing. I would not exclude Interop testing from the scope of this guideline. Could we just say in this subtitle: Facilitating Testing? ''
Test assertions provide a starting point for writing a conformance test suite for a specification that can be used during implementation. They simplify the distribution of the test development effort between different organizations while maintaining consistent test quality. By tying test output to specification statements, test assertions improve confidence in the resulting test and provide a basis for coverage analysis (estimating the extent to which the specification is tested).
= 2 Designing a Simple Test Assertion =
This section aims to cover the simpler aspects of test assertions. Some of the more complex aspects are covered later in Section 3.
== 2.1 The Structure of a Test Assertion ==
Some of the elements which comprise a test assertion are considered core while others are optional.
== Core Test Assertion Parts ==
A test assertion must include, implicitly or explicitly:
==== Test Assertion Identifier (TA id) ====
This unique identifier facilitates tools development and the mapping of assertions to specification statements. It is recommended that the identifier be made universally unique. One way to do this is to designate a universally unique namespace for a set of test assertions and to include this namespace along with the identifier when referencing the test assertion from outside of this set.
==== Normative Source(s) ====
These identify the precise specification requirements or normative statements that the test assertion addresses. The specification requirements may be copied, either literally or by deriving a reworded definition, in addition to the references to the locations of the sources in the specifications.
==== Test Assertion Target ====
Such a target categorizes an implementation or a part of an implementation of the referred specification.
==== Predicate ====
A predicate asserts, in the form of an expression, the feature (a behavior or a property) described in the referred specification statement(s). If the predicate is an expression which evaluates to “true” over the test assertion target, this means that the target exhibits this feature. “False” means the target does not exhibit this feature.
==== Prescription Level ====
The test assertion defines a normative statement which may be '''mandatory''' (MUST/REQUIRED), '''not permitted '''(MUST NOT), '''permitted''' (MAY/OPTIONAL) '''preferred''' (SHOULD/RECOMMENDED) or '''not recommended''' (SHOULD NOT). This property can be termed its prescription level.
== Optional Test Assertion Parts ==
In addition, a test assertion may, optionally, include:
==== Prerequisite(s) ====
* [jacques] '' suggestion: reword to emphasize the logical expression aspect ("conditional" suggest "if...then..." which is not what the Prerequisite is in itself) : "A test assertion prerequisite is a logical expression which further qualifies the applicability of the test assertion to the test assertion target. It may include references to other test assertions. It evaluates to true or false ..." ''
These are either references to other test assertions or conditional expressions which further qualify the applicability of the test assertion to the test assertion target. They evaluate to true or false and if false then the assertion is to be considered 'not applicable', meaning that the test assertion is not relevant to its target.
==== Tag(s) ====
Test assertions may be assigned 'tags' or 'keywords' which allow their grouping and give an opportunity to categorize the targets.
== 2.2 Best Practices ==
In an actual test assertion definition, the above properties are often explicitly represented as elements of the test assertion. For example:
Consider the following as a requirement from a specification on “widgets” (we will build on this example throughout these guidelines):
[requirement 100] “A widget MUST be of rectangular shape”.
Here is a test assertion addressing this requirement:
__TA id__: widget-TA100-1
__Target__: widget
__Normative Source__: “widget specification”, requirement 100
__Predicate__: [the widget] is of rectangular shape
__Prescription Level__: mandatory
The assertion predicate is worded as an assertion, not as a requirement (the 'MUST' keyword is absent from the predicate but reflected in the prescription level). It has a clear Boolean value: either the statement is true, or it is false for a particular target. The case of how to write a predicate for specification statements that convey optionality (for example, using keywords SHOULD, MAY, etc.) is examined later.
Note that a concrete representation of a test assertion may omit some of these elements provided they are implicit, as discussed later.
=== 2.2.1 Granularity of Test Assertions ===
Consider now the following statement in the widget specification:
[requirement 101] “A widget of medium size MUST use exactly one AA battery encased in a battery holder.”
There are actually two requirements here that can be tested separately:
. (requirement 101, part 1) A medium-size widget MUST use exactly one AA battery.
. (requirement 101, part 2) A medium-size widget MUST have a battery holder encasing the battery.
Because of this it is possible to write two test assertions:
__TA id__: widget-TA100-2
__Target__: medium-size widget
__Normative Source__: specification requirement 101, part 1
__Predicate__: [the widget] uses exactly one AA battery.
__Prescription Level__: mandatory
.
__TA id__: widget-TA100-3
__Target__: medium-size widget
__Normative Source__: specification requirement 101, part 2
__Predicate__: [the widget] has a battery holder encasing the battery.
__Prescription Level__: mandatory
The granularity of a test assertion is a matter of judgement. A single test assertion instead of two could have been written here, with the predicate: “[the widget] uses exactly one AA battery AND has a battery holder encasing the battery”. This choice may later have an impact on the outcome of a test suite written for verifying the conformance of widgets. With a single test assertion, a test case derived from this test assertion will not be expected to distinguish between the two failure cases. Using two test assertions - one for each sub-requirement - will ensure that a test suite can assess and report independently about the fulfillment of each sub-requirement. Other considerations such as the different nature of tests implied, the reuse of a test assertion in different conformance profiles[VAR], may also lead to the adoption of “fine-grained” instead of “coarse-grained” test assertions. Usage considerations will dictate the best choice here.
=== 2.2.2 Implicit Test Assertion Parts ===
It was noted earlier that a concrete representation of a test assertion may omit elements provided they are implicit. A common case of an implicit test assertion component is the implicit target: when several test assertions relate to the same target, the latter may be described just once as part of the context where the test assertions are defined, so that it does not need to be repeated. Example: all test assertions related to requirements about the widget power supply in a widget specification, may be grouped in the section “Widget Power Supply Requirements”, suggesting that they share the same target (the widget power supply).
The predicate may be implicit: In some specifications where all requirements follow a similar pattern, it is often possible to straightforwardly derive the assertion predicate from a requirement - for example, using a simple rule - so that the predicate does not need to be explicitly stated every time. Take, for example, requirement 101, part 1
"A medium-size widget MUST use exactly one AA battery". Compare this text with the predicate of its test assertion, widget-TA100-2, "[the widget] uses exactly one AA battery". There is so much similarity between the requirement text and the test assertion predicate text that an implementation may decide there is too much overhead in writing the predicate to warrant it and decide to merely use a quotation of the requirement in the normative source as an implicit predicate.
=== 2.2.3 Optional Statements: Prescription Level ===
The requirement 101 in the widget example in section 2.2.1 has a mandatory character: it is an absolute requirement using the keyword MUST. Interpreting the outcome of such test assertions is straightforward. Test cases derived from such test assertions can make a clear statement of conformance to the specification for the target under test: '(test assertion predicate = “true”)' means not only that the target exhibits the specified feature of the specification, but also that the target fulfills a specification requirement, since this feature is required.
However there might be several ways to conform to a specification, also known as dimensions of variability. While both conforming, two implementations may not exhibit the same features. This section considers one of the most obvious cases of variability: optional features.
What if the specification statement is optional, i.e. it uses RFC keywords SHOULD / RECOMMENDED or MAY / OPTIONAL?
Examples:
. [statement 102] “It is RECOMMENDED for a widget to be waterproof.”
. [statement 103] “A widget MAY have a metallic casing.”
Such (normative) statements cannot be construed as formal requirements – a widget will not fail to conform to the specification if it is not waterproof, or if it has a plastic casing. However, establishing conformance is not the sole objective of test assertions. Some test suites are intended to evaluate the capabilities of an implementation – for example, which options it implements - regardless of conformance considerations. Even with a conformance objective in mind, a clear separation must be made between:
- (a) Describing under which condition a target can be said to exhibit a specified feature. This is the role of a test assertion.
- (b) Deciding if a target satisfies a conformance criterion. This is the role of one or more test cases that are derived from a test assertion, the outcome of which might be interpreted according to a conformance profile.
Therefore, test assertions can be written for such statements 102 and 103, simply focusing on the specified feature and its related predicate, ignoring the prescription level in the predicate and including it instead as the separate component of that name, for which suitable values are enumerated (such as 'mandatory', 'preferred', 'permitted' and 'undefined'):
__TA id__: widget-TA100-4
__Target__: widget
__Normative Source__: specification statement 102
__Predicate__: [the widget] is waterproof.
__Prescription Level__: preferred
.
__TA id__: widget-TA100-5
__Target__: widget
__Normative Source__: specification statement 103
__Predicate__: [the widget] has a metallic casing.
__Prescription Level__: permitted
= 3 Advanced Features =
We have considered the essential elements of the test assertion: identifier, reference, target, predicate and prescription level. In practice there may be a need for further features to better cater for corner cases such as
* specifications where the normative statements are embedded wholly or partly in tables and diagrams
* specifications which normatively reference other specifications
* specification and test assertions versions
* inheritance and dependencies between specifications
* redundancy of excessively repeated assertions elements
* test assertion targets which are categorized in conformance clauses.
== 3.1 Complex Predicates ==
Recalling the previous example requirement:
[requirement 101] “A widget of medium size MUST use exactly one AA battery encased in a battery holder.”
The target could be defined as “medium-size widget” (as in section 2.2.1) or as just “a widget”. The latter is a natural decision in case our specification requirement uses the wording: “[requirement 101] If a widget is medium size, then it MUST use exactly one AA battery encased in a battery holder.” In this case we would have:
* [jacques]'' to illustrate a complex predicate, shouldn't we address the entire req 101 (not just part 1): predicate = if [the widget] is medium-size, then [the widget] uses exactly one AA battery AND the battery is encased in a battery holder. ''
* [steve]'' in 2.2.1 we argued that there should typically be two TAs for this statement to allow sufficient granularity for testing - to allow testing for battery type and battery holder separately. we'd weaken that point if we merged them here, I think, and we'd conflict somewhat with the final paragraph of section 3.1 where separate testing of battery number/type and of battery holder is implied. ''
__TA id__: widget-TA100-2a
__Target__: widget
__Normative Source__: requirement 101, part 1
__Predicate__: if [the widget] is medium-size, then [the widget] uses exactly one AA battery.
__Prescription Level__: mandatory
The target category is broad, but the predicate part is really of interest only for a subset of this category (the medium-size widgets). Usage considerations should again drive the decision here: a test suite that is designed for verifying all widgets, and does not assume a prior categorization of these in small / medium / large sizes, would be better off with test assertions that only use “widget” as target, such as widget-TA100-2a.
* [jacques] '' suggests to remove the 2 following sentences from this section, and move it in the next section where a comparison is made with the TA version that uses a prerequisite. ''
* [steve] '' I agree in view of new position of prerequisites discussion ''
(Note that from a logical viewpoint, the above predicate value is still “true” for widgets that are NOT medium size, meaning the addressed requirement is technically fulfilled, in spite of its irrelevance.) On the other hand if all targets have been categorized according to their claimed size, then a test case implementing widget-TA100-2a will uselessly repeat the “size” test.
''Note: As an important part of the test assertion, even when the target is implicit, not explicit, it must be that the target for each assertion is clearly identifiable.''
A test assertion predicate may, then, be a Boolean expression - a composition of atomic predicates using logical operators AND, OR, NOT. A test assertion predicate may also be of the kind: “if (condition) then (expression)".
The predicate is worded in an abstract way, still close to the wording of the specification. No indication is given of what kind of test procedure will be used (how to determine the number and type of batteries, etc.), nor of the detailed criteria for the condition evaluation (for example, what kind of battery holder is acceptable.) Such details are normally left to the test cases that can be derived from the above test assertions. These test cases will determine the precise criteria for conforming to the specification. However, if a precise criterion for interpreting the battery holder requirement is provided in an external specification - referred to either directly by the widget specification or by a related conformance clause - then a test assertion must use this criterion in its predicate. Such a test assertion must then refer not only to the specification requirement in its reference property, but also to the external specification or to the conformance clause that refers to this specification.
== 3.2 Prerequisites ==
* [jacques] '' suggest to move the core of "prerequisite" section here, while leaving the more advanced case of prerequisites (with references to other test assertions) after the "Test Assertions for Properties" section. Reason: now that prerequisites include any kind of precondition - not just refs to other TAs - they are a very common TA feature (e.g. "context" in Schematron, "precondition" in WS-I). Most prerequisites will just be "conditions" that do NOT refer to other TAs. Just after the Complex Predicates section is a good place to introduce simple prerequisites based on the example in 3.1, and to contrast with the "complex predicate" solution. So this would look like this:''
* [steve] '' I agree. The flow then seems clearer with emphasis on the simpler matter of prerequisites coming before the more complex matter of properties. The flow of examples is not disrupted.''
An issue with the previous TA (widget-TA100-2a ) is that it will apply to all widgets, while the specification requirement is obviously of interest only for targets that are medium-size. With widget-TA100-2a , the target predicate will always evaluate to “true” even when the widget is NOT medium-size. Indeed, from a logical viewpoint, a predicate of the form “if <condition> then <property>” will always be true when <condition> is “false”. (The only way such a predicate evaluates to “false”, is when <condition> is “true” and <property> is “false”).
On the other hand if all widgets are categorized according to their claimed size prior to testing, then a test case implementing widget-TA100-2a will uselessly repeat the “size” test. In this situation the TA s written in section 2.2.1 (widget-TA100-2 and widget-TA100-3) are a better choice than widget-TA100-2a.
Assuming that the size of widgets is not a given but is subject to testing, how can we indicate that a preliminary test on the widget size must be done, and that the TA predicate must only apply if the widget is medium-size, meaning that otherwise the TA is considered as “Not Applicable”? This is done by introducing a Prerequisite element in the TA:
__TA id__: widget-TA100-c
__Target__: widget
__Normative Source__: requirement 101
__Prerequisite__: [the widget] is medium-size
__Predicate__: [the widget] uses exactly one AA battery AND has a battery holder encasing the battery
__Prescription Level__: mandatory
The Prerequisite element is a logical expression of same nature as the Predicate, which concerns the same target instance.
The possible outcomes of a test assertion with prerequisite become:
* '''Not Applicable''': If the Prerequisite evaluates to “false”, then the test assertion does not even apply for this target (its outcome can be stated as “Not Applicable”). A test case derived from this test assertion should not even be executed on this target: the result of doing so would be meaningless.
* '''True''': If the Prerequisite evaluates to “true”, and the test assertion Predicate evaluates to “true”, then the target is exhibiting the feature described in the addressed specification requirement.
* '''False''': If the Prerequisite evaluates to “true”, and the test assertion Predicate evaluates to “false”, then the target is NOT exhibiting the feature described in the addressed specification requirement.
== 3.3 Test Assertions for Properties ==
Requirements addressed by test assertions may be related to specific properties of a target.
Let us assume that there are specification requirements that define under which conditions a widget qualifies as “medium-size”. In other words, widgets do not come with a sticker that makes this categorization obvious by announcing small / medium / large.
Instead, the size label is a property that is itself defined in the widget specification and that is subject to verification, like any other normative statement.
In such a case, when writing test assertions, it is not a good idea to consider this property as part of the definition of the target category as we did in widget-TA100-2 and widget-TA100-3, because the category of a widget could not be identified prior to doing any test on this widget.
Assume that the following requirement defines the “medium-size” property:
[requirement 104] “A widget that weighs between 100g and 300g and is from 5 to 15 centimeters long in its longer dimension, is a medium-size widget.”
There is a major distinction between requirement 104 and requirement 101:
* requirement 101 uses “medium-size” as a prerequisite: its predicates only concern widgets that are already established as being medium-size.
* requirement 104 defines how to qualify the target as medium-sized.
The test assertions for requirement 104 can be written as:
__TA id__: widget-TA100-6
__Target__: widget
__Normative Source__: specification requirement 104
__Predicate__: [the widget] weighs between 100g and 300g.
__Prescription Level__: medium-size:mandatory
Note that the "mandatory" prescription level is now relative to the claimed property ("medium-size"): the specification does not mandate all widgets to be "medium-size", but for a widget to claim the "medium-size" property, the predicate must be "true". The prescription intent must clearly be indicated in the Prescription Level element, e.g. by prefixing the level value (here "mandatory") with the property name ("medium-size").
__TA id__: widget-TA100-7
__Target__: widget
__Normative Source__: specification requirement 104
__Predicate__: [the widget] is from 5 to 15 centimeters long in its longer dimension.
__Prescription Level__: medium-size:mandatory
The test assertions widget-TA100-6 and widget-TA100-7 will be used to derive test cases that verify if the property "medium-size" applies to some widget. A "false" outcome for their predicates is an indicator that the medium-size property does not apply. It is not indicative of a violation of the specification itself. Such test assertions are called here "Property test assertions" to distinguish them from test assertions that are used as indicators of conformance to a specification. However, both types of test assertions are designed in the same way, with a predicate that indicates whether or not a target satisfies some feature or property.
There is no mention of the “medium-size” property at all in the predicates of test assertions ‘widget-TA100-6’ and ‘widget-TA100-7’. This is because this property is precisely what needs to be established by a test suite containing test cases that are derived from these test assertions. Only when a target (here a widget) evaluates to “true” for these two test assertions, will it be considered as medium-size. These test assertions are only concerned with the nature of these tests, not with how to interpret their outcome.
== 3.4 Prerequisites Referring to Other Test Assertions ==
* [jacques] if the core of the prerequisite mechanism is introduced in 3.2, then this section only needs to deal with the case of references to other TAs. Replace: '' "Now that the “medium-size” property has its own specification requirements, it is clear that a widget should be tested first for qualifying as "medium-size", before being tested for requirement 101. This ordering will reflect in the related Test Assertions by adding a “prerequisite” component to widget-TA100-2 and widget-TA100-3. These two Test Assertions become: " '' With:
Now that we have a means to establish the “medium-size” property, we can use a more precise Prerequisite element in the Test Assertion for the requirement 101. Because Test Assertions have been written for testing the property “medium-size”, we can refer to these in the Prerequisite:
* [jacques]'' suggests to use a single TA with a predicate that addresses all of the req 101 (easier to read and shorter: the focus is here on teh prerequisite)''
* [steve] '' as mentioned above, I think this would contradict much of what we say previously. I'm not too keen on adding "for the sake of clarity we will combine the test assertions for specification requirement 100" but if it is the only way to clarify the text I'd accept it. Maybe that is the case. ''
__TA id__: widget-TA100-2b
__Target__: widget
__Normative Source__: specification requirement 101
__Prerequisite__: widget-TA100-6 AND widget-TA100-7
__Predicate__: [the widget] uses exactly one AA battery AND has a battery holder encasing the battery.
__Prescription Level__: mandatory
When a prerequisite element is quoting other test assertion(s), as seen above, with the prerequisite: (widget-TA100-6 AND widget-TA100-7), such references must be understood as short for: (widget-TA100-6 outcome = 'true' AND widget-TA100-7 outcome = 'true').
The prerequisite could have been stated as an explicit predicate in widget-TA100-2b and widget-TA100-3a, i.e. widget-TA100-2b could have been rewritten as:
__TA id__: widget-TA100-2c
__Target__: widget
__Normative Source__: specification requirement 101, part 1
__Prerequisite__: [the widget] weights between 100g and 300g AND [the widget] is from 5 to 15 centimeters long in its longer dimension.
__Predicate__: [the widget] uses exactly one AA battery.
__Prescription Level__: mandatory
Here widget-TA100-2c is semantically equivalent to widget-TA100-2b. However, because the notion of “medium-size” is itself specified as a property that is subject to verification and enforcement, it is useful to write test assertions for this property. It is then preferable to reuse such test assertions as prerequisites whenever this property is assumed. If the notion of medium-size evolves in future releases of the widget specification, a test assertion such as widget-TA100-2b does not need to be altered: only its prerequisite test assertion needs to be, while all test assertions that explicitly state the prerequisite predicate - as widget-TA100-2c does - would need be updated.
== 3.4 Various Normative Sources ==
Consider the case in which a test assertion refers to a single normative source such as a single statement in a specification. Here test assertions are linked to precisely identified parts of a specification. A normative source is in this case a reference pointing to part of a single specification. This case is simple enough but things are not usually so simple.
Sometimes the test assertion is itself a part of the specification. Here the specification authors choose the structure of test assertions as one with which to construct their specification document. In this case the test assertions may be embedded within the specification and may themselves be normative sources. So the normative source may be '''implicit''' in the test assertion.
Other kinds of complexity involve normative sources which include tables and diagrams. Here the test assertion may need to include derived normative source material. Care must therefore be taken to ensure that the logic of a test assertion does not deviate from or add to the logic of the normative statements in the specification. Care must also be taken when literally copying text, diagrams or tables from the specification to ensure that divorcing the material from its context, formatting, etc. does not result in subtle changes to its meaning.
In cases where the normative source is either quoted or derived within the test assertion there is the possibility of omitting the predicate since it is already implicit in the text of the normative source.
There may even be occassions when a single test assertion relates to normative statements spread over several parts of a specification. Here there is a need for not one but several references to parts of a specification as the normative source. So the normative source reference has to handle '''multiple occurrence'''.
Not to be forgotten is the fact that multiple assertions might share the same text fragments so the relationship between test assertion and normative source is 'many-to-many'.
== 3.5 Test Assertion Grouping ==
''''[Kevin: Perhaps there should be some 'introductory' text before jumping into this -
eg. "The Grouping of assertions allows a person analyzing a specification a mechanism for annotating a "similar" set of assertions carrying some common characteristic. Typically, grouping can be accomplished either explicitly (via lists, as in 3.5.1), or implicitly (via tags, as in 3.5.2)"]'''';
=== 3.5.1 Lists (Dimensions of Variability) ===
A common requirement is to be able to group a set of test assertions in a tightly controlled way such that there is certainty that the group contains a specified set of test assertions and no more than that set.
''''
[Kevin: I'm not sure that there is any more certainty that an assertion is explicitly in some group - because it explicitly belongs in a list, or implicitly in some group (because of it's tags). I suppose there could be some error where a given assertion has conflicting tags - but likewise, an assertion might incorrectly appear in two lists with opposite meanings.
From what I can see, there really isn't too much difference in the functionality of Lists vs. Tags.
I also don't think "open vs. closed" is a meaningful way to differentiate the two types of lists -
the difference is better described as 'whether the logic used to describe the list is "implicit" or "explicit"'.
A list created from tags is one where the logic is "explicitly stated", yielding a series of TAs when the logic is applied to all TAs in the spec.
eg list = (TAs with <Tag A> && <Tag B> && !<Tag C>)
= [ TA001, TA002, ....., TA00N]'''';
''''
As opposed to an list of (explicitly enumerated) TAs, in which the logic of inclusion is implicit. As such, the logic is "Frozen" into some abstract definition of the list.
eg. list = (all assertions describing 'Size' requirements)
= [ TA001, TA002, ....., TA00N]'''';
''''Perhaps "frozen logic" is what is meant by 'closed' grouping.]'''';
This can be termed 'closed' grouping (as distinct from 'open' grouping facilitated by tagging - see Section 3.5.2). It is a particularly important method of grouping when test assertions are used for defining conformance requirements.
This designation of a set of test assertions may be accompanied by a header construct which allows the declaration of any variables whose scope is defined as comprising this same set (see section 3.8). A special but ubiquitous case of this is with the grouping of a whole set of test assertions into a test assertion document. This may be a crucial part of defining test assertions which are designated to a conformance clause for a specification or conformance profile (as a key aspect of defining dimensions of variability [VAR]).
''''[Kevin: The paragraph above describes a 'test assertion document' - Is this a concept in the glossary? Should it be?
I think there probably is an "<output>" of the specification analysis process, and it's format includes what we are specifying in this document.
Some choices, we could describe this "output" as:
Test assertion document (vague)
Assertion List (this is a term used in Sun, but it also seems vague)
Specification Analysis (this term is more descriptive of the process, and not the output)
??? (other)'''';
''''Perhaps it is not our job to formalize what this "output" is, and it is our job to describe that Test Assertions are the 'key documentation of specification analysis' that must belong in an "<output>" document.]'''';
In some cases the tests may be allowed to inherit shared elements such as prerequisites or variables from the header of the listed group. However there is a need for careful integrity considerations in case a test assertion is listed in more than one list. This is to avoid contradictions between inherited elements.
An important aspect of test assertion listing is version control (see Section 3.7).
''''[Kevin: We could expand on version control a bit - version control could be applied to various levels of granularity.
Here, I think the implication is that a "assertion grouping list" may change independently from the assertions in a spec.
Or perhaps version control is only applied at the topmost level of a "<output>" (specification analysis document, as described above).
In any case, the statement above about the relevance of version control seems ineffective.]'''';
=== 3.5.2 Tags (Test Assertion Metadata) ===
A technique for defining more 'open' groups of test assertions is to use "tags". The groups creating by assigning tags to test assertions can be described as open because the set of test assertions having a particular tag may change as the same tag can be assigned to a newly created test assertion or by changing an existing test assertion to add or remove the same tag.
''''[Kevin: As I observed above, the two types of lists differ in how implicit/explicit the logic is that is used when creating the list.
eg. Closed list:
[ TA001, TA002, ....., TA00N]
(these are individually enumerated TAs, where there is some implicit logic used to determine inclusion
Open list:
list = TAs annotated with (<Tag A> && <Tag B> && !<Tag C>)
= [ TA001, TA002, ....., TA00N]
(in this case, the list is derived from an explicit logical expression applied to all TAs in a spec.)]'''';
A tag is an optional test assertion element that is given a specific name and in some cases (but not always) a value. It allows tests assertions to be subsequently grouped according to the tags or tag combinations, effectively filtering the test assertions into subsets of the whole. Several such filters can be applied to the same set of assertions and any given assertion can appear in more than one grouping. This is not a suitable method when it is necessary to know with certainty the complete set of test assertions which will ever have a particular tag or set of tags: In such cases the better practise is to list the test assertions in the group (see Section 3.5.1).
''''[Kevin: I'm not sure I agree with this statement.
The 'certainty' of the inclusion of a TA within a grouping seems to comes from appying some logic to the spec, producing the group. It doesn't matter so much whether that logic is applied explicitly (and possibly dynamically), or through some predescribed enumeration.
The things that describe certainty (or lack of certainty) of inclusion within a group is:
a.) whether the TA changes
b.) whether the logic describing the grouping changes.
]'''';
== 3.6 The Case of Multiple Specifications ==
Modularity and succinct description within specifications can be achieved by leveraging existing specifications that are referenced by other specifications. Specification writers often create "umbrella specifications". These are widely scoped specifications that delegate certain normative descriptions to other "referenced specifications".
* [jacques]'' Another case of referencing, are "base specifications" that describe general features that are a basis for implementations of more specialized specifications reusing these features. In that case, the specialized specification references the base specification.
For proper specification analysis, inclusion of test assertions from both the umbrella specification and all the referenced specifications should be considered.
Two aspects of analysis of multiplicity of the normative sources are considered:
* Specification Visibility (Section 3.6.1) deals with how assertions from a referenced specification are considered or described in the umbrella specification
* Composition of Assertions (Section 3.6.2) describes how predicates and prerequisites can express various relationships between umbrella and referenced specifications
=== 3.6.1 Specification Visibility ===
Consider a case where a specification is the normative source but it itself refers normatively to a second specification. Consider firstly the case where there is no visibility of any test assertions covering this second specification. Here there may be a prerequisite that all test assertions for the second specification be treated as a single Boolean requirement.
In a second case the specification refers to a second specification for which the test assertions are given visibility and here the test assertions of the first specification may refer to individual or groups of test assertions of the second specification as explicit prerequisites. Of course, any number of normative sources may be involved, some with test assertions visible, others not.
=== 3.6.2 Composition of Assertion ===
There are three dimensions that describe how assertions from a referenced specification may be included within an umbrella specification:
* [jacques]''I'd rather say more generally "how test assertions for an umbrella specification can refer to another specification" or the like. It may be incorrect to say that "how assertions ... be included within an umbrella specification" because TAs are not always "included" in a spec. (or did you mean "how normative statements from... are included within...")''
'scope of inclusion',
'conditionality of inclusion',
'modification of inclusions'.
These relationships between specifications can be expressed using a test assertion. This form of a test assertion is a specific form of an assertion, as it expresses some form of conformance (like a conformance clause).
* [jacques]''what is an "assertion" if not a "test assertion" ? Is that a normative statement?''
Multiple dimensions can be expressed within these relationships (for example, a subset of test assertions from a reference spec may be conditionally included in an umbrella specification).
* [jacques]''same comment as above: this is not so much about "including TAs in a spec" but more like using TAs of one spec within TAs of an umbrella spec.''
''''' Scope of Inclusions '''''
Relationships between the (current) umbrella specification and a referenced specification are usually described via conformance statements.
* [jacques] Because we never used the expression "conformance statement", I'd rather say something like: ''An umbrella specification usually relates to a referenced specification by assuming or requiring conformance of its implementation to this specification''.
These conformance statements can be expressed in a test assertion's prerequisite or predicates.
The scope of this conformance may be determined by the expressions in these prerequisites or predicates.
A predicate may describe the target conforms to varying scopes of the (current) umbrella specification as follows:
* an (entire) umbrella specification
* a conformance profile of the umbrella specification
* a specific normative statement from the umbrella specification
Similarly, a prerequisite may describe the target conforms to varying scopes of the external specification as follows:
* an (entire) referenced specification
* a conformance profile of an referenced specification
* a specific test assertion from an referenced specification
* [jacques] the main issue I have with above terminology, is when we say a "the target conforms to..." in a TA. We have been careful to avoid mentioning the word "conform" in all our TA semantics, because conformance is so overloaded and has a very specific meaning tied to conformance clause, etc. So we always say "a target fulfills a normative statement" or "exhibits a normative feature" or so. This criticism is concerning the third bullet(s) above. To avoid confusion, these new special cases (first and second bullets) shoudl be properly introduced. I'd suggest to reword like: ''The logical expressions used in the predicate or the prerequisite of a TA, may also include a conformance requirement to (a) an entire specification or (b) a conformance profile of this specification. As a result, such a TA written for an umbrella specification may express in its predicate and prerequisite:'' (and here, the "TA predicate" with its bullet list, and the "TA prerequisite" with its bullet list, adding "conformance to" before each 2 first bullet items.)
A simple example might be a wholesale inclusion that describes where a target in an umbrella specification is completely conformant to the assertions of a referenced specification.
__TA id__: widget-TA100-8
__Target__: Widget
__Normative Source__: WidgetSpec 1.0
__Prerequisite__: [the widget] is conformant to Mini-Widget Small Box Specification 1.2
__Predicate__: [the widget] is conformant to WidgetSpec 1.0
__Prescription Level__: mandatory
* [jacques] ''What is the normative statement that this TA is trying to address? I thought it would rather be the reverse, something like: "before a widget can be declared conformant to Mini-Widget Small Box Specification, it must be conforming to the general Widget specification." Because obviously, a Mini-Widget is a (special) Widget so all the requirements for Widget automatically apply to Mini-Widget. I think this is more like the pattern (specialized spec / base spec), than umbrella spec ?''
* [jacques] also it seems that to be more precise, the Normative Source element shoudl not be set to "WidgetSpec 1.0", but to "Conformance clause of WidgetSpec 1.0". Because generally the spec itself does not say how to conform to itself as a whole, unless it contains (or is associated with a separate) conformance clause.
Interpretation: "All widgets conformant to the WidgetSpec 1.0 must also be conformant to the Mini-Widget Small Box Specification 1.2."
* [jacques] Is there any difference between this interpretation statement, and the "normative requirement" that the TA should address? If not, this interpretation should be stated as the Normative requirement, and appear before the TA.
Another example might be a partial inclusion that describes a case where a target in an umbrella specification is conformant to some subset of assertions in a referenced specification. Subsets of a specification are described as 'Conformance Profiles', and may be expressed via grouping constructs (using 'lists'):
* [jacques] there may be too little difference betwee TA100-9 and TA100-8 to justify a full blown separate example? It may be sufficient to just mention the variant of Prerequisite that makes the difference.
__TA id__: widget-TA100-9
__Target__: Widget
__Normative Source__: WidgetSpec 1.0
__Prerequisite__: [the widget] is conformant to the "smaller box" Conformance Profile of the Mini-Widget Small Box Specification 1.2.
__Predicate__: [the widget] is conformant to WidgetSpec 1.0
__Prescription Level__: mandatory
Note: "Some portion of the TAs for the Mini-Widget Small Box Specification 1.2 are identified in a list to indicate their inclusion in the Smaller Box conformance profile"
Interpretation: "All widgets conformant to the WidgetSpec 1.0 specification must also be conformant to the 'smaller box' assertions of the WidgetMobile Small Box Specification 1.2"
''''' Conditionality of Inclusions '''''
This dimension of inclusion describes the conditionality whether assertions in an umbrella specification is conformant to a referenced specification. The prerequisite of the assertion may:
* a. require that optional portions of the referenced specification be implemented in the umbrella,
* b. conditionally require optional portions of the referenced specification be implemented in the umbrella (for example, based on the presence of hardware or some other such support), or
* c. make remaining (required) portions of the referenced specification be optional
The example of widget-TA100-8 already shows how a widget describes mandatory conformance. The following widget describes a conditional conformance:
__TA id__: widget-TA100-10
__Target__: Widget
__Normative Source__: specification requirement 120 in WidgetSpec 1.0
__Prerequisite__: IF [the widget] has restriction of box size 760 mm x 480 mm x 100 mm or less THEN [the widget] is conformant to the Conformance Profile of the Mini-Widget Small Box Specification 1.2.
__Predicate__: [the widget] is conformant to WidgetSpec 1.0
__Prescription Level__: mandatory
Interpretation: "All widgets conformant to the WidgetSpec1.0 specification must also be conformant to the assertions of the WidgetMobile Small Box Specification 1.2 IF the widget is smaller (or equal) in size to 760 mm x 480 mm x 100 mm "
''''' Modification of Inclusions '''''
This dimension of inclusion describes where an umbrella specification is conformant to a referenced specification, where some subset of assertions must be modified. This assumes some partitioning of the unchanged assertions and modified assertions. You can use "lists of assertions" to describe in the pre-requisite the subset of assertions that the umbrella specification is conformant to "unchanged". The remaining test assertions (changed set) can be individually specified as test assertions of the umbrella specification.
Typically, assertions are modified in a referenced specification to be strengthened in a few ways:
* strengthening the prescription level of an assertion (eg. x MAY do y => x MUST do y), or
* strengthening the meaning of an assertion with additional requirements (eg. IF x THEN z => IF (x AND y) THEN z).
== 3.7 Specification Versions ==
Where a specification is the basis for test assertions there needs to be consideration of how to support further versions and maybe any previous versions of that specification. There may need to be a version strategy for the test assertions. The references to specifications may include the identification of the precise specification version but this may restrict that test assertion to just one version of a specification. Such a simple strategy is less than ideal as it results in a need to re-author the test assertions each time there is a new specification version.
== 3.8 Variables and Target Properties ==
* ''[jacques] We can probably keep just "Variables" in the title.''
Variables have a similar role in test assertions as in other technologies.
They provide a means for consistently sharing values across multiple
assertions and with other processes. The writer of a set of test assertions can use a
variable to assert that all occurrences of that variable must have the
same value, even if that value cannot be known at the time the test assertions are
written. For example, consider a set of assertions all sharing a reference
to the line (mains) voltage supplied by the local electric utility, where
the specific locality will vary. By declaring a variable for this value
and distinguishing the name by, for example, representing it in upper case
(UTILITY-VOLTAGE), each test assertion can use it within its elements and
expressions.
Note: Some people prefer the term "parameter" and would say that
the test assertions in question are "parameterized."
The following example also shows the use of the variable within the
predicate. Here, specification requirement 104 depends on the utility
voltage.
__Variable__: 'UTILITY-VOLTAGE' the voltage commonly provided for hand-held electrical appliances and laptop computers.
__TA id__: widget-TA100-6a
__Target__: widget
__Normative Source__: specification requirement 104
__Predicate__: [the widget] can operate when supplied UTILITY-VOLTAGE via its power cord.
__Prescription Level__: mandatory
A value would then be assigned to 'UTILITY-VOLTAGE' prior to testing.
* [''jacques] ''How about a more meaningful TA from a spec point of view:''
__Variable__: 'UTILITY-VOLTAGE' '''the AC voltage''' commonly provided for hand-held electrical appliances and laptop computers.
__TA id__: widget-TA100-6a
__Target__: '''electrical widget'''
__Normative Source__: specification requirement 104
__Predicate__: '''[the electrical widget] contains an embedded AC adapter for the UTILITY-VOLTAGE'''.
__Prescription Level__: mandatory
* ''[jacques] I would then follow right away with a prerequisite extension (illustrating whats being argued two paragraphs later):''
* [jacques]'''Variables can be used across elements in a single test assertion, e.g. in the prerequisite (see section 3.3) and other parts of a test assertion. For example, requirement 104 may be restricted only to widgets that have a compliance requirement for this voltage:'''
__Prerequisite__: '''[the electrical widget] is compliant with UTILITY-VOLTAGE'''
* ''[jacques] Not sure we need the folllowing paragraph on a rather advanced usage that actually calls for comparison with other solutions (e.g. voltage, line frequency, language might actually be seen as functions, returning specific values when applied to LOCALE variable). Would at the very least move that paragraph at the end of the section. If keep, I'd rather say that the var can be used in a function, e.g. voltage(LOCALE)''
A more realistic extension of the above example is to have one variable
for the geo-political locality of an implementation, along with a
collection of dependent variables for voltage, line frequency, language,
roadway dimensions, and all other values that vary by physical location.
The mapping from the externally-derived variable (LOCALE) to the other
variables can be through bundles of logic statements, lookup tables, or
other means. That way, the widget that has some values from one country
and some from a different country, and is therefore not usable in any
country, is detected by testing.
* ''[jacques] The following paragraph would become redundant with a previous illustration of variables in prereqs as suggested above.''
Variables can be used in prerequisites (see section 3.3) and other parts
of a test assertion. A variable may be used in one element in one test assertion and in a
different element in another test assertion in the same collection. For example, the
widget-TA100-6a assertion above might be in a collection along with a test assertion
whose prerequisite restricts it to locations where LINE-VOLTAGE exceeds
250.
* ''[jacques] The following paragraph is good - keep as is.''
In some cases, the variable has a value known or assigned during test
assertion authoring, and is simply used to allow agile resetting of its
value. In other cases, the variable can be declared in name only, leaving
the value to be assigned at a subsequent stage, such as in an
implementation itself or in a conformance clause for a profile, level or
module. The value might be measured during testing and could be associated
with a property (see section 3.2). For example, if the medium-size
property is true for a widget, the SIZE variable is set to the value
"medium".
* ''[jacques] The following sentence is moved in previous example of vars in prereqs.''
Variables can be used across elements in a single test assertion.
* ''[jacques] The following paragraph is good - keep as is.''
A particular consideration in using variables is variable scope. For
example, a grouping construct might be a place to declare variables whose
scope only applies to those test assertions associated with that grouping.
This allows the same variable and its value to be used across several test
assertions while avoiding problems with name clashes in test assertions
outside of the variable's scope.
=== 3.9 Target Categories and their Dependencies ===
* [jacques] '' this subsection has been entirely rewritten as a new proposal based on F2F comments: ''
As mentioned in section 2.1, the Target element of a test assertion generally defines a category of objects or parts of an implementation under test. For example, the test assertion target “widget” represents any object that qualifies as widget.
It is often the case that different Targets (or categories) are related, for example one target is a subcategory of another target . As a consequence:
* Two test assertions that apparently have different targets, will in fact apply to the same implementation (or part of).
* A test assertion may be used as a prerequisite of another test assertion, in spite of having a different Target definition.
For these reasons such target dependencies should be explicitly stated. Such dependencies may be defined outside test assertions, e.g. in an object-oriented model. But often it may be convenient to remind of them in the test assertion itself.
Example: consider a Target named "electrical-gizmo" that is a subcategory of the "widget" Target. This means all test assertions for widget also apply to electrical-gizmo. One way to express this dependency is to use tags. In the following example, a tag named "target-isa" will remind the reader of the TA that the electrical-gizmo target is (also) a widget target:
__TA id__: gizmo-TA300
__Target__: electrical-gizmo
__Normative Source__: specification requirement 300
__Prerequisite__: [The gizmo] has a low-battery indicator.
__Predicate__: The low-battery indicator of [the gizmo] is a red LED that is flashing below 25% charge.
__Prescription Level__: mandatory
__Tag__: target-isa = widget
Another way is to use the prefix notation widget:electrical-gizmo instead of just electrical-gizmo for the target element in a TA:
__TA id__: gizmo-TA300
__Target__: widget:electrical-gizmo
Both modes of annotation (tag or target prefix) make it clear that an electrical-gizmo is also a widget. This will help the grouping of test assertions based on which target they apply to. It also helps writing prerequisites that refer to other test assertions. For example, knowing that the electrical-gizmo is also a widget, and assuming there is already a test assertion (widget-TA103) written for:
* [statement 103] “A widget MAY have a low-battery indicator.”
Then it is possible to reuse the TA for Req 103 as a prerequisite:
__TA id__: gizmo-TA300
...
__Prerequisite__: widget-TA103.
NOTE: the TA with the most general category (here "widget") can be a prerequisite for the TA with a target subcategory (here electrical-gizmo ) but the reverse is not correct.
Another kind of target dependencies is the Composition relationship: Target T2 is a component of Target T1 if an instance of T1 contains an instance of T2. For example, assuming widgets always have at least one switch to control their operation, Target “switch” is a component of Target “widget”. Knowledge of this target composition relationship brings the same benefits as for subcategories:
* grouping all test assertions that apply to widgets often should include the test assertions that apply to widget components.
* a test assertion on widget may use in its prerequisite a test assertion on the switch that is part of this widget. For example, addressing the requirement "the widget must stop when the switch is off" may use as prerequisite a test assertion that addresses the requirement "A switch must cut its electrical circuit when in OFF position". This would guarantee that the switch is tested first, in related test programs.
= Appendix A. Glossary =
=== Conformance ===
. The fulfillment of specified requirements by a product, document, process, or service.
=== Conformance Clause ===
. A statement in the Conformance section of a specification that provides a high-level description of what is required for an artifact to conform. It, in turn, refers to other parts of the specification for details. A conformance clause must reference one or more normative statements, directly or indirectly, and may refer to another conformance clause.
=== Implementation ===
. A product, document, process, or service that is the realization of a specification or part of a specification.
=== Normative Source(s) ===
. Identify the precise specification requirements or normative statements that the test assertion addresses. (See also Section 2.1.)
=== Normative Statement, or Normative Requirement ===
. A statement made in the body of a specification that defines prescriptive requirements on a conformance target.
=== Predicate ===
. A predicate asserts, in the form of an expression, the feature (a behavior or a property) described in the referred specification statement(s). If the predicate is an expression which evaluates to “true” over the test assertion target, this means that the target exhibits the feature. “False” means the target does not exhibit the feature.
=== Prerequisite ===
. These are either references to other test assertions or conditional expressions which further qualify the applicability of the test assertion to the test assertion target. They evaluate to true or false and if false then the assertion is to be considered 'not applicable', meaning that the test assertion is not relevant to its target.
=== Prescription Level ===
. The test assertion defines a normative statement which may be '''mandatory''' (MUST/REQUIRED), '''not permitted '''(MUST NOT), '''permitted''' (MAY/OPTIONAL) '''preferred''' (SHOULD/RECOMMENDED) or '''not recommended''' (SHOULD NOT). This property can be termed its prescription level.
=== Tag ===
. Test assertions may be assigned 'tags' or 'keywords' which allow their grouping and give an opportunity to categorize the targets.
=== Test Assertion ===
. A testable expression for evaluating adherence of part of an implementation to a normative requirement statement in a specification. It describes the expected output or behavior for the test assertion target within specific operation conditions, in a way that can be measured or tested.
=== Test Assertion Identifier ===
. This is a unique identifier of a test assertion. (See also Section 2.1.)
=== Test Assertion Target ===
. Implementation or part of an implementation that can be the object of a test assertion or test case.
(See also Section 2.1.)
=== Test Case ===
. Consists of a set of a test tools, software or files (data, programs, scripts, or instructions for manual operations) that verifies the adherence of a test assertion target to one or more normative statements in the specification. Typically, a test case is derived from one or more test assertions. Each test case includes: (1) a description of the test purpose (what is being tested - the conditions / requirements / capabilities which are to be addressed by a particular test), (2) the pass/fail criteria, (3) traceability information to the verified normative statement(s), either as a reference to a test assertion, or as a direct reference to the normative statement.
=== Test Metadata ===
. Test metadata is metadata included in test cases to facilitate automation and other processing.
=== Variable ===
. Employed by the writer of a test assertion to refer to a value that is not known at the time the test assertion is written, but will be determined at some later stage, possibly as late as the middle of running a set of tests. It is also employed to enable several assertions to share a value (set once, used by many), like a variable in other technologies.
= Appendix B. References =
[CONF1] Conformance requirements for Specifications (OASIS, March 2002)
see http://www.oasis-open.org/committees/download.php/305/conformance_requirements-v1.pdf
.
[CONF2] Conformance testing and Certification Framework (OASIS, Conformance TC, June 2001)
see http://www.oasis-open.org/committees/download.php/309/testing_and_certification_framework.pdf
.
[TD] Test Development FAQ, WG note (W3C, 2005)
see http://www.w3.org/QA/WG/2005/01/test-faq
.
[VAR] Variability in Specifications, WG note (W3C, 2005)
see http://www.w3.org/TR/2005/NOTE-spec-variability-20050831/
.
[TMD] Test Metadata, QA Interest Group note, (W3C, September 2005)
see http://www.w3.org/TR/2005/NOTE-test-metadata-20050914/
.
= Appendix C. Prior Art =
Numerous prior art examples exist for the structure and usage of test assertions.
Voting Systems Standard (EAC), (2007)
http://www.eac.gov/files/vvsg/Final-TGDC-VVSG-08312007.pdf
The use of targets and their categorization using classes and sub-classes was noted. Object oriented classification is an interesting technique for categorising test assertion targets, though it was considered that the treatment of target categorization in the present guidelines should take a broader scope.
Unisoft have published their own Glossary of Assertion Based Testing Terms, again for POSIX
http://www.unisoft.com/glossary.html
[See also DejaGnu Testing Framework below.]
The DejaGnu Testing Framework - A POSIX conforming test framework – is based on a use of test assertions
http://www.delorie.com/gnu/docs/dejagnu/dejagnu_6.html
This builds on the POSIX assertions definitions and of particular note is the analysis of outcome interpretations. The present guidelines do not give extensive coverage to this because it is considered to fit better with the domain of test suites where outcomes can be related to the knowledge of testing methods to be used. However, it is noted that test assertions may be used by some as a means of generating or authoring test suites and in these case special note may need to made of the various possibilities there may be in interpreting outcomes of tests based on a given test assertion.
Similar test assertions developed with a particular testing framework in mind are those developed by the WS-I for their web services profiles. For example for the Basic Profile version 1.1
http://www.ws-i.org/Testing/Tools/2005/01/BP11_TAD_1-1.htm
Similarly W3C publishes test suites based on test assertions for HTML and SOAP specifications:
HTML 4.01 Test Suite - Assertions (2002)
http://www.w3.org/MarkUp/Test/HTML401/current/assertions/assertions_toc.html
SOAP Version 1.2 Specification Assertions and Test Collection (2003)
http://www.w3.org/TR/soap12-testcollection/
The following documents are milestones in the development of the current thinking on test assertions as presented at a basic level in the present guidelines:
A Test Assertion Guideline draft was initiated within the OASIS Technical Advisory Board, (2004-2005)
http://www.oasis-open.org/committees/document.php?document_id=20661&wg_abbrev=ebxml-iic
Work in W3C lead to the QA Framework: Specification Guidelines (W3C, November 2004)
http://www.w3.org/TR/qaframe-spec/
A Test Assertion Guide was later drafted in W3C (W3C Editors' Draft, 2006)
http://www.w3.org/2006/03/test-assertion-guide
Other methodologies for the creation of test assertions or similar artefacts:
Constraint Languages have special relevance when defining test assertions for specifications
involving specification normative statements which make use of standards such as XML Schemas
and UML diagrams.
Object Constraint Language (OCL) and Unified Modeling Language (UML), Object Management Group, Inc.
http://www.omg.org/technology/documents/modeling_spec_catalog.htm#OCL
OCL is a formal constraint language used to define constraints for UML. It can have a special
role in testing based on UML methodologies. It includes methods for defining preconditions and
prerequisites as well as postconditions: There may be some overlap with test assertion
prerequisites. Where a specification includes UML diagrams it may be useful to use OCL
expressions within or in the place of test assertion predicates. Similarly XPath expressions may
play a key role in predicates involving XML. In both of these special cases there may be good
reason to specialize the structure of the test assertion to facilitate use of such expression
languages.
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]