tag message

Subject: Re: [tag] Groups - Test Assertions Guidelines draft 0.9.1(TestAssertionsGuidelines-draft-0-9-1.doc) uploaded

From: Patrick Curran <Patrick.Curran@Sun.COM>
To: stephen.green@systml.co.uk
Date: Tue, 01 Apr 2008 13:56:43 -0700

I've attached the version of the document that I annotated during our 
first day's discussions. Annotations are in red.

Thanks...

stephen.green@systml.co.uk wrote:
> Many of latest F2F changes included; placeholder text in red
>
>  -- Stephen Green
>
> The document revision named Test Assertions Guidelines draft 0.9.1
> (TestAssertionsGuidelines-draft-0-9-1.doc) has been submitted by Stephen
> Green to the OASIS Test Assertions Guidelines (TAG) TC document repository.
>  This document is revision #12 of TestAssertionsGuidelines-draft-0-2.odt.
>
> Document Description:
> The main output of this committee: Test Assertions Guidelines
>
> View Document Details:
> http://www.oasis-open.org/apps/org/workgroup/tag/document.php?document_id=27810
>
> Download Document:  
> http://www.oasis-open.org/apps/org/workgroup/tag/download.php/27810/TestAssertionsGuidelines-draft-0-9-1.doc
>
> Revision:
> This document is revision #12 of TestAssertionsGuidelines-draft-0-2.odt. 
> The document details page referenced above will show the complete revision
> history.
>
>
> PLEASE NOTE:  If the above links do not work for you, your email application
> may be breaking the link into two pieces.  You may be able to copy and paste
> the entire link address into the address field of your web browser.
>
> -OASIS Open Administration
>

Title: Untitled Document

Test Assertion Guidelines

Latest output document (draft):Test Assertions Guidelines

Ongoing Editorial and TC Group Updates (underlined):

[[OASIS Logo]]

Test Assertions Guidelines

Draft 0.x

2008

This Version:

Previous Version:

Latest Version:

Latest Approved Version:

N/A

Technical Committee:

OASIS Test Assertions Guidelines (TAG) TC

Chair(s):

Patrick Curran, Sun Microsystems

Jacques Durand, Fujitsu

Editor(s):

Stephen Green, SystML

Contributor(s):

David Marston, IBM Research

David Pawson, Royal National Institute for the Blind

Hyunbo Cho, Pohang University

Kevin Looney, Sun Microsystems

Kyoung-Rog Yi, KIEC

Lynne Rosenthal, NIST

Paul Rank, Sun Microsystems

Serm Kulvatunyou, NIST

Tim Boland, NIST

Victor Rudometov, Sun Microsystems

Youngkon Lee, Korea TAG forum

Abstract:

This documents provides guidelines and best practices for writing test assertions along with mandatory and optional components of a test assertion model.

Status:

This document was last revised or approved by the Test Assertions Guidelines Technical Committee on the above date. The level of approval is also listed above. Check the "Latest Version" or "Latest Approved Version" location noted above for possible later revisions of this document.

Technical Committee members should send comments on this document to the Technical Committee’s email list. Others should send comments to the Technical Committee by using the “Send A Comment” button on the Technical Committee’s web page at http://www.oasis-open.org/committees/tag/.

For information on whether any patents have been disclosed that may be essential to implementing this document, and any offers of patent licensing terms, please refer to the Intellectual Property Rights section of the Technical Committee web page (http://www.oasis-open.org/committees/tag/ipr.php).

The non-normative errata page for this document is located at http://www.oasis-open.org/committees/tag/.

Notices

All capitalized terms in the following text have the meanings assigned to them in the OASIS Intellectual Property Rights Policy (the "OASIS IPR Policy"). The full Policy may be found at the OASIS website.

This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published, and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this section are included on all such copies and derivative works. However, this document itself may not be modified in any way, including by removing the copyright notice or references to OASIS, except as needed for the purpose of developing any document or deliverable produced by an OASIS Technical Committee (in which case the rules applicable to copyrights, as set forth in the OASIS IPR Policy, must be followed) or as required to translate it into languages other than English.

The limited permissions granted above are perpetual and will not be revoked by OASIS or its successors or assigns.

This document and the information contained herein is provided on an "AS IS" basis and OASIS DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY OWNERSHIP RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

OASIS requests that any OASIS Party or any other party that believes it has patent claims that would necessarily be infringed by implementations of this OASIS Informational Document, to notify OASIS TC Administrator and provide an indication of its willingness to grant patent licenses to such patent claims in a manner consistent with the IPR Mode of the OASIS Technical Committee that produced this document.

OASIS invites any party to contact the OASIS TC Administrator if it is aware of a claim of ownership of any patent claims that would necessarily be infringed by implementations of this document by a patent holder that is not willing to provide a license to such patent claims in a manner consistent with the IPR Mode of the OASIS Technical Committee that produced this document. OASIS may include such claims on its website, but disclaims any obligation to do so.

OASIS takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Information on OASIS' procedures with respect to rights in any document or deliverable produced by an OASIS Technical Committee can be found on the OASIS website. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this OASIS Informational Document, can be obtained from the OASIS TC Administrator. OASIS makes no representation that any information or list of intellectual property rights will at any time be complete, or that any claims in such list are, in fact, Essential Claims.

The name "OASIS" is a trademarks of OASIS, the owner and developer of this document, and should be used only to refer to the organization and its official outputs. OASIS welcomes reference to, and implementation and use of, specifications, while reserving the right to enforce its marks against misleading uses. Please see http://www.oasis-open.org/who/trademark.php for above guidance.

Introduction

This document is a guide to test assertions. Its purpose is to help the reader understand what are test assertions, their benefits, and most importantly, how they are created. As you will discover, test assertions can be an important and useful tool in promoting quality of specifications, test suites and implementations of the specification. You will learn that there are multiple ways to create test assertions. However, by following the guidance in this document, the likelihood is increased of developing well-defined test assertions that can have many useful purposes and applications (for example, as the starting point for a conformance test suite for a specification). Experiences in developing test assertions will be shared, along with lessons learned, helpful tricks and tools, hazards to avoid, and other tidbits of knowledge that may be helpful in crafting test assertions.Organization of the document section:In Section 1, the rationale for test assertions is presented; followed by Section 2 which describes basic design principles. In Section 3, advanced features related to test assertions will be given. In Section 4, a summary is presented, and in Section 5 a glossary of important terms is given. Appendices provide revision history, references, and a listing of prior art.

The document is structured to accommodate different levels of interest. Those interested in obtaining a basic understanding of test assertions Sections 1 and 2 will be of interest. Developers and users of test assertions who encounter more complex situations and challenges to writing or using test assertions, should read Section 3.

1 Rationale

1.1 What is a Test Assertion?

A test assertion is a statement of behavior, action or condition that can be measured or tested. Each is an independent, testable statement of a normative requirement in a specification to facilitate the creation of a conformance test. Test assertions are derived from the specification and can be mapped to it. In the simplest case, the text of the test assertion may be taken literally from the specification.

[jacques] I know we want to make sure that the option of "not having to write the spec twice" is always there. But this is largely expanded on later in more details. I would advise against stating this early in the guide that "In the simplest case, the text of the test assertion may be taken literally from the specification." With very little context, that will be confusing as we also state later that TAs have mandatory parts. What does "the text of the TA" mean w/r to these parts? The way we have accommodated these two seemingly conflicting requirements, is by stating that these parts can be "implicit", e.g. generically defined for many TAs, or derivable from specification using simple rules (e.g. how to translate a MUST requirement into a predicate). I suggest we leave it to these later sections to reassure the reader s/he wont always have to rewrite things twice.

Test assertions are to be distinguished from test metadata. The metadata is additional data which is useful to form a bridge between the test assertion and conformance tests.

[jacques] Advise to either use "Test Case" instead of "test metadata", or to define "test metadata" in the glossary. Would prefer the former. Also, "conformance tests" is not defined. Can we avoid using this expression in a definition, if not in other parts of the guide?

Test assertions are also to be distinguished from conformance clauses. Conformance clauses take a higher level role and may refer to lists of test assertions. They group together statements about the specification such as which parts are normative or informative. They define how conformance to a specification is broadly to ascertained. They may also detail which parts of the specification must be implemented for certain conformance levels, whether subsets or supersets may be implemented and whether a particular test harness must be used for testing.

[jacques] Thats probably too much details (or repeats) for a forward reference to a concept otherwise defined in the glossary. How about somthing like: "Conformance clauses take a higher level role and may refer to lists of test assertions. See a more detailed definition in the Glossary (section N)."

Test assertions, on the other hand, are typically presented as lists of singular, atomic statements more limited in scope and purpose and occupy a level between a specification with its conformance clauses and the actual tests written to test the conformance of an implementation.

1.2 Benefits of Test Assertions

Test assertions may improve specification quality by helping to uncover inconsistencies, ambiguities, gaps, and non-testable statements in the specification especially around corner cases and exception conditions. This may provide for a tighter specification.

If not developed by the specification authors, test assertions should be reviewed and approved by them. Best results are achieved when assertions are developed in parallel with the specification. This improves quality and time-to-deployment of specifications. On the other hand, writing a set of test assertions can be the best starting point for a testing team to write a test suite for a specification.

Test assertions:

Facilitate and promote the development of conformance tests and testing tools.
Encourage the early development of conformance tests that can be used during implementation.
Simplify the distribution of the test development effort between different organizations while maintaining consistent test quality.
Improve confidence in the resulting tests as a measure of conformance.
- [jacques] this bullet point above is unclear. Isn't that implied by the first bullet? On the other hand, if this bullet list is a general summary of benegits, then I'd add the "improve specification quality when written at same time as the specification".
Enable coverage analysis (estimating the extent to which the specification is tested).

Ed: issue #006

2 Basic Design Principles

Rename this "Developing a simple TA"? 2.1 would be Core structure of a TA, 2.2 would be "Best practices" or "how to"

2.1 The structure of a simple Test Assertion

A test assertion must always explicitly or implicitly:

Be identified by a globally unique test assertion identifier (TA id). This facilitates tools development and the mapping of assertions to specification statements.
Identify the precise specification requirement(s) or normative statement(s) that it addresses.

The identification may be a pointer to a statement in the spec (with as much precision as possible) or the actual text from the spec or both. Note: "reference" is no longer exactly the right term for this.

Identify a test assertion target (or TAT). Such a target is an implementation or a part of an implementation of the referred specification. What is usually defined as a test assertion target is in fact a category of targets of similar type that are all subject to this test assertion.
Define a predicate asserting the feature (a behavior or a property) described in the referred specification statement (s). If the predicate evaluates to “true” over the test assertion target, this means that the target exhibits the feature. “False” means the target does not exhibit the feature.

This may be implicit, particularly if the text from the spec is included in the reference above.

[Editor Note: I suggest that we bring-up the variant of “negative tests” where the “feature” measured by the predicate is in fact a non-conformance case, later in this section, with an appropriate example.]

[jacques] Because of possible variants for the 4th bullet item, suggest to add "In general: " before "If the predicate evaluates to 'true' over ..." See also Issue i020.
F2F: why isn't this example called out in a new section like 2.2 is doing for the most complex example? Resolve this inconsistency.

For example, consider the following requirement from a specification on “widgets”:

[requirement 100] “A widget MUST be of rectangular shape”.

The following test assertion is addressing this requirement:

TA id: widget-TA100

Target: widget

Reference: “widget specification”, requirement 100

Predicate: [the widget] is of rectangular shape

Observation #1: The assertion predicate is worded as an assertion, not as a requirement (the 'MUST' keyword is absent). It has a clear Boolean value: either the statement is true, or it is false for a particular target. The case of how to write a predicate for specification statements that convey optionality (for example, using keywords SHOULD, MAY, etc.) is examined later. "later" = section 4.

[Ed: issue #007 Editorial Note – DM will send contribution to go somewhere around here]

In an actual test assertion definition, the above four properties (numbered 1 to 4) are often explicitly represented as four elements of the test assertion: identifier, reference, target class and predicate, as in our example. Move the preceding two sentences up so that they immediately follow the four bullets? Move the following examples down into the next section that we will turn into a "how to" section?

However, a concrete representation of a test assertion may omit some of these elements provided they are implicit. Some common cases of implicit test assertion components are:

Implicit target: when several test assertions relate to the same target class, the latter may be described just once as part of the context where the test assertions are defined, so that it does not need to be repeated. Example: all test assertions related to requirements about the message envelope in a protocol specification, may be grouped in the section “message envelope requirements”, suggesting that they share the same message envelope target.
Implicit predicate: in some specifications where all requirements follow a similar pattern, it is often possible to straightforwardly derive the assertion predicate from a requirement – for example, using a simple rule - so that the predicate does not need to be explicitly stated every time. Example: consider an API specification where all requirements are of the kind “when given argument values A and B, the API function F MUST return a result R that satisfies property P”.
[jacques]maybe we could simplify the above example

Such requirements are very close to a predicative form like: (F(A,B) returns R) AND (R satisfies P). Deriving the latter from the former is simply a matter of applying simple rules such as “replace ‘MUST do’ by: ‘does’, and similarly with any verb other than ‘do’ .” Note: in many cases, there is no need to use a sophisticated test assertion structure. When there are a very large number of assertions to be created from a large specification it may be that most assertions can just be copied from the specification statements verbatim. However, there may need to be rewording when the normative statement is optional and there may need to be special consideration in doing this when further statements depend on whether such optional features are implemented.

The note above may become unnecessary if we can better clarify the second bullet at the start of 2.1 above.

Also, note that we may also have implicit references (where the assertions are contained within the spec itself, by use of markup, or if the spec is a profile that subsets another spec, but the profile consists merely of a set of test assertions. (See also Lynn's example of the voting spec - which is written as a set of test assertions.) Need a good example.

[jacques] this above paragraph seems a bit redundant. Possibly a good example of "implicit TA elements" could illustrate its point better?

2.3 A More Complex Case

F2F: title is not specific enough about what the goal of the section is. Should we break this section into two: the first dealing with granularity (Observation #2), and the second dealing with more complex predicates? If we do this (and even if not) perhaps the complex predicate stuff belongs in the Advanced section? If we do this, then all we have left is granularity. Calling this section "complex" confuses things. The granularity discussion is really about practice - "how to".
Replace this section with a section called something like "Designing simple test assertions" (it's the practical "howto" that teaches people how to create/identify simple test assertions. Put back the text that was removed from an earlier version - "A test assertion should...." followed by bullet points (addressing granularity). "Designing a simple ta" is maybe too "grandiose" for this section? Need something more modest. Perhaps "Designing a simple TA" could be the title for the whole of Section 2?
This section could also incorporate some of the examples from the discussion of implicit elements in section 2.1.

Consider now the following statement in the widget specification:

[requirement 101] “A widget of medium size MUST use exactly one AA battery and have a red button on top.”

There are actually two requirements here that can be tested separately:

(requirement 101, part 1) A medium-size widget MUST use exactly one AA battery.

(requirement 101, part 2) A medium-size widget MUST have a red button on top.

Because of this it is possible to write two test assertions:

TA id: widget-TA1

Target: medium-size widget

Reference: specification requirement 101, part 1

Predicate: [the widget] uses exactly one AA battery.

TA id: widget-TA2

Target: medium-size widget

Reference: specification requirement 101, part 2

Predicate: [the widget] has a red button on top.

Several observations can be made on the previous example.

Observation #2: The granularity of a test assertion is an arbitrary decision. A single test assertion instead of two could have been written here, with the predicate: “the widget uses exactly one AA battery AND has a red button on top”. This choice may later have an impact on the outcome of a test suite written for verifying the conformance of widgets: with a single test assertion, a test case derived from this test assertion will not be expected to distinguish between the two failure cases. Using two test assertions - one for each sub-requirement - will ensure that a test suite can assess and report independently about the fulfillment of each sub-requirement. Other considerations such as the different nature of tests implied, the reuse of a test assertion in different conformance profiles, may also lead to the adoption of “fine-grained” instead of “coarse-grained” test assertions. Other advantages of fine-grained test assertions are described later. Usage considerations will dictate the best choice here.

Observation #3: The target is defined here as “medium-size widget”. It could have been defined as just “a widget”. This would have been a natural decision if the specification requirement had been worded as: “[requirement 101] If a widget is medium size, then it MUST use exactly one AA battery and have a red button on top.” In this case we could have had instead of the test assertion “widget-TA1” (and similarly for widget-TA2):

TA id: widget-TA1b

Target: widget

Reference: requirement 101, part 1

Predicate: if [the widget] is medium-size, then [the widget] uses exactly one AA battery.

The target class is now broader, but the predicate part is really of interest only for a subset of this class (the medium-size widgets). Usage considerations should again drive the decision here: a test suite that is designed for verifying all widgets, and does not assume a prior classification of these in small / medium / large sizes, would be better off with test assertions that only use “widget” as target, such as widget-TA1b. (Note that from a logical viewpoint, the above predicate value is still “true” for widgets that are NOT medium size, meaning the addressed requirement is technically fulfilled, in spite of its irrelevance.) On the other hand if all targets have been classified according to their claimed size, then a test case implementing widget-TA1b will uselessly repeat the “size” test.

Note: As an important part of the test assertion, even when the target is implicit, not explicit, it must be that the target for each assertion is clearly identifiable.

Observation #4: As seen in Observation #2, a test assertion predicate may be a Boolean expression - a composition of atomic predicates using logical operators AND, OR, NOT. As indicated in Observation #3, a test assertion predicate may also be of the kind: “if (condition) then (expression) – which is just another way to represent the Boolean expression: (expression) OR NOT(condition) – where both condition and expression are in turn predicates as defined here. (Delete the text marked in red...)

Observation #5: The predicate is worded in an abstract way, still close to the wording of the specification. No indication is given on what kind of test procedure will be used (how to determine the number and type of batteries, etc.), nor on the detailed criteria for the condition evaluation (for example, what kind of red pigments are acceptable to determine a “red” color.) Such details are normally left to the test cases that can be derived from above test assertions. These test cases will determine the precise criteria for conforming to the specification. However, if a precise criterion for interpreting the “red” color requirement is provided in an external specification - either referred directly by the widget specification or by a related conformance clause - then a test assertion must use this criterion in its predicate. Such a test assertion must then refer not only to the specification requirement in its reference property, but also to the external specification or to the conformance clause that refers to this specification.

2.4 Test Assertion Prerequisites

The previous test assertion - addressing requirement 101 of the widget specification – only applies to medium-size widgets. Let us assume that there are other requirements that define what “medium-size” means. In other words, the classification of widgets according to small / medium / large is not an obvious one: there is no sticker on the widgets (yet). Instead, it is a property that is itself defined in the widget specification and that is subject to verification, like the other properties.

In such a case, when writing test assertions, it is not a good idea to consider this property as part of the definition of the target class as we did in widget-TA1 and widget-TA2, because the class of a widget could not be identified prior to doing any test on this widget. However, we must make it clear that test assertions widget-TA1 and widget-TA2 are to be exercised ONLY on those widgets that have been proved to be medium-size. This is done by stating a prerequisite to these test assertions.

Assume that the following requirement defines the “medium-size” property:

[requirement 102] “A widget that weights between 100g and 300g and is from 5 to 15 centimeters long in its longer dimension, is a medium-size widget.”

There is a major distinction between requirement 102 and requirement 101:

requirement 101 uses “medium-size” as a prerequisite: its predicates only concern widgets that are already established as being medium-size.
requirement 102 defines how to qualify as medium-size.

The test assertions for requirement 102 can be written as:

TA id: widget-TA3

Target: widget

Reference: specification requirement 102, part 1

Predicate: [the widget] weights between 100g and 300g.

TA id: widget-TA4

Target: widget

Reference: specification requirement 102, part 2

Predicate: [the widget] is from 5 to 15 centimeters long in its longer dimension.

Observation #6: Requirement 102 is a normative statement of a different kind to requirement 101: it does not make use of keywords such as MUST. Instead, it defines a property. It is a rule of the kind: “if (the widget satisfies the condition C) then (the widget exhibits property P)” where P cannot be verified separately from C. For a target, fulfilling this requirement does not have the same meaning as fulfilling requirement 101. Fulfilling R-102 means the target exhibits a property (that it is not necessarily required to exhibit), while fulfilling R-101 means the target adheres to the specification itself. In our case, a predicate value of “false” for R-102 simply means the target is not medium-size, while “false” for R-101 means the target violates the specification. However, this difference in outcome interpretation does not affect the way the test assertions are written: in both cases the focus is exclusively on the feature to be measured.

Observation #7: There is no mention of the “medium-size” property at all in the predicates of test assertions ‘widget-TA3’ and ‘widget-TA4’. This is because this property is precisely what needs to be established by a test suite containing test cases that are derived from these test assertions. Only when a target (here a widget) evaluates to “true” for these two test assertions, will it be considered as medium-size. These test assertions are only concerned with the nature of these tests, not with how to interpret their outcome.

Now that we have a means to establish the “medium-size” property, we can use it to qualify targets for the requirement 101. This is done by adding a “prerequisite” component to widget-TA1 and widget-TA2 and rewrite them respectively as:

TA id: widget-TA1c

Target: widget

Reference: specification requirement 101, part 1

Prerequisite: widget-TA3 AND widget-TA4

Predicate: [the widget] uses exactly one AA battery.

TA id: widget-TA2c

Target: widget

Reference: specification requirement 101, part 2

Prerequisite: widget-TA3 AND widget-TA4

Predicate: [the widget] has a red button on top.

The prerequisite can be construed as a precondition used to qualify the target, prior to evaluating the predicate of the test assertion. The prerequisite is itself a predicate that evaluates to “true” or “false”. When a prerequisite is quoting another test assertion as above, such a reference must be understood as a substitute for the predicate of the referred test assertion (assuming it has same target). The possible outcomes of a test assertion with prerequisite become:

If the prerequisite evaluates to “false”, then the test assertion does not even apply (or its outcome can be stated as “Non Applicable”). A test case derived from this test assertion should not even be executed on this target: the result of doing so would be meaningless.
If the prerequisite evaluates to “true”, and the test assertion predicate evaluates to “true”, then the target is exhibiting the feature described in the addressed specification requirement.
If the prerequisite evaluates to “true”, and the test assertion predicate evaluates to “false”, then the target is NOT exhibiting the feature described in the addressed specification requirement.

The prerequisite could have been stated as an explicit predicate in widget-TA1c and widget-TA2c, i.e. widget-TA1c could have been rewritten as:

TA id: widget-TA1d

Target: widget

Reference: specification requirement 101, part 1

Prerequisite: [the widget] weights between 100g and 300g AND [the widget] is from 5 to 15 centimeters long in its longer dimension.

Predicate: [the widget] uses exactly one AA battery.

widget-TA1d is semantically equivalent to widget-TA1c. However, because the notion of “medium-size” is itself specified as a property that is subject to verification and enforcement, it is useful to write test assertions for this property. It is then preferable to reuse such test assertions as prerequisites whenever this property is assumed. If the notion of medium-size evolves in future releases of the widget specification, a test assertion such as widget-TA1c does not need to be altered: only its prerequisite test assertion needs to be, while all test assertions that explicitly state the prerequisite predicate - as widget-TA1d does - would need be updated.

2.5 The Case of Normative Optional Statement

All the requirements used so far in the widget specification have a mandatory character: they are absolute requirements using the keyword MUST. Even if requirement 102 does not use any keyword for indicating the requirement level, it is a “definition statement”, describing a sufficient condition for establishing the “medium-size” property, as explained in Observation #6”.

Interpreting the outcome of such test assertions is then straightforward. Test cases derived from such test assertions can make a clear statement of conformance to the specification for the target under test:

(test assertion predicate = “true”) means not only that the target exhibits the specified feature of the specification, but also that the target fulfills a specification requirement, since this feature is required.

However there might be several ways to conform to a specification, also known as dimensions of variability. While both conforming, two implementations may not exhibit the same features. This section considers one of the most obvious cases of variability: optional features.

What if the specification statement is optional, i.e. uses RFC keywords SHOULD / RECOMMENDED or MAY / OPTIONAL?

Examples:

[statement 103] “It is RECOMMENDED for a widget to be waterproof.”
[statement 104] “A widget MAY have a metallic casing.”

Such (normative) statements cannot be construed as formal requirements – a widget will not fail to conform to the specification if it is not waterproof, or if it has a plastic casing. But establishing conformance is not the sole objective of test assertions. Some test suites are intended to evaluate the capabilities of an implementation – for example, which options it implements - regardless of conformance considerations. Even with a conformance objective in mind, a clear separation must be made between:

- (a) Describing under which condition a target can be said to exhibit a specified feature. This is the role of a TA.

- (b) Deciding if a target satisfies a conformance criterion. This is the role of one or more test cases that interpret the outcome of a TA.

Therefore, test assertions can be written for such statements 103 and 104, simply by ignoring the RFC keywords in use, and by focusing on the specified feature and its related predicate:

TA id: widget-TA103

Target: widget

Reference: specification statement 103

Predicate: [the widget] is waterproof.

TA id: widget-TA104

Target: widget

Reference: specification statement 104

Predicate: [the widget] has a metallic casing.

How the test assertion predicate value (true/false) is to be interpreted – for example, “false” may be interpreted differently depending on the level of conformance being considered – is not the concern of the person writing the test assertion.

However, as previously mentioned, a test assertion may be written for a specific conformance clause. For example, a conformance clause may define a mandatory degree of “waterproofing” (or refer to an external definition) as withstanding an immersion of 3 feet under water level – but not 50 feet. In that case the test assertion must explicitly refer to this conformance clause item in addition to the specification statement, and it must reflect this specific waterproof definition in its predicate.

3 Advanced Features

Ed: issue #009 Editorial Note: Revisit basic and explore corner cases then address optional.

We have considered the four essential elements of the test assertion: identifier, reference, target class and predicate. In practice there may be a need further features to better cater for corner cases such as

specifications where the normative statements are embedded wholly or partly in tables and diagrams
specifications which normatively reference other specifications
specification and test assertions versions
inheritance and dependencies between specifications
redundancy of excessively repeated assertions elements
test assertion targets which are classified and categorized in conformance clauses.

3.1 Complex References to Specifications

[jacques] suggest we rename section title. "Complex" is not the best word. Also it seems that under the umbrella of "Complexity" this section is mixing two very different things that probably have no other common point: (a) the case of non-trivial references from TA to spec (from the TA "reference" item), e.g. non-textual, multiple location etc. (b) the case of the existence of external specs to which the target is, implicitly or explicitly, required to comply with. This is primarily a spec-to-spec reference issue, which might be handled in a variety of ways, some of them will not show as a TA-to-spec reference (e.g. a blanket statement for a group of TAs). I'd suggest that (b) deserves its own section, with a more focused title (e.g. "When a specification refers to another specification" or the like). Could (a) then be treated as a best practice thing?

When writing test assertions there needs to be clarity about the boundary of the part of the specification which is referenced by the individual test assertion. There is added complexity when the test assertion is based on logic spread over several parts of a specification, such as its tables and diagrams. This may require that there be new, derived text in the test assertion which is not found literally within the specification. Care must therefore be taken to ensure that the logic of test assertion does not deviate from or add to the logic of the normative statements in the specification. Care must also be taken when literally copying text from the specification to ensure that divorcing the text from its context, formatting, etc.does not result in subtle changes to its meaning.

When creating a test assertion from the text of a specification it is quite normal for normative text for an assertion to be a subset of a specification sentence, a superset of many sentences, or combined and disjointed text fragments from multiple sentences. Furthermore, multiple assertions might share the same text fragments. It may be the exception rather than the rule that the boundary of the part of a specification covered by a particular test assertion corresponds to one or more whole sentences as they appear in the specification. Within a single assertion list there may be references to multiple specifications so it should not be assumed that all specification references are to the same specification.

[jacques] with in mind the constant concern of keeping the guideline short and "captivating": would suggest that all except the last sentence in above paragraph, is trivial. Or is too abstract (Can't we make this point as a side-comment on one of our numerous examples?)

Complexity arises when specifications reference other specifications or themselves are referenced. In such cases, specifications may inherit normative statements from the referenced specifications. This may lead to situations where the test assertions inherit from other lists of assertions. The specification inheritance may be partial or may be inheritance with modifications. In such cases the corresponding test assertion behaviors might be altered and should be evaluated in the context of the inherited specification relationship.

[jacques] Legitimate expectation from the reader about this guideline: how do these "clear assumptions" tanspire in the TA expression? We might need a new form of "pre-requisite" that assumes the target is conforming to the external spec ABC at conformance level 3. Suggest we need some concrete proposal here plus some example. As a consequence, I just filed a new "issue" on wiki.

It is not expected that the test assertions will seek to cater for each and every specification referenced by the primary specification and it might be that there are already existing measures available to test conformance to the specifications named in the normative references. The testing of artifacts for conformance to the referenced specifications, if it is not to be covered separately, may be delegated to another test assertion. Then test case writers can concentrate on the primary specification while treating test assertions for normative reference requirements as prerequisites.

3.2 Optional Test Assertion Techniques

Grouping

There are techniques for grouping test assertions such that a given assertion can be considered a member of more than one group. One method is to assign keywords as 'tags' to an assertion or to elements of an assertion such as its target. This provides indispensable value to testers in identifying those assertions that cover a class of target or where dependencies might exist between one test and another. In particular, for variability of specifications such as where modules, profiles or conformance levels are defined, grouping may take on further significance and value. Sometimes a conformance clause itself provides such classification 'tags'. The same tags could be among those added to the test assertions to help in relating them to the conformance clauses. The tags can be defined their significance explained in a header or separate document. Another method is to use a scheme in combining tags into a string for use as either a test assertion identifier or test assertion target name (see also the section on sequences below).

Example:__

In header section -

keywords:

'Widget'
'MediumSizedWidget': subclass of 'Widget'
'LargeSizedWidget': subclass of 'Widget'

(Note: definitions may also be added)

Then in a test assertion -

TA Id: 101.11

...

Applies to: MediumSizedWidget, LargeSizedWidget

Predicate: [target] has a red button on top

...

The following are particularly useful to consider for tags:

Prerequisites can be defined by stating that a particular prerequisite applies to all test assertions with certain tags
Test assertion targets can be classified so that test assertions can be said to apply to targets classified with certain tags
Test assertions can be tagged with keywords which match tags or classifications in conformance clauses and specification variability modules, profiles, etc.

When the test assertions are written before the conformance clauses are drawn up the test assertion tags can aid the creation of the conformance clauses. It should be possible for a conformance clause to state that level A requires conformance to test assertions defined by the group tagged 'xyz'.

Inherited Classifications

Defining test assertion targets as classes, typically in a header section or separate document, provides opportunities for inheritance of properties and behaviors. For example, there may be a target which is an invoice document and a target for another set of assertions for which the target is just a document in general. In this case the invoice could be identified as a subclass of document. Then all assertions applying to the target calls 'document' will also apply to the subclass 'invoice'.

Showing Sequences and Dependencies

Besides using tags to help define dependencies (by using tags with prerequisites) another method is to use a scheme to sequence the test assertion identifiers. Typically the test assertion identifiers will follow a logical sequence and this may have special significance. For example, the sequence might follow the sequence of the relevant statements in the specification, which may in turn follow a gradual development of the theme or logic of the technology. It may be of value to explicitly define the scheme which the identifiers follow, again perhaps in a header section or accompanying document for the test assertions. The sequence scheme might merely act as a hint to test writers about a sequence in which tests would run such that a certain result of one test leaves the implementation in a suitable status for a following test.

Using Variables

Variables may be declared for use in a test assertion. They are similar to tags but can have values assigned to them (including at a subsequent test development stage, as meta-data). The variables are referenced in the test assertion predicate or test assertion target, etc.

For example

TA Id: 101.8

Variable: X

Target: Widget of size X

Predicate: “[widget of size X] has a red button on top”

or, alternatively

TA Id: 101.9

Variable: X

Target: Widget of size X

Precondition: “X = Medium”

Predicate: “[widget of size X] has a red button on top”

There might also be meta-data added later as part of the development of a test suite which provides a value for X.

Identifying Versions

[Ed: issue #015 Editorial Note: Paul will send contribution for this section]

3.3 A More Advanced Structure

A more highly structured assertion may sometimes be required to:

address optionality where expressions are typically derived rather than literal
address complexities such as mandatory requirements dependent on optional requirements
help identify and clarify dependencies between assertions
help identify and clarify conditions and assumptions required in test assertions
facilitate sophisticated testing requirements such as automation

Such a test assertion may have the following as its main components

expressions more formal than free text prose and which always evaluate to true or false ('predicates') to qualify the test assertion target, such as by describing prerequisites to indicate that the test assertion target is adhering to the specification requirement addressed by the test assertion
a structure to allow separation of prose and/or expressions

It may be aimed at test assertions being either formally processed according to a particular notation or methodology or even a set of test assertions which are defined to facilitate automated testing. This might require the use of a specialized notation. There are languages such as ADL, OCL, XML Schema, Schematron and XQuery which may help to formally describe parts of a specification and its assertions. When these are used then, along with the test assertion predicate, specify which dialect is being used (e.g. @dialect="XQuery", @dialect="OCL", etc.).

For example:

An assertion that Invoice_Subset_A includes IssueDate may be more precisely represented with an XPath to the IssueDate.

Test assertion ‘http://example.org/v0-3/#Invoice-SubsetA-v1-0-TA#1’ may then state, using a formal representation, the following structured assertion:

<Assertion>

<ID>http://example.org/v0-3/#Invoice-SubsetA-v1-0-TA#1</ID>

<Reference>http://example.org/v0-3/#Invoice-SubsetA-v1-0</Reference>

<Target dialect="Namespace"> http://example.org/v0-3/#Invoice-v1-0</Target>

<Predicate>

<Property>Contains</Property>

<Value dialect="XPath1">/Invoice/IssueDate</Value>

</Predicate>

<Assertion>

Note in this example how the predicate expression is further divided into a property and a value (property-value-pair). This type of test assertion structure has the features of a 'triple', namely subject (target) + property + value. The subject and property are fixed by controlled vocabulary and the value lies within a range of possible values. In this case the range of values is defined using a scheme, namely XPath but otherwise a datatype such as integer, string or date might be used to constrain the range of values allowed.

If the strings "Contains", "Namespace" and "XPath1" in this example were defined in codelists or an ontology or other controlled vocabulary then this test assertion representation would give the opportunity for automated processing. It might even allowing testing and validation parsing in a manner similar to the use of an XML schema. Of course it may be possible in this example to use an existing technology such as RelaxNG, W3C XML Schema or Schematron or a combination of these for the entire representation of such an assertion or sets of such assertions. Alternatively, artifacts such as Schematron validation schemas could be generated from the test assertions.

There may need to be special considerations when creating test assertions covering parts of a specification involving, tables diagrams, schemas and the like. Note that, effectively, a schema, such as one defined using W3C XML Schema, may equate to a set of test assertions but can actually be used directly for testing. A schema included in a specification may provide the opportunity to go straight to a test, bypassing test assertions but, equally, test assertions may be generated from schemas. Diagrams involving a time element such as sequence diagrams may lend themselves to generation of target behavior assertions (cause-effect or before-and-after assertions). Those involving classes may allow generation of assertions about properties of targets. As with schemas it may be possible to generate tests themselves from diagrams such as UML diagrams, bypassing the assertion step. Diagrams and schemas may be included in a set of test assertions where appropriate.

4 Summary

4.1 Essential Test Assertion Components

Test Assertion Identifier

A globally unique test assertion identifier (TA id). This facilitates tools development and the mapping of assertions to specification statements.

Specification References

A reference to the precise specification requirement(s) or normative statement(s) that it addresses.

Test Assertion Target

A test assertion target (or TAT). Such a target is an implementation or a part of an implementation of the referred specification. What is usually defined as a test assertion target is a category of targets of similar type that are all subject to this test assertion.

Predicate

A predicate asserting the feature (a behavior or a property) described in the referred specification statement (s). If the predicate evaluates to “true” over the test assertion target, this means that the target exhibits the feature. “False” means the target does not exhibit the feature.

4.2 Optional Test Assertion Components

Prerequisites

These are either references to other test assertions or conditional expressions which further qualify the test assertion target or the assertion of its properties or behaviors (such as "if ... then ..." for properties and “when ... then ...” for behaviors). They evaluate to true or false and if false then the assertion is to be considered 'not applicable'.

'Precondition' is a term which may be used to refer to the type of prerequisite which is a predicate in itself rather than a reference to another test assertion. The latter, a qualifying reference to another test assertion, is then referred to as a 'prerequisite'.

Glossary

Normative Statement, or Normative Requirement

A statement made in the body of a specification that defines prescriptive requirements or properties on a conformance target. A normative statement or normative requirement typically uses keywords such as those defined in IETF RFC2119 or ISO/IEC (MUST, SHOULD, MAY...)

Implementation

A product, document, process, or service that is the realization of normative statements of a specification.

Test Assertion Target

Implementation or part of an implementation that can be the object of a test assertion or test case.

Test Assertion

An independent, complete, testable expression for evaluating adherence of a test assertion target to some normative requirement statement in the specification. A test assertion always identifies the normative requirements that it addresses. It identifies a class of test assertion targets, which are expected to fulfill these requirements. A test assertion describes the expected output or behavior for the test assertion target within specific operation conditions, in a way that can be measured or tested.

Test Case

Consists of a set of a test tool(s), software or files (data, programs, scripts, or instructions for manual operations) that verifies the adherence of a test assertion target to one or more normative statements in the specification. Typically, a test case is derived from one or more test assertions. Each test case includes: (1) a description of the test purpose (what is being tested - the conditions / requirements / capabilities which are to be addressed by a particular test, (2) the pass/fail criteria, (3) traceability information to the verified normative statement(s), either as a reference to a test assertion, or as a direct reference to the normative statement.

Conformance

Adherence of an implementation to requirements in specification with possible variations or refinements expressed in a separate clause (conformance clause).

Conformance Clause

A statement in the conformance section of a specification that provides a high-level description of what is required for an artifact to conform. It, in turn, refers to other parts of the specification for details. A conformance clause must reference one or more normative statements, directly or indirectly, and may refer to another conformance clause. A conformance clause may exist in a conformance profile which overrides the requirement level of the normative statements. For example a SHOULD may become a MUST for a particular conformance profile or conformance level. A conformance clause may be expressed by associating a clause with a set of test assertions. The conformance clause may be appended to the main body of a specification as a conformance section.

Revision History

[[Table]]

Ed: issue #014

Appendix A. References

Appendix B. Prior Art

Appendix C. Best Practices

Specifications

Specification architects and implementation guide architects link test assertions to precisely identified parts of a specification. Sometimes the text of the assertion can be quoted literally from the specification while there are more complex cases where the text of the assertion has to be derived from the specification. When writing test assertions there needs to be clarity about the boundary of the part of the specification which is referenced by the individual test assertion. There is added complexity when the test assertion is based on logic spread over several parts of a specification, such as its tables and diagrams. This may require that there be new, derived text in the test assertion which is not found literally within the specification. Care must therefore be taken to ensure that the logic of test assertion does not deviate from or add to the logic of the normative statements in the specification. Care must also be taken when literally copying text from the specification to ensure that divorcing the text from its context, formatting, etc.does not result in subtle changes to its meaning.

When the test assertions are created for a specification which itself normatively references other specifications and when there are dependencies between those specifications (layering, bindings), a test assertion should restrict its scope, yet make clear assumptions on how much of the referenced specification is supposed to be previously tested. It is not expected that the test assertions will seek to cater for each and every specification referenced by the primary specification and it might be that there are already existing measures available to test conformance to the specifications named in the normative references. The testing of artifacts for conformance to the referenced specifications, if it is not to be covered separately, may be delegated to another test assertion. Then test case writers can concentrate on the primary specification while treating test assertions for normative reference requirements as prerequisites.

Grouping

Example:

In header section -

keywords:

'Widget'
'MediumSizedWidget': subclass of 'Widget'
'LargeSizedWidget': subclass of 'Widget'

(Note: definitions may also be added)

Then in a test assertion -

TA Id: 101.11

...

Applies to: MediumSizedWidget, LargeSizedWidget

Predicate: [target] has a red button on top

...

The following are particularly useful to consider for tags:

Prerequisites can be defined by stating that a particular prerequisite applies to all test assertions with certain tags
Test assertion targets can be classified so that test assertions can be said to apply to targets classified with certain tags
Test assertions can be tagged with keywords which match tags or classifications in conformance clauses and specification variability modules, profiles, etc.

Inherited Classifications

Showing Sequences and Dependencies

Advanced Structure

Include version information in the test assertion identifier and in references to the specification.

Consider the opportunities there may be for automation. Any formalism for Specification Analysis will have downstream effects on tools that may be designed to automate the process of analysis - as well as the metrics applied to an analysis. Specify this formalism in a pragmatic unambiguous way, such that analysis tools may be unambiguous and useful.

There are languages such as ADL, OCL, XML Schema, Schematron and XQuery which may help to formally describe parts of a specification and its assertions. When these are used then, along with the test assertion predicate, specify which dialect is being used (e.g. @dialect="XQuery", @dialect="OCL", etc.) using meta-data.

For example:

An assertion that Invoice_Subset_A includes IssueDate may be more precisely represented with an XPath to the IssueDate.

Test assertion ‘http://example.org/v0-3/#Invoice-SubsetA-v1-0-TA#1’ may then state, using a formal representation, the following structured assertion:

<Assertion>

<ID>http://example.org/v0-3/#Invoice-SubsetA-v1-0-TA#1</ID>

<Reference>http://example.org/v0-3/#Invoice-SubsetA-v1-0</Reference>

<Target dialect="Namespace"> http://example.org/v0-3/#Invoice-v1-0</Target>

<Predicate>

<Property>Contains</Property>

<Value dialect="XPath1">/Invoice/IssueDate</Value>

</Predicate>

<Assertion>

Recommendations/resolutions

Versioning (issue 23)

Versioning is important - we should discuss it. (And we should also review the "spec reference" portion of the doc to take versions into account.) However, recommending a specific encoding (such as versionadd/versiondrop) goes too far. We can't be sure that we can get it right (choose an encoding that will work for everyone). We should suggest that people choose an encoding mechanism that works for them (if indeed they need to encode for versions) but no more...

External specs (issue 24)

Patrick: why does it make a difference whether the SOAP-header requirements are expressed in this spec or in another spec? It's not a matter of how the specs are partitioned, but whether we sometimes want to take for granted that a set of requirements (which may or may not be defined in an external spec) have been met. (Not my job to test for them.) This is more about prerequisites than about specification references. Is this just another kind of prerequisite?

References:
- Groups - Test Assertions Guidelines draft 0.9.1 (TestAssertionsGuidelines-draft-0-9-1.doc) uploaded
  - From: stephen.green@systml.co.uk