Test Assertions - A Simple Approach
Introduction
- Common thought at Sun is that assertions are "literal" - their text can
be found within the spec
- Java specifications - JavaDoc, JLS, JVMS
- Most assertion in Java Specifications are literal
- In practice, Many assertions are derived from text (or a diagram, an image,
a cell within a table, an element in a formal grammar, etc) found within a
specification.
- Sometimes an assertion is derived from multiple locations in one specification
or from multiple specifications.
Necessary assertion data:
Definition: Assertion - structured data that exists outside of a specification.
Simplest form of an assertion:
- assertion ID - (should be unique within the spec and remain static)
- assertion text
- reference indicating the location in the specification where the text can
be found or from where it can be derived.
Two assertion classifications:
- literal - Text can be found directly in the specification text
- JLS Chapter 10 Arrays. 10.7 Array Members "Every array implements
the interfaces Cloneable and java.io.Serializable."
- derived - Assertion text is derived from text found within the specification
or specifications.
- simple - A single reference to a set of text within a specification
is sufficient.
- compound - More than one reference to sets of text within a specification
or specifications is necessary.
At Sun, we work mostly with simple literal assertions.
Our goal:
- Define assertion markup for text within specifications
- Markup should be as simple as possible
- simplest case is where a string of text in a specification is an assertion
- flexible to handle complex compound derived assertions
- Should be extendable
- Additional attributes could be added to proposed markup
Assertion markup structure
General Structure
Example of an assertion as part of an assertion list:
<assertion id="123">
<text [type= "literal" | "derived"]>This is the text
of the assertion.</text>
<reference [type = "primary" | "secondary"]>Pointer
into the spec(s)</reference>
[optionally, additional <reference> tags may be provided]
</assertion>
- literal/derived attribute
- identifies whether the text of the assertion is literally copied from
the spec or whether it has been derived from it.
- Optional, default is literal.
- primary/secondary attribute
- Optional, defaulting to primary.
- allows the construction of "compound" assertions where more than one
reference to the spec (or even different specs) is required to create
the assertion.
- Assertions of this type will probably be of type derived rather
than literal.
- If literal, the text of the assertion will be found at the primary
reference.
Example of Simplest Case
a single literal string within a specification:
<assertion id="123">
<text>This is the text of the assertion.</text>
<reference>SpecID:123</reference>
</assertion>
The reference into the specification should include a reference to the specification
(specID) and a pointer to a location (:123) within the specification.
The format of the reference into the spec has not been determined. Some possible
options are:
- Xpath expressions for XML and XHTML
- Section, chapter, sentence, line numbers
- references to HTML anchors
Specification markup structure
Sometimes it makes sense to apply markup directly to a specification so that
assertion lists can be automatically generated. At the minimum, markup process
involves identifying normative text.
- Java Language Specification
- Java Virtual Machine Specification
- Java API Specifications (JavaDoc) - this is typically done with a tool (Spectrac)
or on paper with a highlighter
Example:
<normative id="123" [see-also="SpecRef1 [, SpecRef2, SpecRef3...]]">
This is some normative text.
</normative>
The optional see-also attribute
- allows one or more additional references to other <normative> elements
- Additional references can be in same specification or external specifications.
Example - JLS:
Different format than proposed here - for illustration purpose:
;)
Deriving assertions from marked-up specs - Simple Case
The simplest case of markup within a specification reduces to:
<normative id="123">This is some normative text.</normative>
From this it is possible to automatically generate the simple literal
assertion discussed above:
<assertion id="123">
<text>This is the text of the assertion.</text>
<reference>SpecID:123</reference>
</assertion>
Since this is a literal assertion, the text of the assertion will be the same
as the normative text identified within the specification.
More complex cases.
A specification example with secondary normative text, non-normative text,
and primary normative text:
<normative id="123" type="secondary">
This is some normative text that provides some general background requirements
for a range of cases.
</normative>
This is non-normative text.....
<normative id="456" see-also="specID:123">
This is normative text that states requirements for a particular instance.
</normative>
From this text the following compound literal assertion could
be automatically generated:
<assertion id="456">
<text type="literal">This is normative text that states
requirements for a particular instance.</text>
<reference type="primary">Pointer to 456</reference>
<reference type="secondary">Pointer to 123</reference>
</assertion>
- A human editor might choose in this case to replace the text with something
"derived" from both 123 and 456, in which case the text type would
be changed to derived.
- The primary attribute is optional and could be dropped
Summary
- Simple test assertion framework that allows us to cover the most common
cases here at Sun
- Can support compound assertions
- Does the support for Compound assertions complicate the proposal?
- Would Compound assertions be used in the industry?
- Does the "see-also" attribute complicate the specification mark-up
process
- Is it enough for specification writers to only identify normative text?
- Test writers could manually link together normative statements to form
compound assertions
