Rationale for test assertions (homework for f2f)

[Patrick Curran <Patrick.Curran@Sun.COM>]

Sorry this is late. I'll call in to the meeting to present it.

-- Patrick

Title: Rationale for Test Assertions

What is a test assertion?

• A statement of behavior, action, or condition that can be measured or tested.
• Each test assertion is an independent, complete, testable statement of requirements in the specification.
• Assertions are derived from the specification and can be mapped to it.
• In the simplest case, an assertion may simply consist of text from within the specification.
• Assertions provide a normative foundation from which one or more test cases can be built.

Benefits of test assertions 

• Improve specification quality by helping to uncover inconsistencies, ambiguities, gaps, and non-testable statements in the specification.
• If not developed by the spec authors, should be reviewed and approved by them.
• Best results are achieved when assertions are developed in parallel with the spec.
• Facilitate and promote the development of conformance tests and testing tools.
• Encourage the early development of conformance tests that can be used during implementation.
• Simplifiy the distribution of the test development effort between different organizations while maintaining consistent test quality.
• Improve confidence in the resulting tests as a measure of conformance.
• Enable coverage analysis (estimating the extent to which the specification is tested).

Coverage analysis

• Partition the specification
  • By feature, language elements, logical sections, or even paragraphs or pages.
• Define coverage goals for each section.
  • What test-development resources are available?
  • How critical is it that each section is thoroughly tested?
    • Consequences for incompatible implementations due to low test coverage?
• For each partition, provide a mapping between testable assertions and test coverage.
• Measure or estimate the extent to which each area of the specification has been tested.
• Coverage measurements can be expressed in both breadth and depth terms.
  • Breadth measurements are relatively simple to derive.
    • What percentage of assertions are covered by at least one test?
  • Depth measurements are more subjective.
    • Estimate how thoroughly each assertion is tested.
• Different coverage goals for different areas of the specification may be appropriate.
  • First cover each feature at a minimal level (breadth-first).
  • Then focus on providing deeper coverage
    • in areas that are more difficult to implement,
    • where incompatible implementations are more likely,
    • where the consequences of incompatibility will be more severe.
• Provide a coverage report with the test suite.
  • At a minimum, a simple mapping of tests to areas of the specification.
  • Preferably include counts and averages of the number of tests associated with different areas.
• Coverage reports help users of the test suite understand its strengths and weaknesses.

Guidelines for creating test assertions

• A test assertion should be a simple, atomic statement.
  • Address one feature at a time.
  • Keep each assertion as simple as possible.
    • Multiple single-feature assertions are easier to test than complex multi-part assertions.
    • Simple test assertions make it easier to identify the cause of test failures and to exclude buggy tests.
• Map assertions to text within the specification.
• Don't change the semantics of the specification.
• Group assertions into logical sets, following the structure of the spec.
• Provide a unique ID for each assertion to facilitate tool development.
• Test assertions must be technology-neutral (don't assume features of the implementation).
• Do not state how to test (a test assertion is not a test description).
• Indicate explicit dependencies and constraints.
• Describe features, values, attributes to be measured and indications of success or failure.