Initial draft of test suite reviewer instructions.

[Robert Orosz <roboro@auto-trol.com>]

Hi all,

Lofton and I had a discussion last Friday about what should be included in the test suite reviewer's guidelines. He also sent me a "Test Builder's Manual" which he wrote for the W3C SVG WG, and I've liberally plagiarized from that. Based on those things, I've written and attached an initial draft of test suite reviewer guidelines for WebCGM. My editorial comments and questions are highlighted with a yellow background.

Comments and criticisms are welcomed.

Rob

Title: Instructions for WebCGM 2.1 test suite reviewers.

WebCGM 2.1 Conformance Test Suite
Guidelines for Reviewers

Revision: 0.90 beta

Date: October 7, 2008

Author: Robert Orosz

Preface

This document provides instructions to reviewers of new tests submitted to the WebCGM 2.1 test suite. A careful, thoughtful review is essential to the quality of the test suite and therefore to the WebCGM 2.1 specification itself. The purpose of this document is to describe the necessary items to consider when doing a review.

Test Review Guidelines

Overall Topic(Chapter) Content

Consider a chapter and its associated test cases, and consider:

• Is the coverage complete? Are there features in the specifcation which are not tested?
• We need a couple of volunteers to look at the specification and test suite as a whole.

Individual Test Content

Look at an individual test case and assess the following:

• Is the "look and feel" or "style" consistent with previously established, i.e. existing tests? New tests should not invent new styles.
• Names of files, etc. must use camel case.
• Is the test appropriately named?
• Does the test include the standard comment header (available in header.txt on the FTP site)?
• Is the test traceable; i.e. does it identify what specific part of the WebCGM 2.1 specification is being tested?
• Is the CGM valid?
• Is the (X)HTML valid? Do we want to standarize on one or the other, i.e. HTML or XHTML and a particular DTD?
• Is the ECMAScript valid? Does anyone know of a tool for checking this other than watching a browser fail or not fail?
• The reference image must be in PNG format with 8-bit color. Does it correctly depict the graphical result of the test?
• The tests should be atomic and well-focussed.
• "Complexity density" of the test. An individual test should not be too complex (this is somewhat subjective).
• Minimal prerequisites. The test should not rely other parts of WebCGM beyond what is being tested.
• The test should be self-documenting.
• In addition to the standard header, the ECMAScript, companion XML files, etc. should be well-commented.