Keep It Simple

[Patrick Curran <Patrick.Curran@Sun.COM>]

Hello, everyone:

I've lurked for too long - sorry about that. I've been following the 
discussion with great interest, and it's time for me to jump in. I'll 
start by offering some general opinions and suggestions, and then 
hopefully I'll be able to respond in more detail to some individual 
messages.

In the interests of making progress in a reasonable period of time, and 
of defining something that is likely to be adopted by a significant 
number of spec writers and test developers, I propose that we Keep It 
Simple.

Our primary goal should be to encourage and promote the use of test 
assertions by specification writers. The W3C QA Working Group briefly 
discussed the value of test assertions in the QA Framework: 
Specification Guidelines [1], but much remains to be done to explain and 
justify their use. Publication of a "guidelines" document, even without 
any definition of structure or markup would be extremely valuable. (My 
test developers spend a considerable amount of time identifying 
assertions when this could have been done by the spec authors.)

While it is theoretically possible to write specifications in formal 
languages, and to write (or even automatically derive) detailed, 
structured test assertions from which test cases can be automatically 
generated, most spec authors would not use these approaches. (With some 
notable exceptions - for example the Java Language and Virtual Machine 
specs - most specs in the Java world are written in English by people 
whose primary job is not the creation of formal specifications, but 
rather of creating software which they document as best they can.)

I therefore suggest that we declare these topics out of scope, at least 
for the first phase of our work.

A topic of lively discussion within the W3C Working Group was whether 
it's possible to classify "testable sentences" within a specification as 
test assertions, or whether assertions must necessarily be "derived" 
from the specification and formulated separately in some kind of 
supplement to the spec. I believe that we must allow for both 
possibilities. (Here in the Java world we usually identify text within 
the specs as assertions, but I know that other approach is preferred 
elsewhere.)

The QAWG  eventually agreed that both approaches were valid, and defined 
a test assertion as "a measurable or testable statement of behavior, 
action, or condition [that is] contained within or derived from the 
specification's requirements and provides a normative foundation from 
which one or more test cases can be built."

In addition to promoting the identification/creation of test assertions, 
a simple XHTML markup would be very helpful. For example, when an 
assertion is contained within the specification we might mark it up like 
this:

<assertion id="123">The horizontal and vertical sides of a square are 
equal in length.</assertion>
This means that you can flip the square by 90 degrees and it will look 
just the same.

Alternatively, if the text of an assertion is not found directly within 
the specification but instead is derived from it and recorded 
separately, it might be marked up like this (note that we now need to 
provide a pointer into the specification to indicate where the assertion 
was derived from):

<assertion id="123" 
specref="http://www.foo.com/docs/geometry/square.html#sides";>
The horizontal and vertical sides of a square must be equal in 
length.</assertion>

If the TC achieved no more than this (encouraging spec authors to create 
test assertions and defining a simple markup) I would be very pleased.

A well-defined list of assertions is key to much of the test development 
process, and a great deal can be built from this simple foundation - for 
example, coverage maps and reports, "marked-up" specifications that link 
assertions to test cases, etc. (See "How Many Tests are Enough?" in the 
Test Development FAQ [2] for my thoughts on coverage measurement.) For 
example, here at Sun we have a tool that we use to identify and mark 
test assertions within Javadoc, to associate assertions with test cases, 
and to generate reports that tell us how extensively we've covered 
various areas of the specifications. Industry standards in this area 
would allow us to create a more widely useful version of this tool.

Of course, once we've defined a markup for assertions there will always 
be a great temptation to add more structure and additional elements. 
What version(s) of the spec does this assertion apply to? Is it 
associated with a Profile or another "Dimension of Variability" [3]? How 
important is it that it be tested? Is it tested, and if so how thoroughly?

Some additional structure is probably appropriate for test assertions, 
but we might want to consider starting with a "Basic Profile" that 
defines the minimum necessary elements and expand on this in a later 
phase to create an"Advanced Profile".

However, we should beware of conflating test (case) metadata with test 
assertions. They don't necessarily belong together. (An assertion states 
what is to be tested, whereas test metadata typically helps to define 
how to test.) Given a simple assertion ID as suggested above, test 
metadata stored separately from the specification or assertion list 
could easily be integrated with it. As an author of the Test Metadata 
document [4] David referred to in another thread, I encourage those who 
wish to work on test metadata to do so through the W3C QA Interest Group 
rather than here.

In conclusion, I suggest limiting our phase one activities to a 
"guidelines" document that explains the value and promotes the use of 
test assertions, and the definition of a simple markup that allows them 
to be identified within text specifications and also in separate 
"assertion lists". As the QA Framework: Specification Guidelines [1] 
explains:

"Test assertions facilitate the development of consistent, complete 
specifications and promote the early development of test cases. 
Developing or extracting test assertions helps uncover inconsistencies, 
ambiguities, gaps, and non-testable statements in the specification. It 
can provide early feedback to the editors regarding areas that need 
attention. An added benefit is that the assertions are usable as input 
to test development efforts."

For the record, I do believe we should form a TC and I don't believe 
that we need a workshop before we can do so, though I think there would 
be great benefit to some kind of meeting once we've made a start.

I'm looking forward to participating, and will encourage other members 
of my team to do so too.


Patrick Curran
Manager, Java SE Conformance Test Development, Sun Microsystems

========

[1] QA Framework: Specification Guidelines: 
http://www.w3.org/TR/qaframe-spec/
[2] Test Development FAQ: http://www.w3.org/QA/WG/2005/01/test-faq
[3] Variability in Specifications: http://www.w3.org/TR/spec-variability/
[4] Test Metadata: http://www.w3.org/TR/test-metadata/