Test Cases

["David A. Wheeler" <dwheeler@dwheeler.com>]

Here's some info on test cases, and some commentary.

I earlier said:
> * Test cases: Should we include test cases in the spec? Wheeler STRONGLY recommends it.  Including test cases eliminates many problems of ambiguity in the text; Wheeler believes it is VERY difficult to write unambiguous text, but that well-written text accompanied by some clear test cases can together create an unambiguous specification that is easier to create and to read.  In addition, including test cases makes it much easier to test and assure compliance in implementations.  OpenFormula did this successfully, and the KSpread developers seemed to find the test cases useful.
>   

Mecir notes: "Definitely yes... they were mightily useful."  And I agree!

Rob Weir said:
 > I think it is also very difficult to write good, comprehensive
 > test cases.  So we have our work cut out for us either way.

You're absolutely right.  But by COMBINING text AND test cases, we're 
much MORE likely
to have a good result.  For example, it was hard to write MOD in a clear 
way.
Examples with varying locations of positive and negative numbers were MUCH
easier to explain than trying to explain it SOLELY by text alone.
Not only that, but we could immediately try out the
test cases on various implementations, to make sure we weren't creating
unreasonable requirements.

I also think that having test cases is REALLY motivating for developers.
If you can just load a file and see EXACTLY what level you're at... and what
needs changing to meet the next level... it's really motivating.  In a 
GOOD way!

 > Are you seeing the test cases as being normative?

Yes, absolutely.  They'd be PART of the definition in the spec,
in my thinking.

 > Keep in mind that OASIS does require that this specification be
 > released in HTML and PDF format, so the normative specification
 > is required to be textual, i.e., printable.  But interlinear
 > test cases, as text in the spec, is a great idea.

In OpenFormula, the test cases are presented as tables in a specific format
as part of the (textual) spec. We then had a program that walked through the
text looking for every table in that format, and then it automagically 
assembled a
test spreadsheet from those tables.  You simply edited
the TEXT of the specification, and the test spreadsheet was created for you.


--- David A. Wheeler