RE: [office] Formula: test cases

["Mason, James David \(MXM\)" <masonjd@y12.doe.gov>]

I agree with Patrick. The normative basis for conformance with a
standard must be entirely in the text, not in test cases. 

Test cases can be quite useful, but it is difficult to demonstrate that
a test suite exhaustively exercises all the conforming variations
supported by a standard or even exercises all the interesting/useful
options. The validtation of a test suite opens up all sorts of new
issues that may not even be relevant to the original standard.

When we developed ISO/IEC 8879 (SGML), we recognized the difficulty of
generating appropriate test suites and so chose to standardize only a
testing and reporting methodology that would allow reasonable
interpretation and comparison of results of test suites. This
methodology is documented in ISO/IEC 13673:1994, "Conformance Testing
for Standard Generalized Markup Language (SGML) Systems", available at
http://www1.y12.doe.gov/capabilities/sgml/sc34/document/0131t.pdf.

Several groups, including NIST and the former Omnimark Technologies, did
publish very useful SGML test suites, but they were never candidates for
formal standardization.

Jim Mason

-----Original Message-----
From: Patrick Durusau [mailto:patrick@durusau.net] 
Sent: Thursday, March 29, 2007 10:39 AM
To: OASIS Office
Subject: [office] Formula: test cases

Greetings!

I am deeply uncertain about the test cases for reasons similar to my
concerns about the "hidden" text. If the required behavior is
sufficiently specified, then the test cases should not be adding
anything to the standard. That is to say if I implement the normative
text, it should not be possible to obtain a result that is inconsistent
with the test cases.

No doubt the test cases are very valuable as in drafting or
post-adoption, implementors can test their implementations to insure
that implementing the normative text does result in the correct
outcome(s), but that really isn't part of the normative text. That is to
say that I should not simply implement the test cases and think that I
have implemented the normative text. If someone "cheats" in that
fashion, they may well encounter an edge case that is not covered by the
test cases but is covered by the normative language.

So, test cases are invaluable, but I am leaning towards suggesting that
they should not appear in the normative text of the standard. Actually I
am not entirely sure they should even appear in a non-normative annex. 
In part because if there is any conflict between the normative text and
the test cases, which controls? The normative language or the test
cases?

That is one reason why standards strive to only say any rule once and
only once. Not entirely possible if you want a readable result but it is
something that is a good rule to follow in general. That reduces the
grounds for reaching different interpretations.

Actually I would argue that if I need the test cases to understand the
normative text, that is a good sign there is a problem with the
normative text.

I am starting a slow read of the formula proposal this weekend and will
post comments on specific sections as I reach them. This and the prior
post on notes, rationales, etc., are general comments that I won't
repeat as I encounter those aspects of the various sections.

Hope everyone is having a great day!

Patrick

PS: Actually the test cases offer an interesting way to proof the
normative text. Remember the "check yourself exercises" in textbooks? 
Simply cover up the results of the test cases and after reading the
normative text, see if you get the same results as are set forth in the
test cases. If you don't, it might indicate a problem with the normative
text. Will take longer but should result in a very clean normative text.

-- 
Patrick Durusau
Patrick@Durusau.net
Chair, V1 - Text Processing: Office and Publishing Systems Interface
Co-Editor, ISO 13250, Topic Maps -- Reference Model
Member, Text Encoding Initiative Board of Directors, 2003-2005

Topic Maps: Human, not artificial, intelligence at work!