[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: [office] Formula: test cases
I agree with Patrick. The normative basis for conformance with a standard must be entirely in the text, not in test cases. Test cases can be quite useful, but it is difficult to demonstrate that a test suite exhaustively exercises all the conforming variations supported by a standard or even exercises all the interesting/useful options. The validtation of a test suite opens up all sorts of new issues that may not even be relevant to the original standard. When we developed ISO/IEC 8879 (SGML), we recognized the difficulty of generating appropriate test suites and so chose to standardize only a testing and reporting methodology that would allow reasonable interpretation and comparison of results of test suites. This methodology is documented in ISO/IEC 13673:1994, "Conformance Testing for Standard Generalized Markup Language (SGML) Systems", available at http://www1.y12.doe.gov/capabilities/sgml/sc34/document/0131t.pdf. Several groups, including NIST and the former Omnimark Technologies, did publish very useful SGML test suites, but they were never candidates for formal standardization. Jim Mason -----Original Message----- From: Patrick Durusau [mailto:patrick@durusau.net] Sent: Thursday, March 29, 2007 10:39 AM To: OASIS Office Subject: [office] Formula: test cases Greetings! I am deeply uncertain about the test cases for reasons similar to my concerns about the "hidden" text. If the required behavior is sufficiently specified, then the test cases should not be adding anything to the standard. That is to say if I implement the normative text, it should not be possible to obtain a result that is inconsistent with the test cases. No doubt the test cases are very valuable as in drafting or post-adoption, implementors can test their implementations to insure that implementing the normative text does result in the correct outcome(s), but that really isn't part of the normative text. That is to say that I should not simply implement the test cases and think that I have implemented the normative text. If someone "cheats" in that fashion, they may well encounter an edge case that is not covered by the test cases but is covered by the normative language. So, test cases are invaluable, but I am leaning towards suggesting that they should not appear in the normative text of the standard. Actually I am not entirely sure they should even appear in a non-normative annex. In part because if there is any conflict between the normative text and the test cases, which controls? The normative language or the test cases? That is one reason why standards strive to only say any rule once and only once. Not entirely possible if you want a readable result but it is something that is a good rule to follow in general. That reduces the grounds for reaching different interpretations. Actually I would argue that if I need the test cases to understand the normative text, that is a good sign there is a problem with the normative text. I am starting a slow read of the formula proposal this weekend and will post comments on specific sections as I reach them. This and the prior post on notes, rationales, etc., are general comments that I won't repeat as I encounter those aspects of the various sections. Hope everyone is having a great day! Patrick PS: Actually the test cases offer an interesting way to proof the normative text. Remember the "check yourself exercises" in textbooks? Simply cover up the results of the test cases and after reading the normative text, see if you get the same results as are set forth in the test cases. If you don't, it might indicate a problem with the normative text. Will take longer but should result in a very clean normative text. -- Patrick Durusau Patrick@Durusau.net Chair, V1 - Text Processing: Office and Publishing Systems Interface Co-Editor, ISO 13250, Topic Maps -- Reference Model Member, Text Encoding Initiative Board of Directors, 2003-2005 Topic Maps: Human, not artificial, intelligence at work!
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]