Re: [office] Formula: test cases

["David A. Wheeler" <dwheeler@dwheeler.com>]

Patrick Durusau:
> >>I am deeply uncertain about the test cases for reasons similar to my 
> >>concerns about the "hidden" text. If the required behavior is 
> >>sufficiently specified, then the test cases should not be adding 
> >>anything to the standard. That is to say if I implement the normative 
> >>text, it should not be possible to obtain a result that is inconsistent 
> >>with the test cases.

Hmm. Since the document currently defines the test cases as being part (though not the whole) of the normative text, it is vacuously true that implementing the normative material cannot produce a result inconsistent with the normative material.  So sure, it's not possible already!

But I know what you mean, so let's carry on with what you mean.

The problem is that without the test cases, people would look at the text, and everyone would agree "yes, that's unambiguous".  Then they'd go off and produce INCOMPATIBLE results.

The perfect solution, of course, is to write perfect text, text that is NEVER ambiguous. I have found no magical way of doing so. Review by large groups helps but doesn't eliminate all ambiguity.  Unfortunately, all we had were fallible human beings, and a language (English) that is not designed to force out all imprecise statements.  Math helps, but not enough; traditional mathematical notation makes assumptions too. Using a second mechanism (test cases) as a way to eliminate most ambiguities turns out to be an incredibly powerful mechanism.

> Yes, and you will recall that I raised my misgivings about the inclusion 
> of test cases with you quite some time ago.
> 
> We disagree then and if I understand your "English text" it appears we 
> continue to do so. ;-)

:-).
 
> BTW, I am reviewing another standards proposal that takes the out "its 
> too complicated to explain" approach. ;-) You can imagine my reaction to 
> such statements.

EEK! I hope you'll agree that including test cases is way better than THAT :-).

> I assume you kept a list of cases where the normative text is not 
> controlling and that one has to rely on the test cases?

I'm not aware of any such cases.  English is marvelously adept at being able to make statements that APPEAR precise, without actually BEING precise.  Indeed, I think I could make a case that that is the primary purpose of English :-).

> Err, but test cases remove only some ambiguity as I pointed out in my 
> initial post. If you really want to remove *all* ambiguity, then you 
> would have to formally prove the results of each formula. What test 
> cases do is remove ambiguity that you thought about providing a test 
> case for.
> 
> That is not the same thing as removing Ambiguity (writ large).

If you mean "perfectly removing ambiguity", I agree - but I didn't have the money to do formal proofs at that level :-).

But I _disagree_ if you mean "test cases do not help remove ambiguity".  They do.  We have a lot of experience with text that seemed perfectly clear... until the test case was added :-).


> Err, are you sure you want that rule? That TEST CASES control?
> 
> In that case I can write an application that gives the specified results 
> for the enumerated test cases and utterly random results for any other input.

Nope - see below.

> The problem is that a test case cannot be generalized, well, unless you 
> want to enumerate all possible inputs and results, which would be rather 
> lengthy for the set of integers.

Er, no.  If ONLY the test cases were normative, that'd be true, but that'd be crazy to try.  I'm NOT arguing that ONLY the test cases are normative.  Test cases BY THEMSELVES certainly DO have the problem you've stated, and are obviously completely inadequate for specification.

But this is NOT just a bunch of test cases.  There is normative semantic text, that STRIVES to be unambiguous... backed up by test cases that winnow out any ambiguities left over in the text.  The test cases make it possible for us to clarify the meaning of the text in a way that's difficult to do otherwise.  E.G., "Hmm, SIN doesn't identify if it takes degrees or radians! Oops, yes it does, because the test cases cannot be correct if interpreted as degrees."  And in my experience, any significant amount of "normative" text has ambiguities.

The goal is not either/or.  We need good normative text, and I'm looking forward to improvements that you suggest. But the goal is to minimize ambiguity, and I believe that can be best done by combining what normative text can do (defining the general case) along with what test cases can do (it can be EXTREMELY precise about the correct result for specific inputs).  By COMBINING these specification approaches, general-case normative text and specific-case test cases, we achieve a general yet far less ambiguous result.

> >>I am starting a slow read of the formula proposal this weekend and will 
> >>post comments on specific sections as I reach them.

Great!!

> Sorry, what you have is *not* a conformance test.

Correct; the test cases are NOT a full conformance test. Sorry that there was this misunderstanding... I was _contrasting_ with the experience of full conformance tests, not claiming that the test cases here are a full conformance test. It's a "quickie" test, and without anything better I'm sure it will be useful, but its REAL purpose is to squash latent ambiguities that, despite our best efforts, will always hide in any English prose.

An application has to conform to the spec as written - NOT just the test cases.  It is the COMBINATION that is powerful.


> So, let's at least disagree on what is at issue:
> 
> 1. How to specify normative behavior? (I would not use test cases.)
> 
> 2. Where should test cases go? (I would not include test cases in the 
> standard.)

I would tweak that.  I _agree_ that test cases should not be the ONLY thing that specifies a function.  Instead, the draft includes test cases as PART of the specification of normative behavior, because doing so turns out to eliminate the many latent ambiguities that seem to always crop up in English prose based specifications.

> Please do not take any of my comments as doubting the usefulness of the 
> test cases. I am sure they are very useful but I do disagree on the role 
> you want to assign them in this standard.

I understand.  I knew going in that this was an unusual approach.

I ask that you (and others) keep an open mind, and take a look at it.  It's obviously POSSIBLE to move the test cases into some sort "TestCase" style that is part of a rationale document and not part of the final specification.  As long as there's a single master editable document, from which the rationale and specification are generated, it'd certainly be workable.  But that does not make doing so a good idea.

So take a peek at how it ACTUALLY works, before judging it.  Thanks.

--- David A. Wheeler