Re: [oiic-formation-discuss] Acid Tests

["Matthew Reingold" <matthewreingold@gmail.com>]

Rob, I agree completely. One of the benefits of good standards is that
vendors who wish to participate are doing so purely for their own
benefit and advanced knowledge. Even vendors as large as MS and 3m
recognize the advanced benefits of conformance at my job....and it is
definitely and perpetually ongoing. Heck we are working on ODF
sub-versions for 1.xxx let alone 2.xxx in the forseeable future.

i think a web based test if possible would be convenient for some testing.



On 6/12/08, robert_weir@us.ibm.com <robert_weir@us.ibm.com> wrote:
> jose lorenzo <hozelda@yahoo.com> wrote on 06/12/2008 05:28:45 PM:
>
>
>>
>> I agree with making this a continual effort.
>>
>> Concerning point 4...
>> >> 4) The press will pick up on this, but we should
>> let others deal with the name and shame
>>
>> We (not sure who "we" is) should at all times be clear
>> to the public about the significance of what is
>> getting accomplished with the work and results leading
>> up to the report. The process should be formally
>> recognized somewhere tightly associated with ODF and
>> its conformance clause.
>>
>
> By "we" I refer to the eventual members of the proposed TC.
>
>> As for the other four points, I would like to find a
>> way to tap into the "community" to identify, design,
>> build, maintain, etc, acid and other types of tests,
>> on an ongoing basis. This is valuable, practical,
>> desired (but I haven't formally polled), and quiets
>> claims that too elaborate or extensive a test would
>> consume too many resources. In fact, forget about the
>> complaints, the fact is that the more people helping,
>> the more coverage is done, which has the side-effect
>> of *countering* efforts by vendors to *code to the
>> test suites*.
>>
>
> Of course, one way of tapping into the community would be to propose a new
> TC in OASIS that would create these test cases, and then invite the public
> to discuss the idea, and hopefully to join the new OASIS TC.  That's what
> a number of us in OASIS thought a few weeks ago, and that why this
> discussion list was created.
>
> I'm not sure where you are seeing criticism of "too elaborate or extensive
> a test", but if you are sensing that from me, then let me explain.  A see
> the atomic tests as being the comprehensive tests.  That the traditional,
> ponderous way of doing this kind of testing.  Thousands of test cases,
> thousand of test files.  OASIS has done with with XML and XSLT, and the
> W3C does this kind of test frequently.
>
> There is no good reason that I can think to duplicate such an effort in an
> Acid-style test.  However, there is a good argument for an Acid-test that
> exercises a high priority set of 10 or 50 important features.  The
> suggested size limit is not because of any imagined resource limitation in
> creating a larger test.  It was purely out of a desire to keep the message
> consumable by the press and understandable by the public.  A 20-feature
> Acid test that people can understand may have a greater impact, and get a
> greater response from vendors, then a 1,000 feature that no one gets.
> Maybe it grows incrementally?
>
>> Potentially tapping into the public brings up a host
>> of questions, which I will kindly let others ask.
>>
>> To return to your point 4, I suppose third parties or
>> the users/buyers themselves (or someone on their
>> behalf) would set requirements like " 'officially'
>> failed tests must be passed within 60 days or else X."
>>
>> I think though that we should dive into this area to
>> an extent and not leave it totally in the hands of
>> third parties. We are in a unique position to give
>> guidance on the significance of one test failed vs.
>> another vs. many tests in series, independently, etc.
>> For example, what happens when before 60 days, the
>> failure coming due is fixed but there is a regression
>> of some sort? We can serve end users well by giving
>> guidance (recipe/algorithm, table, framework...) since
>> we can/will build a formal testing framework and
>> can/will keep it categorized/organized (as mentioned,
>> potentially tapping into the public for aid).
>>
>
> I think the guidence and perspective is what I was calling an
> "Implementation Report".  This is a report authored by the TC that gives a
> review of the current state of implementations, in terms of versions
> supported, feature completion, interoperability, etc.
>
> Keep in mind also that a good test suite will improve interoperability in
> ways that you may not even be aware of.  For example, I would expect that
> ODF implementors would run the latest test suite on their product before
> it is released.
>
>
>
>> Alright, "we" might not be the right body to handle
>> this. The point is that those involved most closely
>> with organizing, designing, etc, tests should offer
>> these guidances, resources, and official
>> interpretations to the public.
>>
>> These situations discussed above and others related to
>> (acid or other) testing should be tied to
>> "conformance" officially, clearly, and precisely.
>>
>> Yes? No?
>>
>
> It works like this:  A standard has formal provisions, which are defined
> by normative statements.  These provisions express requirements,
> recommendations, permissions possibilities and capabilities.  Conformance
> is conformance to those provisions.  That is the dependency direction.  We
> can't change the conformance definition of ODF.  But the test cases can
> (and should) be traceable back to the provisions of the ODF standard.
>
> -Rob

-- 
Sent from Gmail for mobile | mobile.google.com

Matthew B Reingold
Cell: 847-553-8097