RE: [oic] "Test Suite": A Rose by Any Other Name ...

["Dennis E. Hamilton" <dennis.hamilton@acm.org>]

Rob,

I am trying to find a way to navigate between

    It is the purpose of the OIC TC to produce materials 
    and host events that will help implementors create 
    applications which conform the ODF standard and which
    are able to interoperate. 
 
and 

    The following activities are explicitly not within the 
    Scope of the OIC TC: 

    1. Acting as a rating or certifying authority or agency
       for conformance of particular ODF implementations; 
    2. Authoring or distributing software that tests the 
       conformance or interoperability of ODF 
       implementations. 

In context, I think the charter's use of assessment and 
test is perfectly fine:

   produce a comprehensive conformity assessment methodology 
   specification ...

   which enumerates ... specific actions recommended to test 
   each provision

My concern with test and test suite has to do with a tendency to assume an
automated mechanism and that people also tend to think of this in terms of
acceptance tests and conformance tests.  

I was looking for some alternative that would not invite such leaps yet the
documents and fragments and materials we use to illustrate a feature in
terms of the specification and interoperability would be understood for
their value.

Finally, I am mindful that we are looking for pain points that are big
payoffs for those seeking interoperable use of ODF-conformant products and
that this is going to be a progressive activity.

That's what had me want to be careful about putting too much weight on test
documents and test suites.  (I must also re-read the charter regularly.)

In closing, I notice that this scope element is more specific about test
documents for  particular purpose:

  3. To select a corpus of ODF interoperability test documents, 
     such documents to be created by the OIC TC, or received 
     as member or public contributions; To publish the ODF 
     interoperability test corpus and promote its use in 
     interoperability workshops and similar events; 

For that one, I guess we need to wait and see what form these test documents
take and how they are presented.

Here's the big issue though: How control of features is presented in an
authoring tool and processor for control of the document is not in terms of
the format but user-interface, presentation, and whatever the organization
of menus and dialogs are (or whatever there is instead of a UI, even).  None
of this is specified for ODF, of course, and we have to make recommended
actions for testing and a "corpus of ODF interoperability test documents"
usable despite that. That makes our tests very different from tests that are
used in software development and what is expected when a test document or a
test suite (even unit tests) is exercised.

So mostly I wanted something that was specific enough but jars us and
observers loose from preconceptions that might not be applicable.  I think
it is worth exploring.  I am not challenging the charter.

 - Dennis
   

-----Original Message-----
From: robert_weir@us.ibm.com [mailto:robert_weir@us.ibm.com] 
Sent: Thursday, November 13, 2008 18:47
To: oic@lists.oasis-open.org
Subject: Re: [oic] "Test Suite": A Rose by Any Other Name ...

> Please respond to dennis.hamilton
> 
> I've been training myself not to use test and test suite when 
> speaking about what we are producing as the work of the ODF 
> Interoperability and Conformance TC.  My purpose is to avoid any 
> suggestion of comparison, assessment, acceptance mechanisms, and the
> use of text fixtures and evaluative suites of some sort. 
[ ... ]

You can always fall back on the language of the TC's charter:

"To collect the provisions of the ODF standard, and of standards 
normatively referenced by the ODF standard, and to produce a comprehensive 
conformity assessment methodology specification which enumerates all 
collected provisions, as well as specific actions recommended to test each 
provision, including definition of preconditions, expected results, 
scoring and reporting;"

But note that even there we use the word "test" and "assessment".  It is 
certainly within scope, and certainly the intent of the charter to allow 
the TC to publish written instruments that allow a third party 
(implementor or otherwise) to test the conformity of an ODF 
implementation.  Calling this a "test suite" does not seem to be an abuse 
of the term, at least not to me.

-Rob

[ ... ]