RE: [oic] "Test Suite": A Rose by Any Other Name ...

["Dennis E. Hamilton" <dennis.hamilton@acm.org>]

Andreas,

I hope that I cleared up some of this in my reply to Rob Weir,
http://lists.oasis-open.org/archives/oic/200811/msg00024.html

I think the whole point is expressed in the Overview statement on the OIC TC
Page:

   The OASIS OIC TC helps implementors create applications
   that conform to the OpenDocument Format (ODF) OASIS 
   Standard. ... The OIC TC works to ensure that the growing
   number of ODF-compliant applications are able to 
   interoperate and conform to the standard.

So I think of us as aiding and abetting interoperability.

I think an important part of the approach to this is given in the first
scope item of the TC charter:

   1. Initially and periodically thereafter, to review the 
      current state of conformance and interoperability 
      among a number of ODF implementations; To produce 
      reports on overall trends in conformance and 
      interoperability that note areas of accomplishment 
      as well as areas needing improvement, and to recommend
      prioritized activities for advancing the state of 
      conformance and interoperability among ODF implementations
      in general without identifying or commenting on particular 
      implementations; 

With regard to feature confirmation, I am thinking not about whether a
feature exists in ODF but whether (1) an implementation can exhibit such a
feature in the ODF it produces and (2) that it deals with the feature in ODF
it accepts in an appropriate manner, depending on whether (2a) it supports
the feature or (2b) it does not.  And then, depending on how it supports the
feature in documents it accepts, (3) how it does or does not preserve the
feature-related ODF elements in its output.  I suppose one also wants to
also entertain consideration of (4) ways that substitute behavior might be
obtained/controlled when (2b) holds.

I'm thinking of this as early work oriented on features where
interoperability is prized and perhaps critical.  I assume that we want to
look at these conditions in cases where interoperability of particular ODF
features is found to be most important in the review of the current state of
conformance and interoperability.

I want to some how differentiate these kinds of ways to test for conformance
from the usual notion of test documents and test suites that apply to an
overall application and go far beyond the provisions of the format.  Yet
interoperability does depend on some qualities of the application software.
I'm concerned that we might see ourselves as producing acceptance tests or
certifying conformance in any way, and I want to avoid the notion that the
"test suite" is something to pass, rather than something to learn from and
use to assess how one deals with particular ODF features.  

Have I clarified my concern, or does this seem even murkier?

I am using different terms as a way of attempting to get my head around all
of this.  I want to avoid automatic assumption that I have a good tacit
understanding of the usual words around tests and testing and, in
particular, that those tacit understandings make sense in the kind of
interoperability and conformance determination there can be with respect to
ODF-supporting software implementations.

 - Dennis
 

-----Original Message-----
From: Andreas J Guelzow [mailto:aguelzow@math.concordia.ab.ca] 
http://lists.oasis-open.org/archives/oic/200811/msg00020.html
Sent: Thursday, November 13, 2008 19:40
To: oic@lists.oasis-open.org
Subject: Re: [oic] "Test Suite": A Rose by Any Other Name ...

On Thu, 2008-11-13 at 16:48 -0800, Dennis E. Hamilton wrote:
http://lists.oasis-open.org/archives/oic/200811/msg00016.html
> I've been training myself not to use test and test suite when speaking
> about what we are producing as the work of the ODF Interoperability
> and Conformance TC.  My purpose is to avoid any suggestion of
> comparison, assessment, acceptance mechanisms, and the use of text
> fixtures and evaluative suites of some sort.

Wow, and I thought the whole point was to _compare_ how various
processors handle the standard and to _assess_ the level of conformance.
> 
> There are multiple reasons for this, in part because what we are
> addressing is far removed from what the particular manifestation might
> be in a viewer and especially in a processor.
> 
> So far, my favorite expression is feature-confirmation documents,
> feature-confirmation exercises, and feature-confirmation
> demonstrations. 

Nice terms, but of course I have no idea what those terms are in fact
supposed to mean. Of course I don't know whose features you are talking
about. I would assume that these are document features embodied in ODF.
But of course then there is little point of confirming that those
features in fact exist.

>  I'm very big on confirmation tests (but am training myself to avoid
> test in that case and say confirmation case).  My inclination is to
> speak of feature-confirmation cases and interoperability-confirmation
> cases. 
[ ... ]