RE: [oic] "Test Suite": A Rose by Any Other Name ...

[robert_weir@us.ibm.com]

"Dennis E. Hamilton" <dennis.hamilton@acm.org> wrote on 11/14/2008 
02:17:15 PM:

> Please respond to dennis.hamilton
> 
> Rob,
> 
> I am trying to find a way to navigate between
> 
>     It is the purpose of the OIC TC to produce materials 
>     and host events that will help implementors create 
>     applications which conform the ODF standard and which
>     are able to interoperate. 
> 
> and 
> 
>     The following activities are explicitly not within the 
>     Scope of the OIC TC: 
> 
>     1. Acting as a rating or certifying authority or agency
>        for conformance of particular ODF implementations; 
>     2. Authoring or distributing software that tests the 
>        conformance or interoperability of ODF 
>        implementations. 
> 

The thrust of the first prohibition is that we design, document and author 
tests, but we do not rate implementations.  This is a common division of 
labor between standards organizations and certification labs.  One body 
designs the tests, another body deals with the execution of the the tests.

The second prohibition prevents us from straying into the creation of 
elaborate test harness code.  We need to take an approach, with OASIS, 
that is documentary in nature, not source-code based.  So this leads us to 
writing test documents and specifications that tell how the test documents 
should behave.  Other, 3rd party organizations, say the ODF Toolkit Union, 
might take the deliverables of this TC and write software that automates 
the actual tests.


> In context, I think the charter's use of assessment and 
> test is perfectly fine:
> 
>    produce a comprehensive conformity assessment methodology 
>    specification ...
> 
>    which enumerates ... specific actions recommended to test 
>    each provision
> 
> My concern with test and test suite has to do with a tendency to assume 
an
> automated mechanism and that people also tend to think of this in terms 
of
> acceptance tests and conformance tests. 
> 

Well, don't assume that then.  Very few ISO standards have test 
automation.  In fact most ISO standards have nothing to do with software. 
But they can have conformance assessment methodology specifications that 
describe in detail how to test conformance.  There is not even a blush of 
automation hinted.  It could be a purely manual test. 

> I was looking for some alternative that would not invite such leaps yet 
the
> documents and fragments and materials we use to illustrate a feature in
> terms of the specification and interoperability would be understood for
> their value.
> 
> Finally, I am mindful that we are looking for pain points that are big
> payoffs for those seeking interoperable use of ODF-conformant products 
and
> that this is going to be a progressive activity.
> 
> That's what had me want to be careful about putting too much weight on 
test
> documents and test suites.  (I must also re-read the charter regularly.)
> 

But we shouldn't read too little into the charter.  Conformance test 
suites are certainly within scope.  A "specific action" as used within a 
conformance test might be, for example, "load document 23 and verify that 
the output displayed shows feature X".

> In closing, I notice that this scope element is more specific about test
> documents for  particular purpose:
> 
>   3. To select a corpus of ODF interoperability test documents, 
>      such documents to be created by the OIC TC, or received 
>      as member or public contributions; To publish the ODF 
>      interoperability test corpus and promote its use in 
>      interoperability workshops and similar events; 
> 

The charter calls out two different things:  conformance assessment and 
interoperability test suites.  These are intended to be distinct. 

> For that one, I guess we need to wait and see what form these test 
documents
> take and how they are presented.
> 
> Here's the big issue though: How control of features is presented in an
> authoring tool and processor for control of the document is not in terms 
of
> the format but user-interface, presentation, and whatever the 
organization
> of menus and dialogs are (or whatever there is instead of a UI, even). 
None
> of this is specified for ODF, of course, and we have to make recommended
> actions for testing and a "corpus of ODF interoperability test 
documents"
> usable despite that. That makes our tests very different from tests that 
are
> used in software development and what is expected when a test document 
or a
> test suite (even unit tests) is exercised.
> 

ODF is a file format specification, not a word processor specification. So 
the user interface of the application is not specified.  But we can assume 
operations like "load" and "display", and I think that is all we really 
need to test conformance.

> So mostly I wanted something that was specific enough but jars us and
> observers loose from preconceptions that might not be applicable.  I 
think
> it is worth exploring.  I am not challenging the charter.
> 

Since the number of people who actually know what an ISO conformance 
assessment methodology specification is, is approximately zero, I think it 
is safe to say that we'll need to educate everyone on what we're doing.

I might help to execute on the first item in the Scope statement:

"Initially and periodically thereafter, to review the current state of 
conformance and interoperability among a number of ODF implementations; To 
produce reports on overall trends in conformance and interoperability that 
note areas of accomplishment as well as areas needing improvement, and to 
recommend prioritized activities for advancing the state of conformance 
and interoperability among ODF implementations in general without 
identifying or commenting on particular implementations; "

That gives ample opportunity to state in more detail what prioritized 
steps we intend to undertake, and to do this in the context of our 
observations on the state of interoperability today. 

-Rob

>  - Dennis
> 
> 
> -----Original Message-----
> From: robert_weir@us.ibm.com [mailto:robert_weir@us.ibm.com] 
> Sent: Thursday, November 13, 2008 18:47
> To: oic@lists.oasis-open.org
> Subject: Re: [oic] "Test Suite": A Rose by Any Other Name ...
> 
> > Please respond to dennis.hamilton
> > 
> > I've been training myself not to use test and test suite when 
> > speaking about what we are producing as the work of the ODF 
> > Interoperability and Conformance TC.  My purpose is to avoid any 
> > suggestion of comparison, assessment, acceptance mechanisms, and the
> > use of text fixtures and evaluative suites of some sort. 
> [ ... ]
> 
> You can always fall back on the language of the TC's charter:
> 
> "To collect the provisions of the ODF standard, and of standards 
> normatively referenced by the ODF standard, and to produce a 
comprehensive 
> conformity assessment methodology specification which enumerates all 
> collected provisions, as well as specific actions recommended to test 
each 
> provision, including definition of preconditions, expected results, 
> scoring and reporting;"
> 
> But note that even there we use the word "test" and "assessment".  It is 

> certainly within scope, and certainly the intent of the charter to allow 

> the TC to publish written instruments that allow a third party 
> (implementor or otherwise) to test the conformity of an ODF 
> implementation.  Calling this a "test suite" does not seem to be an 
abuse 
> of the term, at least not to me.
> 
> -Rob
> 
> [ ... ]
>