The First Wave of Decisions: Trial Answers

["Dennis E. Hamilton" <dennis.hamilton@acm.org>]

In the OIC-TC Call on 2008-12-17, I said I would like to create a Wiki page
on the six questions on David Marston's list (below and on page 9 of his
slide set).  On second thought, I suggested we work on the list until we
have something closer that we might put on a Wiki page and refine over time.
Here are my tries at the six questions:

	1.	What Class(es) of Product are to be tested? 
	2.	How to deal with profiles, SHOULD statements, optionality,
etc. 
	3.	Will we write Test Assertions? 
	4.	How will we take contributions? 
	5.	What is the policy on approvals and challenges? 
	6.	Will there be a mix of manual and automatable test cases?

1. WHAT CLASSES OF PRODUCTS ARE TO BE TESTED?

1.1 PRIMARY PRODUCT CLASS

1.1.1 The primary class of products are office-productivity software
products for word processing, presentations, and spreadsheets.  The products
are typically implemented by free-standing software programs and components
of office-software suites.  The products may also operate via on-line
application services that interact with web browsers or specialty user
agents.

1.1.2 The products accept, manipulate, and produce one or more of the
individual OpenDocument Format document types and their templates for word
processing (OpenDocument Text), presentations (OpenDocument Presentation),
and spreadsheets (OpenDocument Spreadsheet).  


1.2 SECONDARY DOCUMENT FORMATS

There are also secondary document formats that may be supported by software
in the primary product class.  The interoperable use of these free-standing
formats are secondary.

1.2.1 Additional OpenDocument Format document types and corresponding
templates that may also be supported include drawings (OpenDocument
Drawing), charts (OpenDocument Chart), and images (OpenDocument Image),
although the primary product interest is in the occurrence of these forms as
part of the primary OpenDocument Format document types.  Not all of these
document types are known to be implemented.  

1.2.2 In addition, there are free-standing documents and templates for
Math-ML (OpenDocument Formula) and there is a special word-processing master
document (OpenDocument Global Text).  Starting with ODF 1.2 there is also a
database application for local and remote databases, using a document format
(OpenDocument Base).

1.3 SECONDARY PRODUCT CLASSES

1.3.1 A secondary product class that may be of some concern consists of
converters by which ODF documents are imported into programs that are not
designed to support ODF, with or without separate translation into another
format that is supported.  This precedes conversion in the other direction
because the OIC has more to say about what the appropriate interpretation of
ODF format is that what the appropriate interpretation of other formats
might be.  This can also apply in the case of processors that are designed
for earlier or later versions of the OpenDocument specification, their
feature sets, and breaking changes between the specification.

1.3.2 An additional secondary class would consist of converters by which
documents in other formats are converted to a corresponding ODF format.

1.3.3 It is assumed that there is no major concern for special-purpose
software that emit ODF documents using a subset of ODF features for a
specific application.  Likewise, applications that use special-purpose ODF
documents as a form of custom data input are not of primary or secondary
concern.

1.4 CROSS-VERSION INTEROPERABILITY

The primary class tends to involve implementations at the same level of ODF
Specification (e.g., one of 1.0, 1.1, 1.2, ... ).  There are also
cross-version interoperability challenges.

1.4.1 An important consideration for the primary class is the ability to
successfully deal with documents that were produced in accordance with a
different version of the OpenDocument Format standard, whether earlier or
later than the version for which the primary software is designed.  

1.4.2 The ability to exchange documents back-and-forth between different
implementations that provide primary/default support to the same or
different versions of the OpenDocument Format standard are also of concern.


2. HOW TO DEAL WITH PROFILES, SHOULD STATEMENTS, OPTIONALITY, ETC.

2.1 PROFILES

There are no profiles established for ODF and the primary products that
support it.  This work on ODF interoperability and conformance may well
identify important profiles where there conformance is important in
specified broad or narrow contexts.

2.2 SHOULD STATEMENTS

There are SHOULD statements in the ODF specifications.  The initial
challenge is to determine whether a SHOULD item is supported and if not
supported, what are the consequences.

2.3 OPTIONALITY

For ODF, there is a broad range of optionality with regard to what features
are implemented or not, and what the behavior might be when a product
encounters a format feature that it does not support.


2.4 IMPLEMENTATION-DETERMINED QUALITIES

There are also features that are specified to have implementation-determined
qualities.  These cases need to be identified and there impact, if any, on
interoperability cases identified and understood.  Again, the first problem
is to identify what those are, how widespread are any implementations, and
what is the practical significance for interoperability.


3. WILL WE WRITE TEST ASSERTIONS?

The initial understanding is that we will do so.


4. HOW WILL WE TAKE CONTRIBUTIONS?

The submission and acceptance of contributions from OASIS Members is
established under OASIS Technical Committee procedures.  Submissions from
other members of the public and organizations not participating on the OIC
TC are generally required to be via acceptance of a submission agreement
associated with a public comment list, such as oic-comment.  

Other submission avenues may be more appropriate for submissions of test
documents and test information.  The form of such submission and the choice
of entry point (if different than oic-comment) remains to be determined.

5. WHAT IS THE POLICY ON APPROVALS AND CHALLENGES?

5.1 The procedure for compiling material in a form where it can be subject
to the different states public review, committee approval, and wider OASIS
approval remains to be determined.  There are packaging and organization
questions to be resolved.

5.2 In general, comments, including challenges to the correctness or
suitability of a test element, can be submitted at any time and there will
be a procedure to ensure that the TC is responsive and gives due
consideration to the comment.  There will be a public, on-line account of
all comments and their dispositions. 

5.3 In general, the OIC TC does not publish detailed, test-assertion level
assessment of products.  The test assertions and other criteria can be
challenged for their own accuracy and appropriateness.  Assessment of a
particular product is not within the scope of the OIC TC.


6. WILL THERE BE A MIX OF MANUAL AND AUTOMATED TESTS

6.1 By the nature of the primary product class and the distance of the user
and any user-interface model from the OpenDocument Format that is
accepted/produced by the product, there is necessarily a mix of manual and
automated tests.

6.2 It is desirable to organize tests and a corpus of test documents so that
automated tools can be used for selection of tests to apply in a given
scenario.  We will endeavor to produce such an organization and accompanying
computer-processable information.  We are not required to provide software
for the selection and customization of test material for a particular
product.

6.3 It is also desirable to have tools for assessing characteristics of
files that represent ODF document and inspecting those files for features
that are expected as a product of a given test activity.  Test assertions
may indicate what should be confirmed.  We are not required to delivery such
tools.

6.3 It will be necessary, in automating tests, to be able to add additional
provisions by which behavior can be injected into the operation of a
particular software product.   The means for accomplishing that is not a
work product of the OIC TC.

 - Dennis  


-----Original Message-----
From: Dennis E. Hamilton [mailto:dennis.hamilton@acm.org] 
http://lists.oasis-open.org/archives/oic/200812/msg00011.html
Sent: Sunday, December 07, 2008 20:48
To: oic@lists.oasis-open.org
Cc: david_marston@us.ibm.com; 'Bart Hanssens'
Subject: [oic] HOMEWORK: Getting-Started Questions from Expert Talk #1

In e-mail discussions with David Marston, our first Expert Talk guest, David
reminded us that there are a number of getting-started questions that it is
important for us to address so there is context for going deeper into topics
such as scenarios and metadata.

These look like the questions that we need to be asking ourselves and
answering to have a common foundation:  


1.	What Class(es) of Product are to be tested? 
2.	How to deal with profiles, SHOULD statements, optionality, etc. 
3.	Will we write Test Assertions? 
4.	How will we take contributions? 
5.	What is the policy on approvals and challenges? 
6.	Will there be a mix of manual and automatable test cases?

There are probably others, but these seem like a good start.

PROPOSAL: We take the discussion and agreement (or at least arriving at new
questions) of these start-up topics as homework before Expert Talk #2, some
time in January.  I will add this as an Action Item for all of us (we can
decide at the next call if we want to keep it) just so there is some place
to manage our attention.

RESOURCES:

David provided his slide set in a contribution to oic-comment:
http://lists.oasis-open.org/archives/oic-comment/200812/msg00002.html

Bart Hanssens has uploaded the set to our document store:
http://www.oasis-open.org/committees/document.php?document_id=30303&wg_abbre
v=oic
is the public page for downloading the presentation (here named
TestingThoughts1.zip).

Extract all of the pages into a single file-system directory. The
presentation starts at Overview.html.  The set should be viewable in any
contemporary web browser.

[ ... ]