Re: [oiic-formation-discuss] Draft Interoperability and Conformance TCformation proposal (0.2)

[robert_weir@us.ibm.com]

Shawn <sgrover@open2space.com> wrote on 07/26/2008 01:56:23 AM:

> 
> Peter Dolding wrote:
> > Production of software not needed for testing also not fine.
> 
> Peter, this is the sticking point.  This TC will NOT be doing any tests. 

>     Only providing the DEFINITION of what tests can be done to check 
> conformity and interoperability.  The intent is that these definitions 
> would then be used by third party testing shops, or developers.
> 
> I see the TC as defining the requirements (in terms of the traditional 
> software life cycle), then handing the resulting documents to developers 

> (by making them available) who will then write code to meet the 
> requirements.
> 
> Of course, this is only my take on the conversations that have taken 
> place.  Perhaps I've missed something and am wrong.
> 

Probably worth being explicit about what we mean by "doing tests".  One 
way in which it could be done would comprise at least the following steps:

1) A statement of requirements and recommendations (collectively called 
"provisions") which are in the existing ODF standard, and standards 
referenced by ODF.

2) Collection of all such provisions into a categorized list, where each 
provision is given a ID.

3) The statement of one or more specific test assertions for each 
provision.  For example, if a provision says that a given value must be 
between 1 and 5, we might have two test assertions: 
assertNotError(value=3) and assertError(value=6).  Each test assertion 
would be also given an ID, perhaps correlated with the corresponding 
provision's ID.  So test ID 23.4 would correspond to provision 23, test 
assertion 4.

4) The creation of ODF instance documents which correspond to each of the 
test assertions, along with an image or descriptive text that would tell 
how that document should render in a conformant ODF applications. So, we 
would have an ODF test document which has the legal value of 3, and 
another ODF document with the illegal value of 6.  Each document would 
have a file name corresponding to its test assertion ID.

5) The creation of a document which enumerates all of the test assertions, 
categorized functionally, along with a statement of what preconditions are 
necessary to perform each test, what the expected results are, which test 
files are needed for each test, and how the results are to be reported.

6. A test harness that hooks into OpenOffice as a plugin and presents a UI 
that allows the user to select a range of tests to execute.  The selected 
tests are automatically loaded, one-by-one, and a scoring UI is given to 
the tester where they can score each test as pass or fail and enter text 
remarks.  At the end of the test session a report in the required format 
is written out in PDF format.

7. Someone executes a set of tests against OpenOffice and reports it 
publicly, with the statement that OpenOffice conforms or does not conform 
to the ODF standard.


My intent is that steps 1-5 would be in scope for the proposed TC, and 
steps 6 and 7 would be explicitly out of scope.


If in the future ODF defines macros and scripts, then this framework would 
still apply.  The TC could write "software" necessary to create an ODF 
instance document that expresses a test assertion.  But they would not be 
writing test harness software, nor would they reporting on any 
implementation's test performance. I think I can adjust the language to 
make that clear.


-Rob