Re: [oic] some thoughts on test documents

[robert_weir@us.ibm.com]

Bart Hanssens <bart.hanssens@skynet.be> wrote on 10/25/2008 03:40:46 PM:

> Some thoughts and questions on getting started with the test documents.
> 
> Dave, Rob and others already made several detailed comments about this
> topic on the formation mailing list, see also
> http://sites.google.com/a/odfiic.org/tc/Home/odf-interoperability-
> and-conformance
> 
> IMHO, we should start with the atomic documents.
> Now, the opendocumentfellowship has already created a test suite:
> http://develop.opendocumentfellowship.com/testsuite
> 
> Maybe we can reuse it (if the fellowship agrees with this) or do
> something similar, minus the comments on specific implementations, but
> adding remarks on ODF or the testing itself (like "spec not clear")
> 

http://develop.opendocumentfellowship.com/testsuite/

It would be good to get the Fellowship to contribute their test suite to 
the TC so we can work with in.  I know that it is under a Creative Commons 
Attribution 2.5 licence, but this is weaker than the required OASIS 
Feedback Licence, since CC waives copyright only, but does not grant any 
protection against patents.


> We could create a Wiki with a page per test document, and an upload
> directory on the OASIS website (as mentioned during the TC conf call,
> the OASIS Wiki doesn't support file upload) like:
> 
> ODF_1_1/
> - atomic/
> -- 5_Para_Elements/
> --- 5_3_1_Note_Element.odt
> - complex/
> ...
> ODF_1_2/
> - atomic/
> 
> 
> And if anyone feels like creating a document for a specific item, he/she
> can mention it on the wiki, create a test and upload the file.
> 
> Once a week/month/..., a snapshot can be made and archived in a ZIP file
> for convenient downloading.
> 

That's sounds like a reasonable way to structure it.  But it would be good 
to first define exactly what we mean by a "test case", maybe agree on a 
single test case as an example.

In my mind any test cases should have:

1) A unique name or identifier
2) An indication of what version(s) of ODF the test case applies to
3) A list of pre-conditions for executing the test.  What must be done 
first to set up the test case environment?  In most cases it will be 
simple:  load the document in an ODF editor.  But even simple cases may 
presuppose the installation of a particular font, for example.  There may 
be a common set of pre-conditions which all or most test cases.  We can 
list those once.  But we may have test cases that involve special 
requirements, say a connection with an external database which may require 
that such a database is first created.
4) A list of post-conditions used to judge whether the test case passed. 
This could be described in English, or in a formal language.

As you see this is similar to how unit tests are commonly done:

double sqrt(double x)
{
assert(x>=0)

double ret = doCalculation()

assert(ret*ret==x)
}

(and yes, I know one shouldn't really do equality tests of floating point 
calculations...)

This isn't the only model for how we can do test cases, but that is one 
pattern of testing which has proven itself useful in other contexts.

> 
> This doesn't necessarily mean going through the whole spec in sequential
> order, we can start with areas that are known to have issues (I would
> hint: forms, gradients, slide effects, charts...) or easy to test
> 
> 
> Questions:
> - do we create atomic tests with an office suite, or craft it by hand to
> make it really as small as possible ?
> 

It might be easier to start with output from, say OpenOffice.  But we 
could have a tool that then strips out all the unnecessary markup in order 
to reduce the file to the bare essentials.  We should also examine the 
markup by hand and verify that it indeed is correct according to the ODF 
standard.  So at the very least we should validate it with an XML 
validator.

> - should we do ODF 1.0, now that many (all ?) vendors use 1.1 or 1.2
> draft ? Personally, I think we should start with issue-prone parts of
> 1.1 and 1.2's OpenFormula.
> 

There is a lot of overlap between the releases.  So most test cases we do 
will apply to all three versions.  Of course, open formula would only 
apply to ODF 1.2.  But I agree that the "issue-prone" areas are the ones 
to start with.

> - who will run the tests and report the results, so that the OIC can
> create a summary and general recommendations ? Per charter, the OIC
> won't be commenting or identifying implementations (at least not in
> reports, but isn't a wiki some kind of report ?)
> 
> 

That's the sensitive issue.  We, as an OASIS TC, cannot issue a report on 
implementations and their interoperability results.  But as individual 
members, especially members who are also ODF vendors, I'd expect that we 
all run these test cases ourselves against our own implementations.  If an 
individual member then wishes to report on their own product's results, 
then that is fine.  And if a third part, having access to our test suite, 
wishes to test and compare multiple implementations, then that is fine as 
well.

Maybe we want to talk to the OpenDocument Fellowship on this?  Maybe they 
would relinquish the test case creation work to us, and instead take on 
the testing of the applications against our test suite?  I think that 
gives a needed separation of interests.

-Rob