OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

oic message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [oic] some thoughts on test documents

Comments embedded below...

>>> On 10/25/2008 at 04:21 PM, in message
<robert_weir@us.ibm.com> wrote: 
> Bart Hanssens <bart.hanssens@skynet.be> wrote on 10/25/2008 03:40:46 PM:
>> Some thoughts and questions on getting started with the test documents.
>> Dave, Rob and others already made several detailed comments about this
>> topic on the formation mailing list, see also
>> http://sites.google.com/a/odfiic.org/tc/Home/odf-interoperability-
>> and-conformance
>> IMHO, we should start with the atomic documents.
>> Now, the opendocumentfellowship has already created a test suite:
>> http://develop.opendocumentfellowship.com/testsuite
>> Maybe we can reuse it (if the fellowship agrees with this) or do
>> something similar, minus the comments on specific implementations, but
>> adding remarks on ODF or the testing itself (like "spec not clear")
> http://develop.opendocumentfellowship.com/testsuite/
> It would be good to get the Fellowship to contribute their test suite to 
> the TC so we can work with in.  I know that it is under a Creative Commons 
> Attribution 2.5 licence, but this is weaker than the required OASIS 
> Feedback Licence, since CC waives copyright only, but does not grant any 
> protection against patents.
>> We could create a Wiki with a page per test document, and an upload
>> directory on the OASIS website (as mentioned during the TC conf call,
>> the OASIS Wiki doesn't support file upload) like:
>> ODF_1_1/
>> - atomic/
>> -- 5_Para_Elements/
>> --- 5_3_1_Note_Element.odt
>> - complex/
>> ...
>> ODF_1_2/
>> - atomic/
>> And if anyone feels like creating a document for a specific item, he/she
>> can mention it on the wiki, create a test and upload the file.
>> Once a week/month/..., a snapshot can be made and archived in a ZIP file
>> for convenient downloading.
> That's sounds like a reasonable way to structure it.  But it would be good 
> to first define exactly what we mean by a "test case", maybe agree on a 
> single test case as an example.

In trying to think about this from a product testers point of view, each "test case" has at least a two potential test scenarios:
- Validate that product X conforms to the ODF spec as demonstrated through the TC generated reference .odf file downloaded from the wiki
- Validate that product X interoperates with products Y and Z.

The test steps would then be:
A. Using product X open the reference .odf file and verify that it read the file correctly
B. Using product X write a .odf file from step 1 comparing the result with the reference .odf file
C. Using product X create the reference file from scratch, ie using product X visual commands, macros, scripts, etc...
D. Using product X write the resulting .odf file from step 3 comparing the result with the reference .odf file
E. Using product X open a related odf file that was created from product Y or Z verifying that product X read the file correctly
F. Using product X write the resulting odf file from step 5 comparing the result with the referenced odf file (and the original odf file from product Y or Z?)

To execute those steps the tester would need:
1. A reference ODF file
2. Manual steps or automated scripts to create an equivalent reference odf file (I recognize this may be product or project specific)
3. Access to equivalent odf files from other products
4. Information, steps or scripts that will enable the tester for validate test results

As a TC, we should build #1.  To make it useful and of interest to our audience we should enable them to build and contribute #2, #3 and #4

> In my mind any test cases should have:
> 1) A unique name or identifier

1a) A reference odf file

> 2) An indication of what version(s) of ODF the test case applies to
> 3) A list of pre-conditions for executing the test.  What must be done 
> first to set up the test case environment?  In most cases it will be 
> simple:  load the document in an ODF editor.  But even simple cases may 
> presuppose the installation of a particular font, for example.  There may 
> be a common set of pre-conditions which all or most test cases.  We can 
> list those once.  But we may have test cases that involve special 
> requirements, say a connection with an external database which may require 
> that such a database is first created

3a) A list of steps to create an equivalent odf reference file, and/or scripts that could be used to create an equivalent reference odf file  using product X, Y, Z
3b) Ability to store product X,Y,Z generated odf files that equate to the reference odf file

> 4) A list of post-conditions used to judge whether the test case passed. 
> This could be described in English, or in a formal language.

4a) or scripts to automate the verification

> As you see this is similar to how unit tests are commonly done:
> double sqrt(double x)
> {
> assert(x>=0)
> double ret = doCalculation()
> assert(ret*ret==x)
> }
> (and yes, I know one shouldn't really do equality tests of floating point 
> calculations...)
> This isn't the only model for how we can do test cases, but that is one 
> pattern of testing which has proven itself useful in other contexts.
>> This doesn't necessarily mean going through the whole spec in sequential
>> order, we can start with areas that are known to have issues (I would
>> hint: forms, gradients, slide effects, charts...) or easy to test
>> Questions:
>> - do we create atomic tests with an office suite, or craft it by hand to
>> make it really as small as possible ?
> It might be easier to start with output from, say OpenOffice.  But we 
> could have a tool that then strips out all the unnecessary markup in order 
> to reduce the file to the bare essentials.  We should also examine the 
> markup by hand and verify that it indeed is correct according to the ODF 
> standard.  So at the very least we should validate it with an XML 
> validator.
>> - should we do ODF 1.0, now that many (all ?) vendors use 1.1 or 1.2
>> draft ? Personally, I think we should start with issue-prone parts of
>> 1.1 and 1.2's OpenFormula.
> There is a lot of overlap between the releases.  So most test cases we do 
> will apply to all three versions.  Of course, open formula would only 
> apply to ODF 1.2.  But I agree that the "issue-prone" areas are the ones 
> to start with.
>> - who will run the tests and report the results, so that the OIC can
>> create a summary and general recommendations ? Per charter, the OIC
>> won't be commenting or identifying implementations (at least not in
>> reports, but isn't a wiki some kind of report ?)
> That's the sensitive issue.  We, as an OASIS TC, cannot issue a report on 
> implementations and their interoperability results.  

I agree, and may make it difficult to do what I suggested above, but thought I would throw it out in hopes of generating ideas.

> But as individual 
> members, especially members who are also ODF vendors, I'd expect that we 
> all run these test cases ourselves against our own implementations.  

That's what started me thinking this from a testers perspective. If we can ensure that our results are highly useful, easily consumable and re-usable then the vendors will pick up on our work and the TC will thrive.

> If an 
> individual member then wishes to report on their own product's results, 
> then that is fine.  And if a third part, having access to our test suite, 
> wishes to test and compare multiple implementations, then that is fine as 
> well.
> Maybe we want to talk to the OpenDocument Fellowship on this?  Maybe they 
> would relinquish the test case creation work to us, and instead take on 
> the testing of the applications against our test suite?  I think that 
> gives a needed separation of interests.
> -Rob
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php 

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]