RE: [oic] Re: Idea about how to create conformance testing documents

["Hanssens Bart" <Bart.Hanssens@fedict.be>]

Title: RE: [oic] Re: Idea about how to create conformance testing documents

Hello Mingfei,


>> For instance: a document cannot be encrypted and not encrypted at
>> the same time,
>> so we have to create at small test document (well: at least 3, .odt,
>> .odp, .ods)
>> for that.
> I agree that first we create small test document for some atomic
> feature, second create complex document that combines different small
> test documents.

As an experiment, I've added a few test documents to SVN. They already
combine several elements and that works out very well for the chapters
I've been looking at.

It's just iterative trial and error: I grab a not-too-complex chapter
(or part of it), read it sequentially and write out a few elements,
validate it with the ODF validator, then add some more elements and
occasionally open it in a few applications

These are simple and rather isolated elements (meta data and text fields,
so very few styles involved, very few dependencies on other chapters).
So probably it gets more "interesting" when dealing with advanced tables
and such :-) There we might need to keep (or create additional) smaller
"atomic" test documents in addition to the combined document(s).


> And since in the above, we agree to do the things by 2 steps, first
> create small test documents, second create complex scenario documents,
> so now how about just define the feature unit as one element?

Sounds reasonable, probably the definition would be something like
"a feature unit consists of an element with its possible attributes,
attribute values and associated behavior" ? Or is this too broad ?


> Also if this is the case, I would like each test case should contain
> at least 3 test documents for the element feature, so the total test
> cases will be still about 540 in 1.1 or 600 in 1.2.

Mm, you'll need far more test *cases*, but perhaps fewer test *documents*

For instance: office:annotation could be called a "feature unit"
But you'll have to test its attributes as well, so one document could
contain, say, 30 office:annotations (testing round corners capability,
anchor points etc)

(side note: calculating the exact number of test cases / combinations
could be homework for CS or Math students :-) To do it right, you'd first
have to figure out what the exact dependencies between the elements and
attributes are)


> If anyone would like to contribute test case, he need to create test
> documents and test metadata by himself, and submit to TC.

Well, someone will have to keep track of what exactly is the purpose of
the test document. That's where the test metadata is for.
For instance, if one would simply send in a 100 page document, but does
not say what is in it, all testers will need to reverse-engineer it...

I don't say we absolutely have to use my metadata schema proposal :-)

But at least we must track:
- who submitted the document
- which version and what part(s) of the spec is tested by this document
- what are the steps to test implementations with this document
(things like: do you only have to open it and look at it ? or save it
and check some elements to see if they have been updated ? or should
you inspect some document or style property ?)

(and optionally: results, so we can create interop reports from them)


> On the other hand, do the TC plan to use this test metadata
> to describe and present the test case stored in TC repository?
> If that is, how to present the metadata in TC web ?

Presenting is easy :-) If the metadata is stored in XML, you can apply
an XSL and generate XHTML. Then zip it together with the all the test
documents, and store it in the doc repository (as required by OASIS
guidelines) and optionally put it on a website or wiki.


> Maybe redesign a
> web form to let contributor input, or just use OASIS document
> repository ?

The input is the hardest part indeed. My XML proposal isn't exactly
the most userfriendly approach (probably only useful as storage)

One could think of:

A) Allowing non-members to send emails to the oic comment list,
together with a wiki-style description (similar to the way the
requirements for ODF-Next are now being gathered by the ODF TC,
very simple template with "+NAME" etc), that can be parsed and
transformed to an XML.

B) A website with some form that generates an XML (I don't think we
can accept direct form submissions, due to OASIS IPR rules, unless
the form is submitted by the server to the comment list and that's
clearly mentioned on the form)


C) some client-side tool like firefox with javascript/greasemonkey,
or a xul-app, or Adobe AIR, or eclipse plugin (perhaps too heavy),
or OOo/MSO/Symphony/... extension to procude the XML, then send it
to the list

(It would be cool to be able
- to load the spec
- select any part of the text that describes what you're testing,
- drag'n'drop the selection to a pane or clipboard
- having this tool to figure out what section the text belongs to,
so you don't have to copy the numbered headers yourself)


D) using your favorite editor: simply grab the spec, sticking notes /
annotations / comments on it and upload/mail it. Once again, these
notes would need to follow a certain template (wiki or POD style ?)
and this does induce quite some overhead if you mail the entire spec
to the mailing list for every test document


(Regardless of the tool and format, the delivery mechanism for
contributions from non-members will have to be the oic-comment list)


Best regards,

Bart