RE: [oic] Re: Idea about how to create conformance testing documents

["Hanssens Bart" <Bart.Hanssens@fedict.be>]

Title: RE: [oic] Re: Idea about how to create conformance testing documents

Hello Mingfei,


Thanks for this great mail !


* About creating test documents

Personally, I think we'll end up with both a few fairly "large" documents and
a whole lot of specific test documents.

Meeting in the middle, like the test scenarios that Dennis and Stephen are
working on, is a very good way to get things started. Once we start testing them,
we'll find issues and sometimes we might want to isolate this issue in order to
better understand it.
Meaning: creating a small, specific test document for it.

After doing that, or if someone has created separate, small test documents for
some other feature(s), we could try to combine 2 or 3 documents in 1 larger one
and retest it. This retesting is important, because combining documents (well,
actually, features) may expose issues in unexpected ways.

For example:
- the last digit of a page numbers may get truncated after 100 pages
- a colored table might go wrong on a page with an image as background
- some apparently unrelated features may crash an application when combined
(I am not making this up, the vendor correctly identified and fixed this by
the way, but my point is that we *might* encounter strange behavior)


Sometimes we have no choice but to create a small test file.

For instance: a document cannot be encrypted and not encrypted at the same time,
so we have to create at small test document (well: at least 3, .odt, .odp, .ods)
for that.



* Regarding "feature unit"

The easiest way would be to take a look at the spec and look for numbered
parts. For example: 17.4 MIME Type stream

Although I'm not sure if this will always lead to the "unit"
For example: is 17.7.3 File Entry a unit ?
Or should we consider the paragraphs like "Full Path" as a unit ?



* About the process

I like your draft proposal, although I'm not 100% sure about 4.3

"The test document shall pass the semantic verification. That is the behavior
are consistent when the test document is opened and rendered by different
existed ODF applications. Once having confliction, TC vote to make a decision
which behavior is correct by majority voting numbers."

Maybe it's included in the "semantic verification", but the ODF spec itself
must be used.

For example: if the spec says that a thumbnail must be 128x128 pixels PNG, but
the majority of the OIC would decide JPEG is just fine, it's still not correct
according to the spec. Perhaps the spec itself must be changed, but it would
be unfair to decide that not following this version of the spec is OK.

On the other hand, if the spec is NOT clear about it, the OIC could test and
see what is common practice and indeed vote for having things included it in
a profile or a "best practice" guide, or as an improvement for ODF-Next



* Meta data
I don't completely understand your question here:

"A question here, I know test meta data is important, but I am not clear now
where we test metat data is appropriate to use for test document creation.
I understand test meta data should be used for interoperability testing to
record test case, right?"

Do you mean "how could metadata help us for interoperability testing" ?

We could indeed use some kind of metadata (xml) to record the results of the
test cases in various implementations (detailing what part of the spec is to
be tested, and what the result is). Non-members could also send in these test
results as an XML attachment in a mail to the oic-comment list.

Members could also do that using JIRA, so perhaps that's is a more practical
approach:

- providing the test documents + an overview what is to be tested (as an HTML
page) + a very simple text template.
- members and non-members could use this very simple text template to send in
results
- volunteers from the OIC could verify and record the results in JIRA



Best regards,

Bart