OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

oiic-formation-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [Re: [oiic-formation-discuss] Reference implementation. Layout perfect.]


On Sat, Jun 14, 2008 at 9:13 AM, Dave Pawson <dave.pawson@gmail.com> wrote:
2008/6/13 Shawn <sgrover@open2space.com>:
> oops, forgot to make sure the list was addressed...
>
> robert_weir@us.ibm.com wrote:
>>
>> Or just do it by manual inspection.  There are ways of making that
>> approach tractable.
>
> I am hesitant to introduce manual tests.  Doing so introduces a very
> subjective input from humans.  A pleasing shade of red to me, may be jarring
> to you.  An interpretation of an output may be fine for me, but may fail for
> you.  The results are NOT definitive.

Wholly agree that this is not desirable.
Go take a look at ODF though. Many of the requirements result in
subjective output.

It is up to the TC to define tests which are objective, even when assessed
by manual inspection. If this isn't possible, then IMHO it is something
the TC either makes a decision on and informs the main TC, or we
identify it as untestable (objectively) and inform the main TC.

I've had ISO9000 approval on software which included quite
a large proportion of tests being assessment of screen content.
It is possible.

It's true that there may have to be some manual component - the gradient example before is a good one in that while atomic testing of ODF itself will verify conformance with the spec, it won't tell us that the implementation interprets the ODF well and renders what is expected, nor whether users' directives are accurately reflected in the resulting ODF after creating the object. Whether or not the observer likes the shade of red selected will not impair them from indicating whether the red is at the center or the outside of the shape, but testing that UI functions generate 'correct' ODF is even trickier.

This is also an example of something that would currently have to be sent back to the OpenDocument TC for clarification and ideally alignment with existing specifications (eg SVG) /before/ it would be useful in any manual tests; at least we should keep track of such issues if there is no issue tracking system for ODF itself (this could be the first such comprehensive review of the spec).

There are a number of ways one could execute such tests (likely delivered as snippets of ODF along with reference images), including implementors using internal resources (perhaps low cost off-shore labs), the Acid Test(s), and of course more novel approaches like Amazon Web Services' Mechanical Turk (perhaps the most interesting web service available today, if not universally useful).

Sam



[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]