Acid Tests (was: Re: [oiic-formation-discuss] Interoperability versusConformity)

[robert_weir@us.ibm.com]

"Sam Johnston" <samj@samj.net> wrote
on 06/11/2008 10:10:08 AM:

> 
> Indeed an acid test in the form of a (multi-page?) document with 
> progressively complex/esoteric directives could be a useful device
> for 'naming and shaming' poor implementations, as has proven very
> effective for W3C standards. It will be interesting to see if there
> is a place for something like this alongside a more complete test
suite
> , or indeed if there is even a need for both (presumably the former
> will enjoy more eyeballs which is arguably a good thing, if it can
> be brought up to the task).
> 
> This should be a lot easier for spreadsheets (at least formulas) 
> where a green/orange/red matrix could be set up, potentially with
> each field dependent on the last based on implementation priority.
Acid3
> appears to do something like this (eg I just got 71/100 on FF3rc2).
> 

Can we drill into this idea a bit more?  I'm
sure we've all see or heard of the browser Acid tests.  But what is
it exactly, in conformance terms?  

Specifically:

1) Is it a complete conformance assessment of the
tested standards (CSS2/XHTML)? 

2) Or is it more of a "challenge piece"
exercising the 10 or so features that current implementations are missing
or getting wrong, with the idea of drawing attention to those features
in hopes of moving implementations forward?

3) Does it focus on features that no one gets right
(initially)?  Or does it start with those that some get right and
some don't?  (It would seem that the greatest practical pain would
be around features that some implement but others don't.  It seems
to me that features that no one implements cannot be the cause of interoperability
problems.)

4) What is the relationship between Acid and other
forms of conformance testing?  For example, the W3C has a CSS2 test
suite (http://www.w3.org/Style/CSS/Test/).
 Why is Acid so much better known?  Is this because of technical
reasons?  Or the immediacy of the presentation (a smile face when
good, road kill when bad)?

5) What are the essential things that we need to bring
along into a TC deliverable to get the benefits we want?  How do we
define this in the charter?  I think saying "ODF Acid Test"
might mean different things to different people.  

I suggest we don't make "name and shame"
a formal TC goal.  It is probably better to let someone else do that.
 In fact, there are good reasons for drawing a clear line between
those who define the tests and those that evaluate implementations.  Andy
Updegrove has a nice article up on his ConsortiumInfo.org website called
"Certification Testing and Branding" (http://www.consortiuminfo.org/cb/)
where he describes the following categories:

1) Self-assertion without a test suite
2) Self-asserted compliance (self-certification)
3) Self-certification with verification
4) Third-party testing

We're starting at the bottom, with no test suite,
so all a vendor today can do is self-assert that they conform to the ODF
standard.  But the output of this proposed IIC TC would provide the
formal definitions that would enable higher levels of certification.  Although
we cannot, within an OASIS TC, set up shop as a third party test lab and
start testing and certifying implementations, our work would enable others
to do that.

-Rob