OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

oiic-formation-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Acid Tests (was: Re: [oiic-formation-discuss] Interoperability versusConformity)

"Sam Johnston" <samj@samj.net> wrote on 06/11/2008 10:10:08 AM:

> Indeed an acid test in the form of a (multi-page?) document with
> progressively complex/esoteric directives could be a useful device
> for 'naming and shaming' poor implementations, as has proven very
> effective for W3C standards. It will be interesting to see if there
> is a place for something like this alongside a more complete test suite
> , or indeed if there is even a need for both (presumably the former
> will enjoy more eyeballs which is arguably a good thing, if it can
> be brought up to the task).
> This should be a lot easier for spreadsheets (at least formulas)
> where a green/orange/red matrix could be set up, potentially with
> each field dependent on the last based on implementation priority. Acid3
> appears to do something like this (eg I just got 71/100 on FF3rc2).

Can we drill into this idea a bit more?  I'm sure we've all see or heard of the browser Acid tests.  But what is it exactly, in conformance terms?  


1) Is it a complete conformance assessment of the tested standards (CSS2/XHTML)?

2) Or is it more of a "challenge piece" exercising the 10 or so features that current implementations are missing or getting wrong, with the idea of drawing attention to those features in hopes of moving implementations forward?

3) Does it focus on features that no one gets right (initially)?  Or does it start with those that some get right and some don't?  (It would seem that the greatest practical pain would be around features that some implement but others don't.  It seems to me that features that no one implements cannot be the cause of interoperability problems.)

4) What is the relationship between Acid and other forms of conformance testing?  For example, the W3C has a CSS2 test suite (http://www.w3.org/Style/CSS/Test/).  Why is Acid so much better known?  Is this because of technical reasons?  Or the immediacy of the presentation (a smile face when good, road kill when bad)?

5) What are the essential things that we need to bring along into a TC deliverable to get the benefits we want?  How do we define this in the charter?  I think saying "ODF Acid Test" might mean different things to different people.  

I suggest we don't make "name and shame" a formal TC goal.  It is probably better to let someone else do that.  In fact, there are good reasons for drawing a clear line between those who define the tests and those that evaluate implementations.  Andy Updegrove has a nice article up on his ConsortiumInfo.org website called "Certification Testing and Branding" (http://www.consortiuminfo.org/cb/) where he describes the following categories:

1) Self-assertion without a test suite
2) Self-asserted compliance (self-certification)
3) Self-certification with verification
4) Third-party testing

We're starting at the bottom, with no test suite, so all a vendor today can do is self-assert that they conform to the ODF standard.  But the output of this proposed IIC TC would provide the formal definitions that would enable higher levels of certification.  Although we cannot, within an OASIS TC, set up shop as a third party test lab and start testing and certifying implementations, our work would enable others to do that.


[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]