OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

oiic-formation-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [oiic-formation-discuss] Code walkthrough

"David Gerard" <dgerard@gmail.com> wrote on 07/01/2008 03:31:59 PM:

> 2008/7/1  <robert_weir@us.ibm.com>:
> > In any case although I do not prefer test cases that can only be verified by
> > code inspection, I do not agree with the assertion that they are for this
> > reason untestable.  A third-party testing lab could certainly arrange to
> > have access to the code.  And with self-certification, a vendor themselves
> > could verify.
> And in any case, any third party could clearly verify by throwing test
> documents at it that an app was "deliberately implementing a different
> behavior depending on a certain generator string," just as people
> already do with websites doing so for user-agents.

There is a limit to what we can do in an open forum.  I remember years ago a graphics card vendor who route their device driver so it knew when it was running in a popular graphics benchmark.  It looked at the list of executing processes.  If it knew it was being tested, it switched into a different mode of operation that cut some corners and gave artificially fast results.   That is the limitation of having a test suite that is open and visible to all.  It is hard to prevent such deliberatively bad behavior.  

We can't prove conformance.  We can just test some test cases, which is effectively just sampling.  We can disprove conformance, when test cases fail.  But we can never prove, by testing, that the application is conforms when presented with all possible conforming documents.  There is an art to picking test cases such that you will detect the maximum number of errors caused by implementation defects.  This is known to all QA practitioners.  But this assumes that errors are unintentional.  Finding intentional errors, especially when those making the errors know that you are looking for them, this is something else entirely.


[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]