Re: [oiic-formation-discuss] Code walkthrough

[robert_weir@us.ibm.com]

"David Gerard" <dgerard@gmail.com>
wrote on 07/01/2008 03:31:59 PM:

> 2008/7/1  <robert_weir@us.ibm.com>:
> 
> > In any case although I do not prefer test cases that can only
be verified by
> > code inspection, I do not agree with the assertion that they
are for this
> > reason untestable.  A third-party testing lab could certainly
arrange to
> > have access to the code.  And with self-certification, a
vendor themselves
> > could verify.
> 
> 
> And in any case, any third party could clearly verify by throwing
test
> documents at it that an app was "deliberately implementing a
different
> behavior depending on a certain generator string," just as people
> already do with websites doing so for user-agents.
> 

There is a limit to what we can do in an open forum.
 I remember years ago a graphics card vendor who route their device
driver so it knew when it was running in a popular graphics benchmark.
 It looked at the list of executing processes.  If it knew it
was being tested, it switched into a different mode of operation that cut
some corners and gave artificially fast results.   That is the limitation
of having a test suite that is open and visible to all.  It is hard
to prevent such deliberatively bad behavior.  

We can't prove conformance.  We can just test
some test cases, which is effectively just sampling.  We can disprove
conformance, when test cases fail.  But we can never prove, by testing,
that the application is conforms when presented with all possible conforming
documents.  There is an art to picking test cases such that you will
detect the maximum number of errors caused by implementation defects.  This
is known to all QA practitioners.  But this assumes that errors are
unintentional.  Finding intentional errors, especially when those
making the errors know that you are looking for them, this is something
else entirely.

-Rob