OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

oiic-formation-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [oiic-formation-discuss] Code walkthrough


"Tom M." <m42tom-odf@yahoo.com> wrote on 07/01/2008 11:25:12 PM:

> --- robert_weir@us.ibm.com wrote:
> >
> >There is an art to picking test cases such
> >that you will detect the maximum number of errors
> >caused by implementation defects.  This is known to
> >all QA practitioners.  But this assumes that errors
> >are unintentional.  Finding intentional errors,
> >especially when those making the errors know that you
> >are looking for them, this is something else entirely.
> What sort of intentional errors are you thinking of, Rob?

We have an expression in the US, among school teachers, that mandatory testing leads to the practice of "teach to the test".  In other words, if you know in advance the contents of the test, then there is a risk of teaching only that exact material and losing the broader context.  

If an implementation knows in advance the exact test cases that will be given it, it is possible for that implementation to change their code so only those test cases work, but still have a negligible effect on improving interoperability.  This is like over-training a neural network on the same data.  You'll eventually have neutral network that gives the expected outputs, but it won't generalize well.

I think this is why, in addition to the atomic testcases, we also need some real world complex documents that use features in combination and exercise their interactions.  This set of real world documents could be periodically refreshed, and in that way help prevent an implementation from mastering the test cases alone.


[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]