OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

oiic-formation-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [oiic-formation-discuss] Code walkthrough


On Wed, Jul 2, 2008 at 10:37 PM,  <robert_weir@us.ibm.com> wrote:
>
> _
>
> "Tom M." <m42tom-odf@yahoo.com> wrote on 07/01/2008 11:25:12 PM:
>
>>
>> --- robert_weir@us.ibm.com wrote:
>> >
>>
>> >There is an art to picking test cases such
>> >that you will detect the maximum number of errors
>> >caused by implementation defects.  This is known to
>> >all QA practitioners.  But this assumes that errors
>> >are unintentional.  Finding intentional errors,
>> >especially when those making the errors know that you
>> >are looking for them, this is something else entirely.
>>
>> What sort of intentional errors are you thinking of, Rob?
>>
>
> We have an expression in the US, among school teachers, that mandatory
> testing leads to the practice of "teach to the test".  In other words, if
> you know in advance the contents of the test, then there is a risk of
> teaching only that exact material and losing the broader context.
>
> If an implementation knows in advance the exact test cases that will be
> given it, it is possible for that implementation to change their code so
> only those test cases work, but still have a negligible effect on improving
> interoperability.  This is like over-training a neural network on the same
> data.  You'll eventually have neutral network that gives the expected
> outputs, but it won't generalize well.
>
> I think this is why, in addition to the atomic testcases, we also need some
> real world complex documents that use features in combination and exercise
> their interactions.  This set of real world documents could be periodically
> refreshed, and in that way help prevent an implementation from mastering the
> test cases alone.
>
This is a lot harder than you think.  IE 8 beta was caught detecting
the acid test and processing differently so it was passing better than
it should.  So yes we do have at least one known cheat out there.

People will always try to cheat the test no matter what you do.   Even
a set of real world documents will fail if a implementer gets there
hands on them first before releasing so can make there application
look better against them.

Only method I can come up with is fuzing.  Yes its a security
exploiters tool.  Instead of real world documents randomly assembled
to spec ODF documents that users can generate.   This way test is
always changing.  Only way to beat that teach to the test.  A+ tests
use the same system.  Catch is doing that is more than complex because
it needs our own reference engine.

Sorry to say for now just do teach to the test.  It way better than
current and its something we have a chance of doing in a short time.

Peter Dolding


[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]