Re: [sca-assembly] Issue 101: Complete the Conformance Section -Comments on Proposal

[Anish Karmarkar <Anish.Karmarkar@oracle.com>]

One comment inlined below.

-Anish
--

Mike Edwards wrote:
> 
> Anish,
> 
> Reply inline...
> 
> Yours,  Mike.
> 
> Strategist - Emerging Technologies, SCA & SDO.
> Co Chair OASIS SCA Assembly TC.
> IBM Hursley Park, Mail Point 146, Winchester, SO21 2JN, Great Britain.
> Phone & FAX: +44-1962-818014    Mobile: +44-7802-467431  
> Email:  mike_edwards@uk.ibm.com
> 
> Anish Karmarkar <Anish.Karmarkar@oracle.com> wrote on 03/03/2009 07:46:07:
>  > Re: [sca-assembly] Issue 101: Complete the Conformance Section -
>  > Comments on Proposal
>  > Mike Edwards wrote:
>  > >
>  > > Folks,
>  > >
>  > > Some comments on the proposal:
>  > >
>  > >
>  > > 1)  I think that making a series of SCA-related documents into
>  > > conformance points, as per section 12.1,  is unnecessary and unwise.
>  > >
>  > > The only point of making these documents into conformance points is if
>  > > it is intended to write testcase(s) that will validate those
>  > > documents.  I do not believe that we have the resources to write such
>  > > testcases and as a result, the conformance demands made
>  > > here are a waste of time and effort.
>  > >
>  >
>  > I would like to understand what additional tests will have to be written.
>  > There are already requirements on SCA runtime regarding rejecting what I
>  > would call non-conformant SCA documents. To test that we would have to
>  > create non-conformant documents, use it with a runtime and ensure that
>  > the runtime rejects them. What additional test artifact would have to be
>  > created wrt document conformance? I.e., we have a set of documents that
>  > we assert are non-conformant. We of course would have a set of
>  > conformant documents (for the positive test for a runtime).
>  >
>  > I can see us not wanting to create non-conformant (or conformant) SCA
>  > documents for every little feature and element/attribute that schema
>  > allows. But we are now talking about how detailed our test suite is
>  > going to be. Given that there is a requirement for runtimes to reject
>  > invalid SCA documents, this is already an issue. I don't think we need
>  > to Turing complete.
>  >
> 
> The tests you describe above are simply tests of an SCA runtime - and we 
> have
> these already, including "invalid documents".  However, they only check on
> the valid behaviour of an SCA runtime.
> 
> If we make claims about documents themselves being valid or invalid, it is
> my opinion that such claims are pretty worthless unless there are tests to
> back them up.  Such tests would actually have to be able to accept some
> random document and evaluate whether that document was a valid document or
> not.  I even heard one person on a previous call ask for a tool of exactly
> this kind.

But that isn't a test that is a document validator.
I don't see the need for a document validator defined by the TC.
It would be a nice tool to have around, but not required IMHO.

> 
> At the moment I don't think it is worth the time and effort of building
> tests that could validate documents.  As a result, I take the view that
> there is little to be gained in having documents as conformance points.
> The specification concentrates on what an SCA runtime does with a document
> and we have the tests to check out that the runtimes are doing the right
> things.
> 
> Back to answer Martin's question of whether the runtime is right or whether
> the document is right when some failure occurs.  If the SCA runtime passes
> our tests then the first reaction should be to say that the runtime is
> right and the document is incorrect.  If this isn't the case, it actually
> shows a hole in our SCA runtime tests - go plug the hole!!
> 
>  
>  > -Anish
>  > --
>  >
> <snip>
> 
> 
> ------------------------------------------------------------------------
> 
> /
> /
> 
> /Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number 
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU/
> 
> 
> 
> 
> 
>