Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC - Proposal

[Jeff Mischkinsky <jeff.mischkinsky@oracle.com>]

hi jim,
    If some entity (a vendor, e.g.) wants to claim in the market that  
they have a product that conforms to SCA, they can do that. The SCA  
specs should be quite clear what is needed in order for that claim to  
be legitimate. I agree that OASIS has nothing to say about that  
vendor's product claims. Nor does the implementer of that product HAVE  
to run the OASIS test suites in order to make that claim.

  Now if i were a customer who believed in "trust, but verify", i'd  
ask the implementer to convince me that their claims are true. One way  
might be to say "see, here's my product passing the oasis test  
suites". Another way might be to say, "i wrote these really cool  
tests, if you sign this NDA, I'll show you the test results". It might  
be  "if you buy my product and can demonstrate where it is non- 
conformant, I'll pay you a million dollars, and fix the problem within  
1 month". Or it might be as simple as "Just trust me. I did it  
correctly".  All that is between the customer and the vendor as part  
of their business relationship. (Note: this goes for open source  
implementations also - just that no money changes hands, and the  
customer might have to do the convincing themselves.)

All this is also independent of the exit criteria which essentially  
says:  there should be (at least) 2 conformant implementations, and a  
demonstration of (at least) 2 implementations which pass the OASIS  
tests -- as a demonstration that the specs have been debugged.

Please note: that an implementation that claims to be conformant (and  
may indeed have its own set of tests) does not help meet the criteria  
of having 2 implementations that can pass the OASIS tests.

cheers,
  jeff

On Feb 24, 2011, at 11:26 AM, Jim Marino wrote:

> On Feb 23, 2011, at 3:06 PM, Martin Chapman wrote:
>
>> Jim,
>>
>>
>> From: Jim Marino [mailto:jim.marino@gmail.com]
>> Sent: 23 February 2011 19:35
>> To: OASIS Assembly
>> Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA  
>> Assembly TC - Proposal
>>
>> Hi Martin,
>>
>> I guess I may now be even more confused, probably because of my  
>> ignorance of OASIS rules. Why would the exit criteria for the TC  
>> require an implementation to adhere to something that is not  
>> normative, specifically running and passing the TC-supplied test  
>> suite?
>>
>> <MC> Because that is why we have been writing a test suite in the  
>> TC. The intention is that two indepe! ndent runtimes pass the TC’s  
>> test suite. That way we demonstrate that the spec  has been  
>> independently  implemented and that the test suite isn’t (too)  
>> specific to a particular runtime
>>
>> Also, who determines whether a runtime passes or not?
>> <MC> the Test suite, saying whether a particular test passes or not?
>>
>
> Hi Martin,
>
> I think you raise two important but separate issues here: (1)  
> demonstrating the spec has been independently implemented; and (2)  
> that the test suite is too tailored to a specific runtime.
>
> Regarding (1), I thought it was sufficient for an entity responsible  
> for a runtime to declare in some public and documented way that it  
> conforms to an SCA specification.  The entity could decide to use  
> the TC tests, adapt them, or write their own. Policing conformance  
> would be left to the court of public opinion.  Is it now a  
> requirement that a runtime passes the TC-supplied test suite to  
> claim conformance? If so, what does an entity need to do to pass the  
> tests? Is it a requirement that the entity publish test artifacts,  
> including source and/or binaries of the integration code required to  
> plug the runtime into the test suite? Or can they just say they do?  
> Are they allowed to modify the test cases to fit them into their own  
> integration testing infrastructure?
>
> My concern, similar to Eric's, is that we are going down the path of  
> over-specification.
>
> Regarding (2), my understanding was the test suite was non-normative  
> and similar to a set of "guidelines". If that is the case and the  
> suite is too tailored to a particular runtime or set of runtimes, so  
> be it. There is always the option of taking the test suite as  
> inspiration and developing one more suited to a particular platform.  
> For example, if I am developing a non-Java runtime, I am likely to  
> want to develop a test suite using the build and CI tools currently  
> in use. I wouldn't want to be forced to write Java code that  
> integrates my non-Java runtime into the test suite and have to run  
> the JDK-based test client.
>
> Jim
>
>

--
Jeff Mischkinsky			          		jeff.mischkinsky@oracle.com
Sr. Director, Oracle Fusion Middleware 				+1(650)506-1975
	and Web Services Standards           			500 Oracle Parkway, M/S 2OP9
Oracle								Redwood Shores, CA 94065