Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC - Proposal

[Jim Marino <jim.marino@gmail.com>]

On Mar 3, 2011, at 12:33 PM, Jeff Mischkinsky wrote:

> 
> On Mar 03, 2011, at 10:49 AM, Eric Johnson wrote:
> 
>> Are we arguing two different things in this thread?
> 
> i think we are conflating 2 different topics:
>   I agree that claiming conformance does not REQUIRE passing the test suite. That's about what products have to do in order to make legitimate claims.
> 

Since OASIS cannot police conformance in the same way as Java EE is done, how would you propose verification be handled? For example, do you think all vendors should make artifacts (e.g. runtimes, configuration of the test suite, test suite integration code and/or modifications) available publicly so that passing tests can be independently verified?  If not, how is simply claiming the test suites pass and providing no formal evidence more "legitimate" than claiming a runtime conforms to the normative sections of the SCA specifications? 

FWIW, I think a vendor can make a strong, legitimate claim without passing the tests publicly by supplying their own test suites or relying on reputation (and the fear of legal consequences associated with false product claims). In other words, the risks are too great to cheat.
 
> 
>> 
>> Danny is saying that the charter does not require two implementations passing the test suite.
>> 
>> Jeff is saying that the charter does require two implementations passing the test suite. At least, that's what I interpreted his response to mean.
> 
> yes but that's not for the purposes of conformance. It's for satisfying the exit criteria, 2 impls passing the test suites which cover all the normative portions
> 
> 
>> 
>> I side with Danny - conforming to the normative portions of the specification does not (currently) imply passing the test suite.
> 
> Actually i think it does as a practical matter - irrespective of charter requirements, etc.
> How could a conformant implementation not pass the test suites? (assuming for the moment the test suites are correct and have no bugs)
> 

Formally the exit criteria do no require two implementations to pass the test suites. I also don't think the criteria do as a practical manner either even if the test suites are perfect. That is because the test suites require integration code between the test harness and runtime. One of the reasons this integration code is required is because there is no normative technology-specific deployment API defined by SCA (normative deployment semantics are defined). It is possible for a vendor not to implement this integration code or provide an API to do so. In that case, the runtime could be conformant but not capable of being run inside the test suite harness.

Jim