Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC - Proposal

[Jim Marino <jim.marino@gmail.com>]

On Feb 28, 2011, at 6:17 PM, Anish Karmarkar wrote:

> On 2/25/2011 6:08 AM, Martin Chapman wrote:
>> Jim,
>> 
>> There are two orthogonal discussions here. One relates to TC activities
>> and progressing the spec and test suite to OASIS Standard level, and the
>> other relates to how people claim conformance outside of the TC process
>> or OASIS.
>> 
>> Let me start with the last one. Assuming SCA Assembly gets adopted as an
>> OASIS Standard, then anyone in the world (OASIS member or not) will be
>> able to claim conformance if they so choose. Such claims are not
>> **required** to be backed up by passing any test suite, though this of
>> course is strongly encouraged and we hope public opinion will demand it.
>> OASIS is not (yet) in the conformance testing and validation business so
>> one would rely on customers keeping the vendors honest. I don’t see
>> anyone disagreeing to these principles, so we don’t need to discuss this
>> aspect anymore.
> 
> True, unless we change the conformance section of the main spec to require running the tests. ;-)
> 
I would hope this doesn't come about for reasons below.

> As it currently stands, this is not a requirement and conforming implementations can completely ignore the tests.
> 
> 1) OASIS is not in a business of certifying, policing or branding. So any impl out there can claim that they are conformant regardless of what they do. The hope is that public naming and shaming would discourage anyone who doesn't conform from claiming so.
> 
This approach seems to be reasonable and has worked well in the past.

 
> 2) I see the test suite as similar to a JCP TCK. The test suite has been designed appropriately for adaption/customization and has all top-level composites designed to contain implementation.composites. To run the test though you need a specific (or >1) implementation type. Runtimes that use specific implementation technology, bindings can customize the test suite and adapt it for its purpose. And C/C++, Java, BPEL have already done so.
> 
> So what I'm wondering is: why would we not have adapting and passing the test suite be a requirement to claiming conformance (necessary but not sufficient condition) to the main spec? Wouldn't that keep implementations and vendor's marketing departments honest? It always happens that there are ambiguities and various interpretation of a spec despite best efforts. The test suite, is a runnable artifact with little doubt as to what one is supposed to do. Certainly, the test suite won't cover everything and isn't intended to do so. But I think a as close to an iron-clad conformance criteria that we can get to, the better off we are wrt ensuring portability and interop.

JCP TCKs usually involve stringent licensing, policing and branding requirements.  My understanding is an OASIS TC cannot require something along those lines.

In the end, I don't think a required TCK along the lines of the JCP is going to keep a vendor/entity "honest." Nor is it going to remove any more ambiguities than a non-normative test suite will. My experience is vendors take conformance claims very seriously regardless if there is a TCK or not because they can be held legally accountable by licensees if they misrepresent their software. We shouldn't need to worry about "cheaters" because there aren't any. Ambiguities aren't going to be resolved either unless we say the test cases take precedence over the documents. Moreover, the test suite may contain bugs. This has happened numerous times with the Java EE TCK and the vendors I worked for that were impacted had to petition Sun to resolve the issues. What would be the process for resolving this type of issue?

My concern with making the test suite required for conformance is fourfold:

1. Openness of the SCA standards
2. Who decides what is valid
3. What is normative in the test suite and what is not
4. The test suite has not been developed as a normative part of the specifications

First, as Eric mentioned, requiring a Java-centric test suite restricts the openness of the standards. Even though the test suite may be adapted to multiple implementation languages, it is still Java-centric. For example, it requires the JDK and Maven or Ant to run. I think requiring those technologies was the right decision in the context of providing a non-normative test suite. However, if an organization builds a non-Java implementation, why should they be required to accept the JDK end-user licensing terms to claim conformance? I suspect this would effectively preclude some software vendors from considering SCA.  The TC did a lot of work to open the specifications to alternative, non-Java implementations. Why should we go back on that now? 

My second concern centers on who decides what is valid. What would be required to demonstrate passing the tests? Does the vendor have to publish a distribution capable of passing the conformance tests? Is OASIS going to run and verify the tests?

My third concern is that the normative test suite artifacts are not well-defined. Can the build system be changed to remove the requirement on those artifacts? Can the test suite be modified and restructured to not use Maven and Ant?  What can and can't be changed? At this point, it seems more practical to stay with the existing decision to have the test suite be a guideline for conformance.   

My final issue is that for well over a year, the test suite has been developed as a set of non-normative artifacts. Personally, although I have spent a fair amount of time reviewing the test cases and submitting issues, I have not given the same attention to the test suite as I have the normative parts of the specification. Based on the JIRA issue and Subversion commit logs, my impression is many of the other TC members have allocated their time in a similar fashion. It seems to me a substantial risk to change course when the TC work is close to being completed and add significant surface area to the normative portions of the specification.  

Jim