Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC- Proposal

[Eric Johnson <eric@tibco.com>]

From the charter:

http://www.oasis-open.org/committees/sca-assembly/charter.php#item-4

"The TC shall define concrete exit criteria that include at least two independent offerings that implement and are compliant with the all normative portions of specifications and demonstrate interoperability and portability as appropriate. Note that these are minimums and that the TC is free to set more stringent criteria. "

The ambiguity of English gets in the way here a little. Does this mean:

• For each normative statement, at least two implementations?
• a minimum of two implementations that each fully implement all normative statements?

I suspect we strongly want to prefer the former. That means we can potentially rely on lots of implementations that don't claim conformance to the spec, but do help us to achieve coverage.

What this discussion turns on, then, I think is the meaning of "implement", and "compliant". Is it sufficient for a vendor to declare that they've implemented and are in compliance? Or do we have to "prove" that in some way? I'm satisfied if a vendor that claims to implement a normative statement.

As for "and demonstrate interoperability and portability", based on what I understand of the existing test suites, that two implementations that pass the test suite is a pretty darn good demonstration, and I wouldn't go further than that. I'm with Jim in that at some point we have to rely on the court of public opinion.

From a pure "quality of work" perspective, I think if two implementations pass the test suite, and they comment about the normative statements not covered by the test suite, then we've acheived sufficient "code coverage" if you will, of the spec itself.

I've expressed an ongoing concern that we've over-specified parts of SCA. Certainly, within the bindings TC, some of you have heard me express that point. To pick on a random point (really, I just looked at the normative statements and this one caught my eye - hadn't thought about it before), the "sca-contribution-generated.xml" file declared for deployment - how is this file an essential part of testing conformance? Since there's no language that I'm aware of that *excludes* other files from being processed as part of the contribution, why couldn't this detail be left to an implementation? Do we *require* this for the specification?

If a side-effect of the "two implementations" constraint leads us to de-normalize (de-normativize?) some of the current required language, that might be a good thing.

I'll reiterate what I said in the call. We should check with the implementations that we're aware of, and find out which normative statements they don't believe they've implemented. With that information in hand, we'll better understand our options for this question. (And I'm not volunteering to take on this task because I don't know who to talk to, and I'm not familiar with the existing test suite.)

-Eric.

On 2/22/11 12:08 PM, Jim Marino wrote:

74664AB4-833B-46BF-BFBC-56D0631456C9@gmail.com" type="cite">

Hi,

The first bullet makes sense to me. Concerning the second, my understanding is the TC Test Suite is not normative and the only thing required for conformance is for the entity responsible for a particular runtime to declare that it adheres to the normative portions of the Assembly Specification. In other words, it should be valid for a runtime to claim conformance that supplies its own test suite, which may or may not be based on the TC one. This essentially goes back to using the "court of public opinion" to judge conformance claims as opposed to something akin to a TCK.

Jim   

On Feb 18, 2011, at 6:19 AM, Mike Edwards wrote:

Folks,

I propose the following as the Exit Criteria to be adopted by the SCA Assembly TC:

"The Concrete Exit Criteria for the SCA Assembly V1.1 specification are that:

o there shall be 2 independent SCA runtimes that are compliant with all normative portions of the
specification as described in Section 12.2 of the SCA Assembly V1.1 specification

o the 2 independent runtimes shall pass the Test Suite for SCA Assembly as described in the document
'TestCases for the SCA Assembly Model Version 1.1 Specification' "


I believe that this is sufficient to cover:
- the question of conformance to the spec
- the issues of portability and interoperability

since the test suite is a pretty thorough examination of conformance to the spec (there are testcases for
all normative statements with testable & observable consequences) and the test suite is also a test of portability,
since the test suite has an extensive set of artifacts that are ported to each runtime and interoperability is
checked through the use of Web services being used between the SCA runtime and a non-SCA (JAXWS) client.



Yours, Mike

Dr Mike Edwards		Mail Point 146, Hursley Park	<Mail Attachment.gif>
STSM		Winchester, Hants SO21 2JN
SCA & Services Standards		United Kingdom
Co-Chair OASIS SCA Assembly TC
IBM Software Group
Phone:	+44-1962 818014
Mobile:	+44-7802-467431 (274097)
e-mail:	mike_edwards@uk.ibm.com

---

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU