OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

sca-assembly message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC- Proposal

On 2/25/2011 6:08 AM, Martin Chapman wrote:
> Jim,
> There are two orthogonal discussions here. One relates to TC activities
> and progressing the spec and test suite to OASIS Standard level, and the
> other relates to how people claim conformance outside of the TC process
> or OASIS.
> Let me start with the last one. Assuming SCA Assembly gets adopted as an
> OASIS Standard, then anyone in the world (OASIS member or not) will be
> able to claim conformance if they so choose. Such claims are not
> **required** to be backed up by passing any test suite, though this of
> course is strongly encouraged and we hope public opinion will demand it.
> OASIS is not (yet) in the conformance testing and validation business so
> one would rely on customers keeping the vendors honest. I don’t see
> anyone disagreeing to these principles, so we don’t need to discuss this
> aspect anymore.

True, unless we change the conformance section of the main spec to 
require running the tests. ;-)

As it currently stands, this is not a requirement and conforming 
implementations can completely ignore the tests.

1) OASIS is not in a business of certifying, policing or branding. So 
any impl out there can claim that they are conformant regardless of what 
they do. The hope is that public naming and shaming would discourage 
anyone who doesn't conform from claiming so.

2) I see the test suite as similar to a JCP TCK. The test suite has been 
designed appropriately for adaption/customization and has all top-level 
composites designed to contain implementation.composites. To run the 
test though you need a specific (or >1) implementation type. Runtimes 
that use specific implementation technology, bindings can customize the 
test suite and adapt it for its purpose. And C/C++, Java, BPEL have 
already done so.

So what I'm wondering is: why would we not have adapting and passing the 
test suite be a requirement to claiming conformance (necessary but not 
sufficient condition) to the main spec? Wouldn't that keep 
implementations and vendor's marketing departments honest? It always 
happens that there are ambiguities and various interpretation of a spec 
despite best efforts. The test suite, is a runnable artifact with little 
doubt as to what one is supposed to do. Certainly, the test suite won't 
cover everything and isn't intended to do so. But I think a as close to 
an iron-clad conformance criteria that we can get to, the better off we 
are wrt ensuring portability and interop.


> What we need to get agreement on is the first discussion: what do we!
> need to do in the TC to progress the SCA Assembly spec to OASIS
> Standard. The OASIS TC Process requires Statements of Use[1] from at
> least three OASIS Organizational Members before a Spec can be put to
> OASIS Member vote to become an OASIS Standard. The definition of
> Statement of Use in OASIS is quite vague, so when chartering the SCA TCs
> we agreed to strengthen the requirement and borrow from other
> organisations that require more rigour. To this end we agreed to add
> testing deliverables to these charters (which are not required by the TC
> process). We also put in the charters wording about exit criteria [2] to
> make it clear that we wanted at least two compliant implementations so
> that at least two of the Statements of Use will to real and fully
> compliant implementations. The only thing that is being debated is the
> precise interpretation of our exit criteria, and the main issue is
> whether the TC requires the implementations to pass the TC’s test suite
> in ! order to qualify as one of the implementations that meet our exit
> criteria. It’s a yes/no question!
> As a Chair I won’t use this email to argue one way or another, I just
> want to make clear what the exact topic of the debate is.
> Cheers,
> Martin.
> [1]"Statement of Use", with respect to a Committee Specification, is a
> written statement by an OASIS Organizational Member stating that it is
> successfully using or implementing that specification in accordance with
> the conformance clauses specified in Section 2.18
> <http://www.oasis-open.org/committees/process.php#specQuality>, and
> stating whether its use included the interoperation of multiple
> independent implementations.
> [2] *Exit Criteria*
> The TC shall define concrete exit criteria that include at least two
> independent offerings that implement and are compliant with the all
> normative portions of specifications and demonstrate interoperability
> and portability as appropriate. Note that these are minimums and that
> the TC is free to set more stringent criteria.
> *From:* Jim Marino [mailto:jim.marino@gmail.com]
> *Sent:* 24 February 2011 19:27
> *To:* OASIS Assembly
> *Subject:* Re: [sca-assembly] Concrete Exit Criteria for the SCA
> Assembly TC - Proposal
> On Feb 23, 2011, at 3:06 PM, Martin Chapman wrote:
> Jim,
> *From:* Jim Marino [mailto:jim.marino@gmail.com]
> *Sent: 23 February 2011 19:35
> To: OASIS Assembly
> Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly
> TC - Proposal*
> * *
> *Hi Martin,*
> * *
> *I guess I may now be even more confused, probably because of my
> ignorance of OASIS rules. Why would the exit criteria for the TC require
> an implementation to adhere to something that is not normative,
> specifically running and passing the TC-supplied test suite?*
> * ***
> *<MC> Because that is why we have been writing a te! st suite in the TC.
> The intention is that two indepe! ndent runtimes pass the TC’s test
> suite. That way we demonstrate that the spec has been independently
> implemented and that the test suite isn’t (too) specific to a particular
> runtime***
> * ***
> *Also, who determines whether a runtime passes or not?*
> *<MC> the Test suite, saying whether a particular test passes or not?***
> * *
> Hi Martin,
> I think you raise two important but separate issues here: (1)
> demonstrating the spec has been independently implemented; and (2) that
> the test suite is too tailored to a specific runtime.
> Regarding (1), I thought it was sufficient for an entity responsible for
> a runtime to declare in some public and documented way that it conforms
> to an SCA specification. The entity could decide to use the TC tests,
> adapt them, or write their own. Policing conformance would be left to
> the court of public opinion. Is it now a requirement that a runtime
> passes the TC-supplied test suite to claim conformance? If so, what does
> an entity need to do to pass the tests? Is it a requirement that the
> entity publish test artifacts, including source and/or binaries of the
> integration code required to plug the runtime ! into the test suite? Or
> can they just say they do? Are they allowed to modify the test cases to
> fit them into their own integration testing infrastructure?
> My concern, similar to Eric's, is that we are going down the path of
> over-specification.
> Regarding (2), my understanding was the test suite was non-normative and
> similar to a set of "guidelines". If that is the case and the suite is
> too tailored to a particular runtime or set of runtimes, so be it. There
> is always the option of taking the test suite as inspiration and
> developing one more suited to a particular platform. For example, if I
> am developing a non-Java runtime, I am likely to want to develop a test
> suite using the build and CI tools currently in use. I wouldn't want to
> be forced to write Java code that integrates! my non-Java runtime into
> the test suite and have to run the JDK-based test client.
> Jim

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]