[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC- Proposal
On 2/25/2011 6:08 AM, Martin Chapman wrote: > Jim, > > There are two orthogonal discussions here. One relates to TC activities > and progressing the spec and test suite to OASIS Standard level, and the > other relates to how people claim conformance outside of the TC process > or OASIS. > > Let me start with the last one. Assuming SCA Assembly gets adopted as an > OASIS Standard, then anyone in the world (OASIS member or not) will be > able to claim conformance if they so choose. Such claims are not > **required** to be backed up by passing any test suite, though this of > course is strongly encouraged and we hope public opinion will demand it. > OASIS is not (yet) in the conformance testing and validation business so > one would rely on customers keeping the vendors honest. I don’t see > anyone disagreeing to these principles, so we don’t need to discuss this > aspect anymore. True, unless we change the conformance section of the main spec to require running the tests. ;-) As it currently stands, this is not a requirement and conforming implementations can completely ignore the tests. 1) OASIS is not in a business of certifying, policing or branding. So any impl out there can claim that they are conformant regardless of what they do. The hope is that public naming and shaming would discourage anyone who doesn't conform from claiming so. 2) I see the test suite as similar to a JCP TCK. The test suite has been designed appropriately for adaption/customization and has all top-level composites designed to contain implementation.composites. To run the test though you need a specific (or >1) implementation type. Runtimes that use specific implementation technology, bindings can customize the test suite and adapt it for its purpose. And C/C++, Java, BPEL have already done so. So what I'm wondering is: why would we not have adapting and passing the test suite be a requirement to claiming conformance (necessary but not sufficient condition) to the main spec? Wouldn't that keep implementations and vendor's marketing departments honest? It always happens that there are ambiguities and various interpretation of a spec despite best efforts. The test suite, is a runnable artifact with little doubt as to what one is supposed to do. Certainly, the test suite won't cover everything and isn't intended to do so. But I think a as close to an iron-clad conformance criteria that we can get to, the better off we are wrt ensuring portability and interop. -Anish -- > > What we need to get agreement on is the first discussion: what do we! > need to do in the TC to progress the SCA Assembly spec to OASIS > Standard. The OASIS TC Process requires Statements of Use[1] from at > least three OASIS Organizational Members before a Spec can be put to > OASIS Member vote to become an OASIS Standard. The definition of > Statement of Use in OASIS is quite vague, so when chartering the SCA TCs > we agreed to strengthen the requirement and borrow from other > organisations that require more rigour. To this end we agreed to add > testing deliverables to these charters (which are not required by the TC > process). We also put in the charters wording about exit criteria [2] to > make it clear that we wanted at least two compliant implementations so > that at least two of the Statements of Use will to real and fully > compliant implementations. The only thing that is being debated is the > precise interpretation of our exit criteria, and the main issue is > whether the TC requires the implementations to pass the TC’s test suite > in ! order to qualify as one of the implementations that meet our exit > criteria. It’s a yes/no question! > > As a Chair I won’t use this email to argue one way or another, I just > want to make clear what the exact topic of the debate is. > > Cheers, > > Martin. > > [1]"Statement of Use", with respect to a Committee Specification, is a > written statement by an OASIS Organizational Member stating that it is > successfully using or implementing that specification in accordance with > the conformance clauses specified in Section 2.18 > <http://www.oasis-open.org/committees/process.php#specQuality>, and > stating whether its use included the interoperation of multiple > independent implementations. > > [2] *Exit Criteria* > > The TC shall define concrete exit criteria that include at least two > independent offerings that implement and are compliant with the all > normative portions of specifications and demonstrate interoperability > and portability as appropriate. Note that these are minimums and that > the TC is free to set more stringent criteria. > > *From:* Jim Marino [mailto:jim.marino@gmail.com] > *Sent:* 24 February 2011 19:27 > *To:* OASIS Assembly > *Subject:* Re: [sca-assembly] Concrete Exit Criteria for the SCA > Assembly TC - Proposal > > On Feb 23, 2011, at 3:06 PM, Martin Chapman wrote: > > > > Jim, > > *From:* Jim Marino [mailto:jim.marino@gmail.com] > *Sent: 23 February 2011 19:35 > To: OASIS Assembly > Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly > TC - Proposal* > > * * > > *Hi Martin,* > > * * > > *I guess I may now be even more confused, probably because of my > ignorance of OASIS rules. Why would the exit criteria for the TC require > an implementation to adhere to something that is not normative, > specifically running and passing the TC-supplied test suite?* > > * *** > > *<MC> Because that is why we have been writing a te! st suite in the TC. > The intention is that two indepe! ndent runtimes pass the TC’s test > suite. That way we demonstrate that the spec has been independently > implemented and that the test suite isn’t (too) specific to a particular > runtime*** > > * *** > > *Also, who determines whether a runtime passes or not?* > > *<MC> the Test suite, saying whether a particular test passes or not?*** > > * * > > Hi Martin, > > I think you raise two important but separate issues here: (1) > demonstrating the spec has been independently implemented; and (2) that > the test suite is too tailored to a specific runtime. > > Regarding (1), I thought it was sufficient for an entity responsible for > a runtime to declare in some public and documented way that it conforms > to an SCA specification. The entity could decide to use the TC tests, > adapt them, or write their own. Policing conformance would be left to > the court of public opinion. Is it now a requirement that a runtime > passes the TC-supplied test suite to claim conformance? If so, what does > an entity need to do to pass the tests? Is it a requirement that the > entity publish test artifacts, including source and/or binaries of the > integration code required to plug the runtime ! into the test suite? Or > can they just say they do? Are they allowed to modify the test cases to > fit them into their own integration testing infrastructure? > > My concern, similar to Eric's, is that we are going down the path of > over-specification. > > Regarding (2), my understanding was the test suite was non-normative and > similar to a set of "guidelines". If that is the case and the suite is > too tailored to a particular runtime or set of runtimes, so be it. There > is always the option of taking the test suite as inspiration and > developing one more suited to a particular platform. For example, if I > am developing a non-Java runtime, I am likely to want to develop a test > suite using the build and CI tools currently in use. I wouldn't want to > be forced to write Java code that integrates! my non-Java runtime into > the test suite and have to run the JDK-based test client. > > Jim >
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]