[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC -Proposal
Jim, There are two orthogonal discussions here. One relates to TC activities and progressing the spec and test suite to OASIS Standard level, and the other relates to how people claim conformance outside of the TC process or OASIS. Let me start with the last one. Assuming SCA Assembly gets adopted as an OASIS Standard, then anyone in the world (OASIS member or not) will be able to claim conformance if they so choose. Such claims are not *required* to be backed up by passing any test suite, though this of course is strongly encouraged and we hope public opinion will demand it. OASIS is not (yet) in the conformance testing and validation business so one would rely on customers keeping the vendors honest. I don’t see anyone disagreeing to these principles, so we don’t need to discuss this aspect anymore. What we need to get agreement on is the first discussion: what do we!
need to
do in the TC to progress the SCA Assembly spec to OASIS Standard. The OASIS TC Process requires Statements of Use[1] from at least three OASIS Organizational Members before a Spec can be put to OASIS Member vote to become an OASIS Standard. The definition of Statement of Use in OASIS is quite vague, so when chartering the SCA TCs we agreed to strengthen the requirement and borrow from other organisations that require more rigour. To this end we agreed to add testing deliverables to these charters (which are not required by the TC process). We also put in the charters wording about exit criteria [2] to make it clear that we wanted at least two compliant implementations so that at least two of the Statements of Use will to real and fully compliant implementations. The only thing that is being debated is the precise interpretation of our exit criteria, and the main issue is whether the TC requires the implementations to pass the TC’s test suite in !
order to
qualify as one of the implementations that meet our exit criteria. It’s a yes/no question! As a Chair I won’t use this email to argue one way or another, I just want to make clear what the exact topic of the debate is. Cheers, Martin. [1]"Statement of Use", with respect to a Committee Specification, is a written statement by an OASIS Organizational Member stating that it is successfully using or implementing that specification in accordance with the conformance clauses specified in Section 2.18, and stating whether its use included the interoperation of multiple independent implementations. [2] Exit Criteria The TC shall define concrete exit criteria that include at least two independent offerings that implement and are compliant with the all normative portions of specifications and demonstrate interoperability and portability as appropriate. Note that these are minimums and that the TC is free to set more stringent criteria. From: Jim Marino [mailto:jim.marino@gmail.com] On Feb 23, 2011, at 3:06 PM, Martin Chapman wrote: Jim, From: Jim Marino [mailto:jim.marino@gmail.com] Hi Martin, I guess I may now be even more confused, probably because of my ignorance of OASIS rules. Why would the exit criteria for the TC require an implementation to adhere to something that is not normative, specifically running and passing the TC-supplied test suite? <MC> Because that is why we have been writing a te!
st suite
in the TC. The intention is that two indepe! ndent runtimes pass the TC’s test suite. That way we demonstrate that the spec has been independently implemented and that the test suite isn’t (too) specific to a particular runtime Also, who determines whether a runtime passes or not? <MC> the Test suite, saying whether a particular test passes or not? Hi Martin, I think you raise two important but separate issues here: (1) demonstrating the spec has been independently implemented; and (2) that the test suite is too tailored to a specific runtime. Regarding (1), I thought it was sufficient for an entity responsible for a runtime to declare in some public and documented way that it conforms to an SCA specification. The entity could decide to use the TC tests, adapt them, or write their own. Policing conformance would be left to the court of public opinion. Is it now a requirement that a runtime passes the TC-supplied test suite to claim conformance? If so, what does an entity need to do to pass the tests? Is it a requirement that the entity publish test artifacts, including source and/or binaries of the integration code required to plug the runtime !
into the
test suite? Or can they just say they do? Are they allowed to modify the test cases to fit them into their own integration testing infrastructure? My concern, similar to Eric's, is that we are going down the path of over-specification. Regarding (2), my understanding was the test suite was non-normative and similar to a set of "guidelines". If that is the case and the suite is too tailored to a particular runtime or set of runtimes, so be it. There is always the option of taking the test suite as inspiration and developing one more suited to a particular platform. For example, if I am developing a non-Java runtime, I am likely to want to develop a test suite using the build and CI tools currently in use. I wouldn't want to be forced to write Java code that integrates!
my
non-Java runtime into the test suite and have to run the JDK-based test client. Jim |
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]