[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC - Proposal
On Mar 01, 2011, at 1:41 PM, Danny van der Rijn wrote: > Jeff - > > The charter requires none. > > Exit Criteria > > The TC shall define concrete exit criteria that include at least two > independent offerings that implement and are compliant with the all > normative portions of specifications and demonstrate > interoperability and portability as appropriate. Note that these are > minimums and that the TC is free to set more stringent criteria. > > 2 implementations are required to comply with the normative portions > of the specification. This discussion is about actually defining > our exit criteria, and whether we should include the test suite in > those criteria in any way. > > This discussion is far from moot if, as it sounds, we only have one > implementation that seems ready to claim conformance to our tests, > even if we have 2 or more that claim conformance to the specification. I disagree. :-) The intent was to make sure there were at least 2; and if there weren't 2, then the spec stayed in a CD state. The goal wasn't to make it easy; it was to set a high quality bar. Why would you want to call something a standard if you can't even get at least 2 implementations of it. (except for the obvious point about marketing for that one implementation.) -jeff > > On 3/1/2011 1:08 PM, Jeff Mischkinsky wrote: >> >> hi eric, >> You might be satisfied with one, but the charter requires 2. Many >> of us would argue, based on many years of hard won experience, that >> one is not enough. Two is a bare minimum. My experience is every >> time you add a new implementation to the mix, you uncover some new >> can of worms. Clearly there is a law of diminishing returns, i.e. >> the curve is a pretty steep (something approximating an inverse >> square law). After you get past 4 or 5, you are normally getting >> down to uncovering nits in the spec. >> >> We can argue and disagree all day about the correct number. Given >> that the charter says at least 2, I think its moot. I don't hear >> anybody arguing that we should require more than 2 at this time. >> cheers, >> jeff >> >> >> On Feb 25, 2011, at 9:27 AM, Eric Johnson wrote: >> >>> Hi Martin, >>> >>> As I said, I think I would be satisfied with one implementation >>> that passes the test suite. Then again, of the two announced >>> implementations I'm aware of, I don't know how close the non- >>> Tuscany one is to passing the test suite. If it passes already, >>> then the distinction between 0, 1, and 2 is mostly academic, isn't >>> it? >>> >>> -Eric. >>> >>> On 2/25/11 6:25 AM, Martin Chapman wrote: >>>> >>>> Eric, >>>> >>>> See my answer to Jim. We can argue why we built test suites, >>>> either to validate the spec or to test conformance of >>>> implementations (they are both right IMHO) but the real >>>> question is do we want two implementations to pass the TC Test >>>> suite before we can progress to an OASIS Standard vote. >>>> >>>> Martin. >>>> >>>> From: Eric Johnson [mailto:eric@tibco.com] >>>> Sent: 24 February 2011 21:40 >>>> To: Jim Marino >>>> Cc: OASIS Assembly >>>> Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA >>>> Assembly TC - Proposal >>>> >>>> Hi Jim, Jeff, >>>> >>>> On 2/24/11 10:40 AM, Jim Ma! rino wrote: >>>> Hi Jeff, >>>> >>>> Comments inline. >>>> >>>> Jim >>>> >>>> On Feb 23, 2011, at 3:05 PM, Jeff Mischkinsky wrote: >>>> >>>> >>>> >>>> On Feb 23, 2011, at 11:34 AM, Jim Marino wrote: >>>> >>>> >>>> Hi Martin, >>>> >>>> I guess I may now be even more confused, probably because of my >>>> ignorance of OASIS rules. Why would the exit criteria for the TC >>>> require an implementation to adhere to something that is not no! >>>> rmative, specifically running and passing the TC-supplied test >>>> suite? Also, who determines whether a runtime passes or not? >>>> >>>> I am having a hard time pinning down your confusion. >>>> >>>> >>>> I guess my confusion was not clear :-) >>>> >>>> I understand that the the TC-supplied test suite is not normative >>>> and that an entity responsible for an SCA runtime can >>>> legitimately claim conformance without running the TC-supplied >>>> test suite. It is up to the court of public opinion to validate >>>> conformance claims. >>>> >>>> My question was why the exit criteria would require two runtimes >>>> to also implement/run non-normative TC artifacts? That seems >>>> inconsistent with! how other non-normative aspects of the >>>> specification are handled, such as optional sections, which are >>>> not required. Are there other OASIS TCs that have adopted exit >>>> criteria which require implementations to pass conformance tests? >>>> >>>> <eej> >>>> This point, at least, I understand. As I understand it, the test >>>> suite is normative material, although not, as you note, a >>>> criteria for conforming. >>>> >>>> If I understood him correctly, during the bindings call this >>>> morning, Mike Edwards indicated he disagrees with me, but I do >>>> think that a key part of the test suite is to validate the >>>> normative statements of the spec. If you can't write a test for a >>>> particular normative statement, or a particular clause of a >>>> normative statement, then perhaps the normative statement >>>> shouldn't be normative. Or maybe you write the test, and realize >>>> while thinking through all the cases that the normative statement >>>> leaves possibilities out. Or maybe writing t! he tests highlights >>>> an area where there's too much implementation leeway, and more >>>> normative statements might be in order. The best way to find any >>>> of that out is to write a test suite. As I recall, all of these >>>> situations arose during the development of the test suites. >>>> >>>> Now that we have the tests, do we need the them to run against >>>> two implementations? Personally, I'd be satisfied by one. >>>> </eej> >>>> >>>> >>>> >>>> >>>> >>>> There is a test suite that has been developed by the TC. When you >>>> take your implementation and run it through the test suite, you >>>> get an "answer". That's the answer you are looking for. >>>> >>>> The exit criteria require that there be 2 independent impls that >>>> run the test suite, and get answer that says "yes" (or all >>>> green ;-). >>>> >>>> Did you intend to say "the proposed exit criteria"? As it >>>> currently stands, the TC charter says nothing about mandating an >>>> implementation run the test suite. Assuming exit criteria were >>>> adopted that mandated the test suite be passed by two independent >>>> runtimes, how would that be done? In other words, who decides and >>>> verifies the tests were "all green"? Does the entity responsible >>>> for a runtime need to make the test suite and integration code >>>> (or binaries) required to plug the runtime into the test suite >>>> publicly available? Or is it enough for the entity to claim its >>>> runtime passes? >>>> <eej> >>>> I concur with Jim here. If an implementation professes to >>>> implement the spec, but then fails the test suite, that's a >>>> fairly obvious point that the marketplace can sort out. Vendors >>>> might declare "we pass 90+%" for example. Leave it to the end >>>> users to figure out whether that last X perc! ent matters. >>>> </eej> >>>> >>>> >>>> >>>> jeff >>>> >>>> >>>> Jim >>>> >>>> On Feb 23, 2011, at 10:54 AM, Martin Chapman wrote: >>>> >>>> Jim, >>>> >>>> I think you are confusing the normative-ness of the test suite >>>> with TC exit criteria. >>>> You are correct that to claim conformance you don’t have to pass >>>> the TC’s test suite, but that is different from the TC saying >>>> that in order to progress the spec to Committee Specification/ >>>> OASIS Standard we want two implementations passing the TC’s test >>>> suite. >>>> >>>> Martin. >>>> >>>> From: Jim Marino [mailto:jim.marino@gmail.com] >>>> Sent: 22 February 2011 20:08 >>>> To: OASIS Assembly >>>> Subject: Re: [sca-assembly] Concrete Exit Criteria for t! he SCA >>>> Assembly TC - Proposal >>>> >>>> Hi, >>>> >>>> The first bullet makes sense to me. Concerning the second, my >>>> understanding is the TC Test Suite is not normative and the only >>>> thing required for conformance is for the entity responsible for >>>> a particular runtime to declare that it adheres to the normative >>>> portions of the Assembly Specification. In other words, it should >>>> be valid for a runtime to claim conformance that supplies its own >>>> test suite, which may or may not be based on the TC one. This >>>> essentially goes back to using the "court of public opinion" to >>>> judge conformance claims as opposed to something akin to a TCK. >>>> >>>> Jim >>>> >>>> >>>> >>>> On Feb 18, 2011, at 6:19 AM, Mike Edwards wrote: >>>> >>>> >>>> >>>> Folks, >>>> >>>> I propose the following as the Exit Criteria to be adopted by the >>>> SCA Ass! embly TC: >>>> >>>> "The Concrete Exit Criteria for the SCA Assembly V1.1 >>>> specification are that: >>>> >>>> o there shall be 2 independent SCA runtimes that are compliant >>>> with all normative portions of the >>>> specification as described in! Section 12.2 of the SCA Assembly >>>> V1.1 specification >>>> >>>> o the 2 independent runtimes shall pass the Test Suite for SCA >>>> Assembly as described in the document >>>> 'TestCases for the SCA Assembly Model Version 1.1 Specification' " >>>> >>>> >>>> I believe that this is sufficient to cover: >>>> - the question of conformance to the spec >>>> - the issues of portability and interoperability >>>> >>>> since the test suite is a pretty thorough examination of >>>> conformance to the spec (there are testcases for >>>> all normative statements with testable & observable consequences) >>>> and the test suite is also a test of portability, >>>> since the test suite has an extensive set of ar! tifacts that are >>>> ported to each runtime and interoperability is >>>> checked through the use of Web services being used between the >>>> SCA runtime and a non-SCA (JAXWS) client. >>>> >>>> >>>> >>>> Yours, Mike >>>> Dr Mike Edwards >>>> Mail Point 146, Hursley Park >>>> <Mail Attachment.gif> >>>> STSM >>>> Winchester, Hants SO21 2JN >>>> SCA & Services Standards >>>> United Kingdom >>>> Co-Chair OASIS SCA Assembly TC >>>> >>>> IBM Software Group >>>> >>>> Phone: >>>> +44-1962 818014 >>>> >>>> Mobile: >>>> +44-7802-467431 (274097) >>>> >>>> e-mail: >>>> mike_edwards@uk.ibm.com >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> Unless stat! ed otherwise above: >>>> IBM United Kingdom Limited - Registered in England and Wales with >>>> number 741598. >>>> Registered office: PO Box 41, North Harbour, Portsmouth, >>>> Hampshire PO6 3AU >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> Jeff Mischkinsky jeff.mischkinsky@oracle.com >>>> Sr. Director, Oracle Fusion Middleware +1(650)506-1975 >>>> and Web Services Standards 500 Oracle Parkway, M/S >>>> 2OP9 >>>> Oracle Redwood Shores, CA 94065 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >> >> -- >> Jeff Mischkinsky jeff.mischkinsky@oracle.com >> Sr. Director, Oracle Fusion Middleware >> +1(650)506-1975 >> and Web Services Standards 500 Oracle >> Parkway, M/S 2OP9 >> Oracle Redwood Shores, CA 94065 >> >> >> >> >> >> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe from this mail list, you must leave the OASIS TC that >> generates this mail. Follow this link to all your TCs in OASIS at: >> https://www.oasis-open.org/apps/org/workgroup/portal/ >> my_workgroups.php -- Jeff Mischkinsky jeff.mischkinsky@oracle.com Sr. Director, Oracle Fusion Middleware +1(650)506-1975 and Web Services Standards 500 Oracle Parkway, M/S 2OP9 Oracle Redwood Shores, CA 94065
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]