[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC - Proposal
On Mar 03, 2011, at 10:49 AM, Eric Johnson wrote: > Are we arguing two different things in this thread? i think we are conflating 2 different topics: I agree that claiming conformance does not REQUIRE passing the test suite. That's about what products have to do in order to make legitimate claims. > > Danny is saying that the charter does not require two > implementations passing the test suite. > > Jeff is saying that the charter does require two implementations > passing the test suite. At least, that's what I interpreted his > response to mean. yes but that's not for the purposes of conformance. It's for satisfying the exit criteria, 2 impls passing the test suites which cover all the normative portions > > I side with Danny - conforming to the normative portions of the > specification does not (currently) imply passing the test suite. Actually i think it does as a practical matter - irrespective of charter requirements, etc. How could a conformant implementation not pass the test suites? (assuming for the moment the test suites are correct and have no bugs) > And further, based on the email from Mike about what the test cases > suite covers, passing the test suite does not imply conformance. Passing any given set of tests never implies conformance. Passing the test suites is a necessary, not sufficient condition. -jeff > > Of course, you're welcome to raise an issue that you think > conformance should mean passing the test suite as well, in which > case you're going above and beyond what the charter says. > > -Eric. > > On 3/2/11 8:20 PM, Jeff Mischkinsky wrote: >> >> On Mar 01, 2011, at 1:41 PM, Danny van der Rijn wrote: >> >>> Jeff - >>> >>> The charter requires none. >>> >>> Exit Criteria >>> >>> The TC shall define concrete exit criteria that include at least >>> two independent offerings that implement and are compliant with >>> the all normative portions of specifications and demonstrate >>> interoperability and portability as appropriate. Note that these >>> are minimums and that the TC is free to set more stringent criteria. >>> >>> 2 implementations are required to comply with the normative >>> portions of the specification. This discussion is about actually >>> defining our exit criteria, and whether we should include the test >>> suite in those criteria in any way. >>> >>> This discussion is far from moot if, as it sounds, we only have >>> one implementation that seems ready to claim conformance to our >>> tests, even if we have 2 or more that claim conformance to the >>> specification. >> >> I disagree. :-) The intent was to make sure there were at least 2; >> and if there weren't 2, then the spec stayed in a CD state. The >> goal wasn't to make it easy; it was to set a high quality bar. >> >> Why would you want to call something a standard if you can't even >> get at least 2 implementations of it. (except for the obvious point >> about marketing for that one implementation.) >> >> -jeff >> >>> >>> On 3/1/2011 1:08 PM, Jeff Mischkinsky wrote: >>>> >>>> hi eric, >>>> You might be satisfied with one, but the charter requires 2. >>>> Many of us would argue, based on many years of hard won >>>> experience, that one is not enough. Two is a bare minimum. My >>>> experience is every time you add a new implementation to the mix, >>>> you uncover some new can of worms. Clearly there is a law of >>>> diminishing returns, i.e. the curve is a pretty steep (something >>>> approximating an inverse square law). After you get past 4 or 5, >>>> you are normally getting down to uncovering nits in the spec. >>>> >>>> We can argue and disagree all day about the correct number. Given >>>> that the charter says at least 2, I think its moot. I don't hear >>>> anybody arguing that we should require more than 2 at this time. >>>> cheers, >>>> jeff >>>> >>>> >>>> On Feb 25, 2011, at 9:27 AM, Eric Johnson wrote: >>>> >>>>> Hi Martin, >>>>> >>>>> As I said, I think I would be satisfied with one implementation >>>>> that passes the test suite. Then again, of the two announced >>>>> implementations I'm aware of, I don't know how close the non- >>>>> Tuscany one is to passing the test suite. If it passes already, >>>>> then the distinction between 0, 1, and 2 is mostly academic, >>>>> isn't it? >>>>> >>>>> -Eric. >>>>> >>>>> On 2/25/11 6:25 AM, Martin Chapman wrote: >>>>>> >>>>>> Eric, >>>>>> >>>>>> See my answer to Jim. We can argue why we built test suites, >>>>>> either to validate the spec or to test conformance of >>>>>> implementations (they are both right IMHO) but the >>>>>> real question is do we want two implementations to >>>>>> pass the TC Test suite before we can progress to an OASIS >>>>>> Standard vote. >>>>>> >>>>>> Martin. >>>>>> >>>>>> From: Eric Johnson [mailto:eric@tibco.com] >>>>>> Sent: 24 February 2011 21:40 >>>>>> To: Jim Marino >>>>>> Cc: OASIS Assembly >>>>>> Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA >>>>>> Assembly TC - Proposal >>>>>> >>>>>> Hi Jim, Jeff, >>>>>> >>>>>> On 2/24/11 10:40 AM, Jim Ma! rino wrote: >>>>>> Hi Jeff, >>>>>> >>>>>> Comments inline. >>>>>> >>>>>> Jim >>>>>> >>>>>> On Feb 23, 2011, at 3:05 PM, Jeff Mischkinsky wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Feb 23, 2011, at 11:34 AM, Jim Marino wrote: >>>>>> >>>>>> >>>>>> Hi Martin, >>>>>> >>>>>> I guess I may now be even more confused, probably because of my >>>>>> ignorance of OASIS rules. Why would the exit criteria for the >>>>>> TC require an implementation to adhere to something that is not >>>>>> no! rmative, specifically running and passing the TC-supplied >>>>>> test suite? Also, who determines whether a runtime passes or not? >>>>>> >>>>>> I am having a hard time pinning down your confusion. >>>>>> >>>>>> >>>>>> I guess my confusion was not clear :-) >>>>>> >>>>>> I understand that the the TC-supplied test suite is not >>>>>> normative and that an entity responsible for an SCA runtime can >>>>>> legitimately claim conformance without running the TC-supplied >>>>>> test suite. It is up to the court of public opinion to validate >>>>>> conformance claims. >>>>>> >>>>>> My question was why the exit criteria would require two >>>>>> runtimes to also implement/run non-normative TC artifacts? That >>>>>> seems inconsistent with! how other non-normative aspects of the >>>>>> specification are handled, such as optional sections, which are >>>>>> not required. Are there other OASIS TCs that have adopted exit >>>>>> criteria which require implementations to pass conformance tests? >>>>>> >>>>>> <eej> >>>>>> This point, at least, I understand. As I understand it, the >>>>>> test suite is normative material, although not, as you note, a >>>>>> criteria for conforming. >>>>>> >>>>>> If I understood him correctly, during the bindings call this >>>>>> morning, Mike Edwards indicated he disagrees with me, but I do >>>>>> think that a key part of the test suite is to validate the >>>>>> normative statements of the spec. If you can't write a test for >>>>>> a particular normative statement, or a particular clause of a >>>>>> normative statement, then perhaps the normative statement >>>>>> shouldn't be normative. Or maybe you write the test, and >>>>>> realize while thinking through all the cases that the normative >>>>>> statement leaves possibilities out. Or maybe writing t! he >>>>>> tests highlights an area where there's too much implementation >>>>>> leeway, and more normative statements might be in order. The >>>>>> best way to find any of that out is to write a test suite. As I >>>>>> recall, all of these situations arose during the development of >>>>>> the test suites. >>>>>> >>>>>> Now that we have the tests, do we need the them to run against >>>>>> two implementations? Personally, I'd be satisfied by one. >>>>>> </eej> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> There is a test suite that has been developed by the TC. When >>>>>> you take your implementation and run it through the test suite, >>>>>> you get an "answer". That's the answer you are looking for. >>>>>> >>>>>> The exit criteria require that there be 2 independent impls >>>>>> that run the test suite, and get answer that says "yes" (or all >>>>>> green ;-). >>>>>> >>>>>> Did you intend to say "the proposed exit criteria"? As it >>>>>> currently stands, the TC charter says nothing about mandating >>>>>> an implementation run the test suite. Assuming exit criteria >>>>>> were adopted that mandated the test suite be passed by two >>>>>> independent runtimes, how would that be done? In other words, >>>>>> who decides and verifies the tests were "all green"? Does the >>>>>> entity responsible for a runtime need to make the test suite >>>>>> and integration code (or binaries) required to plug the runtime >>>>>> into the test suite publicly available? Or is it enough for the >>>>>> entity to claim its runtime passes? >>>>>> <eej> >>>>>> I concur with Jim here. If an implementation professes to >>>>>> implement the spec, but then fails the test suite, that's a >>>>>> fairly obvious point that the marketplace can sort out. Vendors >>>>>> might declare "we pass 90+%" for example. Leave it to the end >>>>>> users to figure out whether that last X perc! ent matters. >>>>>> </eej> >>>>>> >>>>>> >>>>>> >>>>>> jeff >>>>>> >>>>>> >>>>>> Jim >>>>>> >>>>>> On Feb 23, 2011, at 10:54 AM, Martin Chapman wrote: >>>>>> >>>>>> Jim, >>>>>> >>>>>> I think you are confusing the normative-ness of the test suite >>>>>> with TC exit criteria. >>>>>> You are correct that to claim conformance you don’t have to >>>>>> pass the TC’s test suite, but that is different from the TC >>>>>> saying that in order to progress the spec to Committee >>>>>> Specification/OASIS Standard we want two implementations >>>>>> passing the TC’s test suite. >>>>>> >>>>>> Martin. >>>>>> >>>>>> From: Jim Marino [mailto:jim.marino@gmail.com] >>>>>> Sent: 22 February 2011 20:08 >>>>>> To: OASIS Assembly >>>>>> Subject: Re: [sca-assembly] Concrete Exit Criteria for t! he >>>>>> SCA Assembly TC - Proposal >>>>>> >>>>>> Hi, >>>>>> >>>>>> The first bullet makes sense to me. Concerning the second, my >>>>>> understanding is the TC Test Suite is not normative and the >>>>>> only thing required for conformance is for the entity >>>>>> responsible for a particular runtime to declare that it adheres >>>>>> to the normative portions of the Assembly Specification. In >>>>>> other words, it should be valid for a runtime to claim >>>>>> conformance that supplies its own test suite, which may or may >>>>>> not be based on the TC one. This essentially goes back to using >>>>>> the "court of public opinion" to judge conformance claims as >>>>>> opposed to something akin to a TCK. >>>>>> >>>>>> Jim >>>>>> >>>>>> >>>>>> >>>>>> On Feb 18, 2011, at 6:19 AM, Mike Edwards wrote: >>>>>> >>>>>> >>>>>> >>>>>> Folks, >>>>>> >>>>>> I propose the following as the Exit Criteria to be adopted by >>>>>> the SCA Ass! embly TC: >>>>>> >>>>>> "The Concrete Exit Criteria for the SCA Assembly V1.1 >>>>>> specification are that: >>>>>> >>>>>> o there shall be 2 independent SCA runtimes that are compliant >>>>>> with all normative portions of the >>>>>> specification as described in! Section 12.2 of the SCA Assembly >>>>>> V1.1 specification >>>>>> >>>>>> o the 2 independent runtimes shall pass the Test Suite for SCA >>>>>> Assembly as described in the document >>>>>> 'TestCases for the SCA Assembly Model Version 1.1 >>>>>> Specification' " >>>>>> >>>>>> >>>>>> I believe that this is sufficient to cover: >>>>>> - the question of conformance to the spec >>>>>> - the issues of portability and interoperability >>>>>> >>>>>> since the test suite is a pretty thorough examination of >>>>>> conformance to the spec (there are testcases for >>>>>> all normative statements with testable & observable >>>>>> consequences) and the test suite is also a test of portability, >>>>>> since the test suite has an extensive set of ar! tifacts that >>>>>> are ported to each runtime and interoperability is >>>>>> checked through the use of Web services being used between the >>>>>> SCA runtime and a non-SCA (JAXWS) client. >>>>>> >>>>>> >>>>>> >>>>>> Yours, Mike >>>>>> Dr Mike Edwards >>>>>> Mail Point 146, Hursley Park >>>>>> <Mail Attachment.gif> >>>>>> STSM >>>>>> Winchester, Hants SO21 2JN >>>>>> SCA & Services Standards >>>>>> United Kingdom >>>>>> Co-Chair OASIS SCA Assembly TC >>>>>> >>>>>> IBM Software Group >>>>>> >>>>>> Phone: >>>>>> +44-1962 818014 >>>>>> >>>>>> Mobile: >>>>>> +44-7802-467431 (274097) >>>>>> >>>>>> e-mail: >>>>>> mike_edwards@uk.ibm.com >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Unless stat! ed otherwise above: >>>>>> IBM United Kingdom Limited - Registered in England and Wales >>>>>> with number 741598. >>>>>> Registered office: PO Box 41, North Harbour, Portsmouth, >>>>>> Hampshire PO6 3AU >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Jeff Mischkinsky jeff.mischkinsky@oracle.com >>>>>> Sr. Director, Oracle Fusion Middleware +1(650)506-1975 >>>>>> and Web Services Standards 500 Oracle Parkway, M/S >>>>>> 2OP9 >>>>>> Oracle Redwood Shores, CA 94065 >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>> >>>> -- >>>> Jeff Mischkinsky jeff.mischkinsky@oracle.com >>>> Sr. Director, Oracle Fusion Middleware >>>> +1(650)506-1975 >>>> and Web Services Standards 500 Oracle >>>> Parkway, M/S 2OP9 >>>> Oracle Redwood Shores, CA 94065 >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe from this mail list, you must leave the OASIS TC >>>> that >>>> generates this mail. Follow this link to all your TCs in OASIS at: >>>> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php >> >> -- >> Jeff Mischkinsky jeff.mischkinsky@oracle.com >> Sr. Director, Oracle Fusion Middleware >> +1(650)506-1975 >> and Web Services Standards 500 Oracle >> Parkway, M/S 2OP9 >> Oracle Redwood Shores, CA 94065 >> >> >> >> >> >> >> >> >> > > --------------------------------------------------------------------- > To unsubscribe from this mail list, you must leave the OASIS TC that > generates this mail. Follow this link to all your TCs in OASIS at: > https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php -- Jeff Mischkinsky jeff.mischkinsky@oracle.com Sr. Director, Oracle Fusion Middleware +1(650)506-1975 and Web Services Standards 500 Oracle Parkway, M/S 2OP9 Oracle Redwood Shores, CA 94065
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]