Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC - Proposal

[Jeff Mischkinsky <jeff.mischkinsky@oracle.com>]

On Mar 03, 2011, at 10:49 AM, Eric Johnson wrote:

> Are we arguing two different things in this thread?

i think we are conflating 2 different topics:
    I agree that claiming conformance does not REQUIRE passing the  
test suite. That's about what products have to do in order to make  
legitimate claims.


>
> Danny is saying that the charter does not require two  
> implementations passing the test suite.
>
> Jeff is saying that the charter does require two implementations  
> passing the test suite. At least, that's what I interpreted his  
> response to mean.

yes but that's not for the purposes of conformance. It's for  
satisfying the exit criteria, 2 impls passing the test suites which  
cover all the normative portions


>
> I side with Danny - conforming to the normative portions of the  
> specification does not (currently) imply passing the test suite.

Actually i think it does as a practical matter - irrespective of  
charter requirements, etc.
  How could a conformant implementation not pass the test suites?  
(assuming for the moment the test suites are correct and have no bugs)



> And further, based on the email from Mike about what the test cases  
> suite covers, passing the test suite does not imply conformance.
Passing any given set of tests never implies conformance. Passing the  
test suites is a necessary, not sufficient condition.

-jeff

>
> Of course, you're welcome to raise an issue that you think  
> conformance should mean passing the test suite as well, in which  
> case you're going above and beyond what the charter says.
>
> -Eric.
>
> On 3/2/11 8:20 PM, Jeff Mischkinsky wrote:
>>
>> On Mar 01, 2011, at 1:41 PM, Danny van der Rijn wrote:
>>
>>> Jeff -
>>>
>>> The charter requires none.
>>>
>>> Exit Criteria
>>>
>>> The TC shall define concrete exit criteria that include at least  
>>> two independent offerings that implement and are compliant with  
>>> the all normative portions of specifications and      demonstrate  
>>> interoperability and portability as appropriate. Note that these  
>>> are minimums and that the TC is free to set more stringent criteria.
>>>
>>> 2 implementations are required to comply with the normative  
>>> portions of the specification.  This discussion is about actually  
>>> defining our exit criteria, and whether we should include the test  
>>> suite in those criteria in any way.
>>>
>>> This discussion is far from moot if, as it sounds, we only have  
>>> one implementation that seems ready to claim conformance to our  
>>> tests, even if we have 2 or more that claim conformance to the  
>>> specification.
>>
>> I disagree. :-) The intent was to make sure there were at least 2;  
>> and if there weren't 2, then the spec stayed in a CD state. The  
>> goal wasn't to make it easy; it was to set a high quality bar.
>>
>> Why would you want to call something a standard if you can't even  
>> get at least 2 implementations of it. (except for the obvious point  
>> about marketing for that one implementation.)
>>
>> -jeff
>>
>>>
>>> On 3/1/2011 1:08 PM, Jeff Mischkinsky wrote:
>>>>
>>>> hi eric,
>>>>  You might be satisfied with one, but the charter requires 2.  
>>>> Many of us would argue, based on many years of hard won  
>>>> experience, that one is not enough. Two is a bare minimum. My  
>>>> experience is every time you add a new implementation to the mix,  
>>>> you uncover some new can of worms. Clearly there is a law of  
>>>> diminishing returns, i.e. the curve is a pretty steep (something  
>>>> approximating an inverse square law). After you get past 4 or 5,  
>>>> you are normally getting down to uncovering nits in the spec.
>>>>
>>>> We can argue and disagree all day about the correct number. Given  
>>>> that the charter says at least 2, I think its moot. I don't hear  
>>>> anybody arguing that we should require more than 2 at this time.
>>>> cheers,
>>>>  jeff
>>>>
>>>>
>>>> On Feb 25, 2011, at 9:27 AM, Eric Johnson wrote:
>>>>
>>>>> Hi Martin,
>>>>>
>>>>> As I said, I think I would be satisfied with one implementation  
>>>>> that passes the test suite. Then again, of the two announced  
>>>>> implementations I'm aware of, I don't know how close the non- 
>>>>> Tuscany one is to passing the test suite. If it passes already,  
>>>>> then the distinction between 0, 1, and 2 is mostly academic,  
>>>>> isn't it?
>>>>>
>>>>> -Eric.
>>>>>
>>>>> On 2/25/11 6:25 AM, Martin Chapman wrote:
>>>>>>
>>>>>> Eric,
>>>>>>
>>>>>> See my answer to Jim. We can argue why we built  test suites,  
>>>>>> either to validate the spec or to test conformance of  
>>>>>> implementations (they are both right IMHO) but the  
>>>>>> real          question is do we want two implementations to  
>>>>>> pass the TC Test suite before we can progress to an OASIS  
>>>>>> Standard vote.
>>>>>>
>>>>>> Martin.
>>>>>>
>>>>>> From: Eric Johnson [mailto:eric@tibco.com]
>>>>>> Sent: 24 February 2011 21:40
>>>>>> To: Jim Marino
>>>>>> Cc: OASIS Assembly
>>>>>> Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA  
>>>>>> Assembly TC - Proposal
>>>>>>
>>>>>> Hi Jim, Jeff,
>>>>>>
>>>>>> On 2/24/11 10:40 AM, Jim Ma! rino wrote:
>>>>>> Hi Jeff,
>>>>>>
>>>>>> Comments inline.
>>>>>>
>>>>>> Jim
>>>>>>
>>>>>> On Feb 23, 2011, at 3:05 PM, Jeff Mischkinsky wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 23, 2011, at 11:34 AM, Jim Marino wrote:
>>>>>>
>>>>>>
>>>>>> Hi Martin,
>>>>>>
>>>>>> I guess I may now be even more confused, probably because of my  
>>>>>> ignorance of OASIS rules. Why would the exit criteria for the  
>>>>>> TC require an implementation to adhere to something that is not  
>>>>>> no! rmative, specifically running and passing the TC-supplied  
>>>>>> test suite? Also, who determines whether a runtime passes or not?
>>>>>>
>>>>>> I am having a hard time pinning down your confusion.
>>>>>>
>>>>>>
>>>>>> I guess my confusion was not clear :-)
>>>>>>
>>>>>> I understand that the the TC-supplied test suite is not  
>>>>>> normative and that an entity responsible for an SCA runtime can  
>>>>>> legitimately claim conformance without running the TC-supplied  
>>>>>> test suite. It is up to the court of public opinion to validate  
>>>>>> conformance claims.
>>>>>>
>>>>>> My question was why the exit criteria would require two  
>>>>>> runtimes to also implement/run non-normative TC artifacts? That  
>>>>>> seems inconsistent with! how other non-normative aspects of the  
>>>>>> specification are handled, such as optional sections, which are  
>>>>>> not required. Are there other OASIS TCs that have adopted exit  
>>>>>> criteria which require implementations to pass conformance tests?
>>>>>>
>>>>>> <eej>
>>>>>> This point, at least, I understand. As I understand it, the  
>>>>>> test suite is normative material, although not, as you note, a  
>>>>>> criteria for conforming.
>>>>>>
>>>>>> If I understood him correctly, during the bindings call this  
>>>>>> morning, Mike Edwards indicated he disagrees with me, but I do  
>>>>>> think that a key part of the test suite is to validate the  
>>>>>> normative statements of the spec. If you can't write a test for  
>>>>>> a particular normative statement, or a particular clause of a  
>>>>>> normative statement, then perhaps the normative statement  
>>>>>> shouldn't be normative. Or maybe you write the test, and  
>>>>>> realize while thinking through all the cases that the normative  
>>>>>> statement leaves possibilities out. Or maybe writing t! he  
>>>>>> tests highlights an area where there's too much implementation  
>>>>>> leeway, and more normative statements might be in order. The  
>>>>>> best way to find any of that out is to write a test suite. As I  
>>>>>> recall, all of these situations arose during the development of  
>>>>>> the test suites.
>>>>>>
>>>>>> Now that we have the tests, do we need the them to run against  
>>>>>> two implementations? Personally, I'd be satisfied by one.
>>>>>> </eej>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> There is a test suite that has been developed by the TC. When  
>>>>>> you take your implementation and run it through the test suite,  
>>>>>> you get an "answer". That's the answer you are looking for.
>>>>>>
>>>>>> The exit criteria require that there be 2 independent impls  
>>>>>> that run the test suite, and get answer that says "yes" (or all  
>>>>>> green ;-).
>>>>>>
>>>>>> Did you intend to say "the proposed exit criteria"? As it  
>>>>>> currently stands, the TC charter says nothing about mandating  
>>>>>> an implementation run the test suite. Assuming exit criteria  
>>>>>> were adopted that mandated the test suite be passed by two  
>>>>>> independent runtimes, how would that be done? In other words,  
>>>>>> who decides and verifies the tests were "all green"? Does the  
>>>>>> entity responsible for a runtime need to make the test suite  
>>>>>> and integration code (or binaries) required to plug the runtime  
>>>>>> into the test suite publicly available? Or is it enough for the  
>>>>>> entity to claim its runtime passes?
>>>>>> <eej>
>>>>>> I concur with Jim here. If an implementation professes to  
>>>>>> implement the spec, but then fails the test suite, that's a  
>>>>>> fairly obvious point that the marketplace can sort out. Vendors  
>>>>>> might declare "we pass 90+%" for example. Leave it to the end  
>>>>>> users to figure out whether that last X perc! ent matters.
>>>>>> </eej>
>>>>>>
>>>>>>
>>>>>>
>>>>>>  jeff
>>>>>>
>>>>>>
>>>>>> Jim
>>>>>>
>>>>>> On Feb 23, 2011, at 10:54 AM, Martin Chapman wrote:
>>>>>>
>>>>>> Jim,
>>>>>>
>>>>>> I think you are confusing the normative-ness of the test suite  
>>>>>> with TC exit criteria.
>>>>>> You are correct that to claim conformance you don’t have to  
>>>>>> pass the TC’s test suite, but that is different from the TC  
>>>>>> saying that in order to progress the spec to Committee  
>>>>>> Specification/OASIS Standard we want two implementations  
>>>>>> passing the TC’s test suite.
>>>>>>
>>>>>> Martin.
>>>>>>
>>>>>> From: Jim Marino [mailto:jim.marino@gmail.com]
>>>>>> Sent: 22 February 2011 20:08
>>>>>> To: OASIS Assembly
>>>>>> Subject: Re: [sca-assembly] Concrete Exit Criteria for t! he  
>>>>>> SCA Assembly TC - Proposal
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> The first bullet makes sense to me. Concerning the second, my  
>>>>>> understanding is the TC Test Suite is not normative and the  
>>>>>> only thing required for conformance is for the entity  
>>>>>> responsible for a particular runtime to declare that it adheres  
>>>>>> to the normative portions of the Assembly Specification. In  
>>>>>> other words, it should be valid for a runtime to claim  
>>>>>> conformance that supplies its own test suite, which may or may  
>>>>>> not be based on the TC one. This essentially goes back to using  
>>>>>> the "court of public opinion" to judge conformance claims as  
>>>>>> opposed to something akin to a TCK.
>>>>>>
>>>>>> Jim
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Feb 18, 2011, at 6:19 AM, Mike Edwards wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> I propose the following as the Exit Criteria to be adopted by  
>>>>>> the SCA Ass! embly TC:
>>>>>>
>>>>>> "The Concrete Exit Criteria for the SCA Assembly V1.1  
>>>>>> specification are that:
>>>>>>
>>>>>> o there shall be 2 independent SCA runtimes that are compliant  
>>>>>> with all normative portions of the
>>>>>> specification as described in! Section 12.2 of the SCA Assembly  
>>>>>> V1.1 specification
>>>>>>
>>>>>> o the 2 independent runtimes shall pass the Test Suite for SCA  
>>>>>> Assembly as described in the document
>>>>>> 'TestCases for the SCA Assembly Model Version 1.1  
>>>>>> Specification' "
>>>>>>
>>>>>>
>>>>>> I believe that this is sufficient to cover:
>>>>>> - the question of conformance to the spec
>>>>>> - the issues of portability and interoperability
>>>>>>
>>>>>> since the test suite is a pretty thorough examination of  
>>>>>> conformance to the spec (there are testcases for
>>>>>> all normative statements with testable & observable  
>>>>>> consequences) and the test suite is also a test of portability,
>>>>>> since the test suite has an extensive set of ar! tifacts that  
>>>>>> are ported to each runtime and interoperability is
>>>>>> checked through the use of Web services being used between the  
>>>>>> SCA runtime and a non-SCA (JAXWS) client.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Yours, Mike
>>>>>> Dr Mike Edwards
>>>>>> Mail Point 146, Hursley Park
>>>>>> <Mail Attachment.gif>
>>>>>> STSM
>>>>>> Winchester, Hants SO21 2JN
>>>>>> SCA & Services Standards
>>>>>> United Kingdom
>>>>>> Co-Chair OASIS SCA Assembly TC
>>>>>>
>>>>>> IBM Software Group
>>>>>>
>>>>>> Phone:
>>>>>> +44-1962 818014
>>>>>>
>>>>>> Mobile:
>>>>>> +44-7802-467431 (274097)
>>>>>>
>>>>>> e-mail:
>>>>>> mike_edwards@uk.ibm.com
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Unless stat! ed otherwise above:
>>>>>> IBM United Kingdom Limited - Registered in England and Wales  
>>>>>> with number 741598.
>>>>>> Registered office: PO Box 41, North Harbour, Portsmouth,  
>>>>>> Hampshire PO6 3AU
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -- 
>>>>>> Jeff Mischkinsky               jeff.mischkinsky@oracle.com
>>>>>> Sr. Director, Oracle Fusion Middleware     +1(650)506-1975
>>>>>> and Web Services Standards              500 Oracle Parkway, M/S  
>>>>>> 2OP9
>>>>>> Oracle        Redwood Shores, CA 94065
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>
>>>> -- 
>>>> Jeff Mischkinsky                              jeff.mischkinsky@oracle.com
>>>> Sr. Director, Oracle Fusion Middleware                  
>>>> +1(650)506-1975
>>>>    and Web Services Standards                       500 Oracle  
>>>> Parkway, M/S 2OP9
>>>> Oracle                                Redwood Shores, CA 94065
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe from this mail list, you must leave the OASIS TC  
>>>> that
>>>> generates this mail.  Follow this link to all your TCs in OASIS at:
>>>> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
>>
>> -- 
>> Jeff Mischkinsky                              jeff.mischkinsky@oracle.com
>> Sr. Director, Oracle Fusion Middleware                  
>> +1(650)506-1975
>>    and Web Services Standards                       500 Oracle  
>> Parkway, M/S 2OP9
>> Oracle                                Redwood Shores, CA 94065
>>
>>
>>
>>
>>
>>
>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

--
Jeff Mischkinsky			          		jeff.mischkinsky@oracle.com
Sr. Director, Oracle Fusion Middleware 				+1(650)506-1975
	and Web Services Standards           			500 Oracle Parkway, M/S 2OP9
Oracle								Redwood Shores, CA 94065