OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

sca-assembly message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA Assembly TC- Proposal


Hi Jim, Jeff,

On 2/24/11 10:40 AM, Jim Marino wrote:
B7784E86-F596-41D7-9522-B2A8BA62FD1E@gmail.com" type="cite">Hi Jeff,

Comments inline.

Jim

On Feb 23, 2011, at 3:05 PM, Jeff Mischkinsky wrote:


On Feb 23, 2011, at 11:34 AM, Jim Marino wrote:

Hi Martin,

I guess I may now be even more confused, probably because of my ignorance of OASIS rules. Why would the exit criteria for the TC require an implementation to adhere to something that is not normative, specifically running and passing the TC-supplied test suite? Also, who determines whether a runtime passes or not?

I am having a hard time pinning down your confusion.


I guess my confusion was not clear :-)†

I understand that the the TC-supplied test suite is not normative and that an entity responsible for an SCA runtime can legitimately claim conformance without running the TC-supplied test suite. It is up to the court of public opinion to validate conformance claims. †

My question was why the exit criteria would require two runtimes to also implement/run non-normative TC artifacts? That seems inconsistent with how other non-normative aspects of the specification are handled, such as optional sections, which are not required. Are there other OASIS TCs that have adopted exit criteria which require implementations to pass conformance tests?

<eej>
This point, at least, I understand. As I understand it, the test suite is normative material, although not, as you note, a criteria for conforming.

If I understood him correctly, during the bindings call this morning, Mike Edwards indicated he disagrees with me, but I do think that a key part of the test suite is to validate the normative statements of the spec. If you can't write a test for a particular normative statement, or a particular clause of a normative statement, then perhaps the normative statement shouldn't be normative. Or maybe you write the test, and realize while thinking through all the cases that the normative statement leaves possibilities out. Or maybe writing the tests highlights an area where there's too much implementation leeway, and more normative statements might be in order. The best way to find any of that out is to write a test suite. As I recall, all of these situations arose during the development of the test suites.

Now that we have the tests, do we need the them to run against two implementations? Personally, I'd be satisfied by one.
</eej>

B7784E86-F596-41D7-9522-B2A8BA62FD1E@gmail.com" type="cite">


There is a test suite that has been developed by the TC. When you take your implementation and run it through the test suite, you get an "answer". That's the answer you are looking for.

The exit criteria require that there be 2 independent impls that run the test suite, and get answer that says "yes" (or all green ;-).

Did you intend to say "the proposed exit criteria"? As it currently stands, the TC charter says nothing about mandating an implementation run the test suite. Assuming exit criteria were adopted that mandated the test suite be passed by two independent runtimes, how would that be done? In other words, who decides and verifies the tests were "all green"? Does the entity responsible for a runtime need to make the test suite and integration code (or binaries) required to plug the runtime into the test suite publicly available? Or is it enough for the entity to claim its runtime passes?
<eej>
I concur with Jim here. If an implementation professes to implement the spec, but then fails the test suite, that's a fairly obvious point that the marketplace can sort out. Vendors might declare "we pass 90+%" for example. Leave it to the end users to figure out whether that last X percent matters.
</eej>
B7784E86-F596-41D7-9522-B2A8BA62FD1E@gmail.com" type="cite">


††jeff

Jim

On Feb 23, 2011, at 10:54 AM, Martin Chapman wrote:

Jim,

I think you are confusing the normative-ness of the test suite with TC exit criteria.
You are correct that to claim conformance you donít have to pass the TCís test suite, but that is different from the TC saying that in order to progress the spec to Committee Specification/OASIS Standard we want two implementations passing the TCís test suite.

Martin.

From: Jim Marino [mailto:jim.marino@gmail.com]
Sent: 22 February 2011 20:08
To: OASIS Assembly
Subject: Re: [sca-assembly] Concrete Exit Criteria for t! he SCA Assembly TC - Proposal

Hi,

The first bullet makes sense to me. Concerning the second, my understanding is the TC Test Suite is not normative and the only thing required for conformance is for the entity responsible for a particular runtime to declare that it adheres to the normative portions of the Assembly Specification. In other words, it should be valid for a runtime to claim conformance that supplies its own test suite, which may or may not be based on the TC one. This essentially goes back to using the "court of public opinion" to judge conformance claims as opposed to something akin to a TCK.

Jim



On Feb 18, 2011, at 6:19 AM, Mike Edwards wrote:



Folks,

I propose the following as the Exit Criteria to be adopted by the SCA Assembly TC:

"The Concrete Exit Criteria for the SCA Assembly V1.1 specification are that:

o there shall be 2 independent SCA runtimes that are compliant with all normative portions of the
specification as described in! Section 12.2 of the SCA Assembly V1.1 specification

o the 2 independent runtimes shall pass the Test Suite for SCA Assembly as described in the document
'TestCases for the SCA Assembly Model Version 1.1 Specification' "


I believe that this is sufficient to cover:
- the question of conformance to the spec
- the issues of portability and interoperability

since the test suite is a pretty thorough examination of conformance to the spec (there are testcases for
all normative statements with testable & observable consequences) and the test suite is also a test of portability,
since the test suite has an extensive set of artifacts that are ported to each runtime and interoperability is
checked through the use of Web services being used between the SCA runtime and a non-SCA (JAXWS) client.



Yours, Mike
Dr Mike Edwards
Mail Point 146, Hursley Park
<Mail Attachment.gif>
STSM
Winchester, Hants SO21 2JN
SCA & Services Standards
United Kingdom
Co-Chair OASIS SCA Assembly TC

IBM Software Group

Phone:
+44-1962 818014

Mobile:
+44-7802-467431 (274097)

e-mail:
mike_edwards@uk.ibm.com









Unless stat! ed otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU









--
Jeff Mischkinsky ††††††††† jeff.mischkinsky@oracle.com
Sr. Director, Oracle Fusion Middleware +1(650)506-1975
and Web Services Standards †††††††††† 500 Oracle Parkway, M/S 2OP9
Oracle Redwood Shores, CA 94065












[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]