Jeff -
The charter requires none.
Exit
Criteria
The TC shall define concrete exit criteria that include
at least two independent offerings that implement and are
compliant with the all normative portions of specifications and
demonstrate interoperability and portability as appropriate. Note
that these are minimums and that the TC is free to set more
stringent criteria.
2 implementations are required to comply with the normative portions
of the specification. This discussion is about actually defining
our exit criteria, and whether we should include the test suite in
those criteria in any way.
This discussion is far from moot if, as it sounds, we only have one
implementation that seems ready to claim conformance to our tests,
even if we have 2 or more that claim conformance to the
specification.
On 3/1/2011 1:08 PM, Jeff Mischkinsky wrote:
D39FD8D1-CC2B-499D-9FC5-EC0116F37A26@oracle.com"
type="cite">hi eric,
You might be satisfied with one, but the charter requires 2.
Many of us would argue, based on many years of hard won
experience, that one is not enough. Two is a bare minimum. My
experience is every time you add a new implementation to the mix,
you uncover some new can of worms. Clearly there is a law of
diminishing returns, i.e. the curve is a pretty steep (something
approximating an inverse square law). After you get past 4 or 5,
you are normally getting down to uncovering nits in the spec.
We can argue and disagree all day about the correct number. Given
that the charter says at least 2, I think its moot. I don't hear
anybody arguing that we should require more than 2 at this time.
cheers,
jeff
On Feb 25, 2011, at 9:27 AM, Eric Johnson wrote:
Hi Martin,
As I said, I think I would be satisfied with one implementation
that passes the test suite. Then again, of the two announced
implementations I'm aware of, I don't know how close the
non-Tuscany one is to passing the test suite. If it passes
already, then the distinction between 0, 1, and 2 is mostly
academic, isn't it?
-Eric.
On 2/25/11 6:25 AM, Martin Chapman wrote:
Eric,
See my answer to Jim. We can argue why we built test suites,
either to validate the spec or to test conformance of
implementations (they are both right IMHO) but the real
question is do we want two implementations to pass the TC Test
suite before we can progress to an OASIS Standard vote.
Martin.
From: Eric Johnson [mailto:eric@tibco.com]
Sent: 24 February 2011 21:40
To: Jim Marino
Cc: OASIS Assembly
Subject: Re: [sca-assembly] Concrete Exit Criteria for the SCA
Assembly TC - Proposal
Hi Jim, Jeff,
On 2/24/11 10:40 AM, Jim Ma! rino wrote:
Hi Jeff,
Comments inline.
Jim
On Feb 23, 2011, at 3:05 PM, Jeff Mischkinsky wrote:
On Feb 23, 2011, at 11:34 AM, Jim Marino wrote:
Hi Martin,
I guess I may now be even more confused, probably because of
my ignorance of OASIS rules. Why would the exit criteria for
the TC require an implementation to adhere to something that
is not no! rmative, specifically running and passing the
TC-supplied test suite? Also, who determines whether a runtime
passes or not?
I am having a hard time pinning down your confusion.
I guess my confusion was not clear :-)
I understand that the the TC-supplied test suite is not
normative and that an entity responsible for an SCA runtime
can legitimately claim conformance without running the
TC-supplied test suite. It is up to the court of public
opinion to validate conformance claims.
My question was why the exit criteria would require two
runtimes to also implement/run non-normative TC artifacts?
That seems inconsistent with! how other non-normative aspects
of the specification are handled, such as optional sections,
which are not required. Are there other OASIS TCs that have
adopted exit criteria which require implementations to pass
conformance tests?
<eej>
This point, at least, I understand. As I understand it, the
test suite is normative material, although not, as you note, a
criteria for conforming.
If I understood him correctly, during the bindings call this
morning, Mike Edwards indicated he disagrees with me, but I do
think that a key part of the test suite is to validate the
normative statements of the spec. If you can't write a test
for a particular normative statement, or a particular clause
of a normative statement, then perhaps the normative statement
shouldn't be normative. Or maybe you write the test, and
realize while thinking through all the cases that the
normative statement leaves possibilities out. Or maybe writing
t! he tests highlights an area where there's too much
implementation leeway, and more normative statements might be
in order. The best way to find any of that out is to write a
test suite. As I recall, all of these situations arose during
the development of the test suites.
Now that we have the tests, do we need the them to run against
two implementations? Personally, I'd be satisfied by one.
</eej>
There is a test suite that has been developed by the TC. When
you take your implementation and run it through the test
suite, you get an "answer". That's the answer you are looking
for.
The exit criteria require that there be 2 independent impls
that run the test suite, and get answer that says "yes" (or
all green ;-).
Did you intend to say "the proposed exit criteria"? As it
currently stands, the TC charter says nothing about mandating
an implementation run the test suite. Assuming exit criteria
were adopted that mandated the test suite be passed by two
independent runtimes, how would that be done? In other words,
who decides and verifies the tests were "all green"? Does the
entity responsible for a runtime need to make the test suite
and integration code (or binaries) required to plug the
runtime into the test suite publicly available? Or is it
enough for the entity to claim its runtime passes?
<eej>
I concur with Jim here. If an implementation professes to
implement the spec, but then fails the test suite, that's a
fairly obvious point that the marketplace can sort out.
Vendors might declare "we pass 90+%" for example. Leave it to
the end users to figure out whether that last X perc! ent
matters.
</eej>
jeff
Jim
On Feb 23, 2011, at 10:54 AM, Martin Chapman wrote:
Jim,
I think you are confusing the normative-ness of the test suite
with TC exit criteria.
You are correct that to claim conformance you don’t have to
pass the TC’s test suite, but that is different from the TC
saying that in order to progress the spec to Committee
Specification/OASIS Standard we want two implementations
passing the TC’s test suite.
Martin.
From: Jim Marino [mailto:jim.marino@gmail.com]
Sent: 22 February 2011 20:08
To: OASIS Assembly
Subject: Re: [sca-assembly] Concrete Exit Criteria for t! he
SCA Assembly TC - Proposal
Hi,
The first bullet makes sense to me. Concerning the second, my
understanding is the TC Test Suite is not normative and the
only thing required for conformance is for the entity
responsible for a particular runtime to declare that it
adheres to the normative portions of the Assembly
Specification. In other words, it should be valid for a
runtime to claim conformance that supplies its own test suite,
which may or may not be based on the TC one. This essentially
goes back to using the "court of public opinion" to judge
conformance claims as opposed to something akin to a TCK.
Jim
On Feb 18, 2011, at 6:19 AM, Mike Edwards wrote:
Folks,
I propose the following as the Exit Criteria to be adopted by
the SCA Ass! embly TC:
"The Concrete Exit Criteria for the SCA Assembly V1.1
specification are that:
o there shall be 2 independent SCA runtimes that are compliant
with all normative portions of the
specification as described in! Section 12.2 of the SCA
Assembly V1.1 specification
o the 2 independent runtimes shall pass the Test Suite for SCA
Assembly as described in the document
'TestCases for the SCA Assembly Model Version 1.1
Specification' "
I believe that this is sufficient to cover:
- the question of conformance to the spec
- the issues of portability and interoperability
since the test suite is a pretty thorough examination of
conformance to the spec (there are testcases for
all normative statements with testable & observable
consequences) and the test suite is also a test of
portability,
since the test suite has an extensive set of ar! tifacts that
are ported to each runtime and interoperability is
checked through the use of Web services being used between the
SCA runtime and a non-SCA (JAXWS) client.
Yours, Mike
Dr Mike Edwards
Mail Point 146, Hursley Park
<Mail Attachment.gif>
STSM
Winchester, Hants SO21 2JN
SCA & Services Standards
United Kingdom
Co-Chair OASIS SCA Assembly TC
IBM Software Group
Phone:
+44-1962 818014
Mobile:
+44-7802-467431 (274097)
e-mail:
mike_edwards@uk.ibm.com
Unless stat! ed otherwise above:
IBM United Kingdom Limited - Registered in England and Wales
with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth,
Hampshire PO6 3AU
--
Jeff Mischkinsky jeff.mischkinsky@oracle.com
Sr. Director, Oracle Fusion Middleware +1(650)506-1975
and Web Services Standards 500 Oracle Parkway,
M/S 2OP9
Oracle Redwood Shores, CA 94065
--
Jeff Mischkinsky
jeff.mischkinsky@oracle.com
Sr. Director, Oracle Fusion Middleware
+1(650)506-1975
and Web Services Standards 500 Oracle
Parkway, M/S 2OP9
Oracle Redwood Shores, CA 94065
---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC
that
generates this mail. Follow this link to all your TCs in OASIS
at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
|