Minutes
Bob sends regrets, he's in a F2F
Opening
Roll call 11/19 = 57% - Quorate but not 2/3
Agenda accepted as posted
Approval of Minutes
Resolution: Minutes of 2011-03-08 approved w/o
Action Items
Action: id=2010-09-22-8 status=pending owner="EricJ" produce new proposal for ASSEMBLY-227
Action: id=2011-01-04-2 status=pending Edwards to write a new proposal for the resolution of Assembly-246 along the lines contained
in wsra
No updates or changes to AI status
Administrivia
Exit Criteria
The TC needs to adopt concrete exit criteria.
Discussion:
Lots of debate on the Assembly mailing list (too many emails to list!
MikeE:
Some confusion with subject of conformance - which is different from Exit Criteria
...if people are unhappy with Conformance then a new issue(s) should be raised
MikeE:
At the moment need two independent runtimes...
...but what is independent and how to cover optional items (SHOULD, MAY, etc)
MikeE:
Second issue is Test Suite and if required for Exit Criteria
MartinC:
Separate consideration for Spec vs. Test Suite
...they're separate deliverables and maybe we need different Exit Criteria
...Not sure if Test Suite should be an OASIS Standard
BryanA:
Actualy 5 deliverable docs - may need exit criteria for all and need to decide if they would be "standards"
JeffM:
Does proposal for Exit Criteria require all normative statements in spec to be tested?
DannyV:
Doesn't include optional statements
JimM:
Not sure it's necessary for optional
MikeE:
Charter does not say anything about a requirement for testing
MartinC:
Normative is anything with RFC2119
<Anish Karmarkar (Oracle)>
Here is the charter language regd exit criteria:
<Anish Karmarkar (Oracle)>
The TC shall define concrete exit criteria that include at least two independent offerings that implement and are compliant
with the all normative portions of specifications and demonstrate interoperability and portability as appropriate. Note that
these are minimums and that the TC is free to set more stringent criteria.
<Bob (himself for Hitachi)>
I suggest the following grades:
<Bob (himself for Hitachi)>
1. mandatory must be tested with two implementations
<Bob (himself for Hitachi)>
2. Optional - two imples, first class
<Bob (himself for Hitachi)>
3. optional with one impl mark as such
<Bob (himself for Hitachi)>
4. optional with no impl - remove from spec
SanjayP:
Purpose of Exit Criteria is to show spec is implementable
...if that is the case then two implementations should be enough regadless of passing Test Suite
...but if they do then this can be taken as proof of implemetation - not just a claim
<Anish Karmarkar (Oracle)>
Mike's proposal 1:
<Anish Karmarkar (Oracle)>
"The Concrete Exit Criteria for the SCA Assembly V1.1 specification are that:
o there shall be 2 independent SCA runtimes that are compliant with all normative portions of the
specification as described in Section 12.2 of the SCA Assembly V1.1 specification
o the 2 independent runtimes shall pass the Test Suite for SCA Assembly as described in the document
'TestCases for the SCA Assembly Model Version 1.1 Specification' "
<Anish Karmarkar (Oracle)>
Mike's proposal 2:
<Anish Karmarkar (Oracle)>
"The Concrete Exit Criteria for the SCA Assembly V1.1 specification are that:
o there shall be at least 2 independent SCA runtimes that are compliant with each normative portion of the
specification as described in Section 12.2 of the SCA Assembly V1.1 specification
<Anish Karmarkar (Oracle)>
o the 2 independent runtimes must provide a demonstration of their conformance to the SCA Assembly specification
and it is recommended but not required that this is done by the runtime passing the Test Suite for SCA Assembly as described
in the
document 'TestCases for the SCA Assembly Model Version 1.1 Specification'
EricJ:
Need to be careful - E.G. Test cases have only ONE Test Suite and may need TWO for Exit
MikeE:
Think its fair for TC to ask for some evidence of conformance - I.E. passing Test Suite
<Mike Edwards>
Mike's proposal email:
Motion: Accept 1st proposal above as Exit Criteria for Specification m=AnishK s:BryanA
BryanA:
Would this allow different Exit Criteria for other document?
MikeE:
Yes - this is for SCA v1.1 Specification only
EricJ:
Does this require passing the optional normative parts? If so what coverage is needed?
AnishK:
Does cover optional, but...
...would interpet as only needing coverage of optional parts by not necessarily by both (all) runtimes
DannyV:
Would argue that both (all) implementations would have to cover all optional parts
MikeE:
Implementations are only required to "prove to TC" - which would be agreed/accepted by vote
<MartinC>
we don't have a normalized test report
<MartinC>
but some test output would be nice
<Bob (himself for Hitachi)>
for each normarive statement there exists at least two implementations that operate with evidence satisfactory to the TC
DannyV:
The problem then is that "concrete evidence" is subjective (to TC interpretation)
DannyV:
Wants to ammend to say "evidence for passing Test Suite is subject to TC decision"
<MartinC>
o the 2 independent runtimes shall pass the Test Suite for SCA Assembly as described in the document
'TestCases for the SCA Assembly Model Version 1.1 Specification' To the satisfaction of the TC "
Motion: Amend second point of first proposal as posted in Chat above m=DannyV s=EricJ
JeffM:
Intent of test was same as W3C - if you pass the test then by definition implementation is conformant
...I will not vote for anything that does not pss the test suite (green lightss for all tests)
<Anish Karmarkar (Oracle)>
my attempt at translating what is on the white board in the room: for each normative statement, there exists at least two
implementations that implement it
<Anish Karmarkar (Oracle)>
[much nicer and precise using set notation]
BryanA:
Does this mean we are not dictating format of evidence but still require that evidence
MikeE:
That is the intent
JeffM:
The amendment does not state that
(Discussion of what it means to "Pass Test Suite" if there's no standard form of proof)
<MartinC>
test output in any form which reports pass/fail against each test...how hard is that?
JimM:
Just need a sheet of paper or and "independently runnable" vereification
<Anish Karmarkar (Oracle)>
o the 2 independent runtimes shall pass the Test Suite for SCA Assembly as described in the document
'TestCases for the SCA Assembly Model Version 1.1 Specification'. The TC shall use the results (expected output) of the relevant
tests, submitted by the implementer (in any form), to verify that the runtime passes the tests.
JeffM:
I want to see output of running tests - to satisfy myself that results have not been "fixed"
BobF:
Looking for reasonable evidence that test have been passed - not proof.
AOB
DannyV, EricJ, DianeJ, SanjayP
<MartinC>
diane and peter for roll
<MartinC>
(danny, bob, sanjay recorded)
<Mike Edwards>
thanks Eric
Schreiber diagnostics output
[Delete this section before publishing the minutes]
final validation: Date not specified, the date '2011-03-08' was assumed
final validation: Title not specified, default title 'Oasis SCA-Assembly Teleconference...' was assumed
final validation: Chair not specified, default chair was assumed
statistics: Schreiber found 101 input lines
edits: Schreiber found the following text-edit commands:
edits: Line 6: s/1. Intro/agenda: 1. Intro/
edits: Line 92: s/there shall be 2 independent SCA runtimes that are compliant with all normative portions/there shall be
at least 2 independent SCA runtimes that are compliant with each normative portion/
edits: Line 123: s/iut/out
edits: Line 143: s/DianneJ/DianeJ/
edits: Line 147: s/OPening/Opening/
edits: Line 148: s/19 57%/19 = 57%/
edit-substitute: command on line 6 succeeded, changed line 1 from '1. Intro' to 'agenda: 1. Intro'
command-scribe: Line 2: Since the line number is less than or equal to 20 we will interpret this as a scribename command,
note that the scribe command is deprecated
command-scribe: Line 2: Scribe 'Eric Wells' is recognized by use of the nick 'EricW'
command-scribe: Line 2: EricW's nick 'EricW' has been selected
edit-delete: Line 6 was deleted
edit-substitute: command on line 147 succeeded, changed line 7 from 'OPening' to 'Opening'
edit-substitute: command on line 148 succeeded, changed line 8 from '19 57%' to '19 = 57%'
citation-detection-scribed: Line 21: Check for possible unrecognized nick 'The TC needs to adopt concrete exit criteria.
Discussion'
edit-substitute: command on line 92 succeeded, changed line 61 from 'there shall be 2 independent SCA runtimes that are compliant
with all normative portions' to 'there shall be at least 2 independent SCA runtimes that are compliant with each normative
portion'
edit-delete: Line 92 was deleted
edit-substitute: command on line 123 succeeded, changed line 121 from 'iut' to 'out'
edit-delete: Line 123 was deleted
edit-substitute: command on line 143 succeeded, changed line 139 from 'DianneJ' to 'DianeJ'
edit-delete: Line 143 was deleted
edit-delete: Line 147 was deleted
edit-delete: Line 148 was deleted
system: Transformer: SAXON 9.2.1.2
[End of Schreiber diagnostic output]