OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

sca-assembly message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [sca-assembly] Normative language for conformance testing requirement


This is a worthwhile debate.  And I believe that the outcome of our deliberations here will have a wider influence on the way that
other OASIS TCs proceed, given that OASIS has a desire to improve the way in which conformance to its specifications is
defined and measured.

The W3C has clearly gone much further down the Testing road than has OASIS, as can be seen from the following page:


...I believe that OASIS would benefit greatly from building similar pages of advice and guidance for all OASIS TCs.

I also note that W3C in that page covers many of the points that we have been debating in the Assembly TC.
In particular they describe:

- Conformance testing = checking that some implementation does conform to the normative requirements of the spec
- Publishing of test results (ie the results of running the test suite against some implementation)
- Branding / Certification program
- Maintenance of the test suite including bug fixes and versions of the test suite developed over time

I note in particular that they take the view that certification should be self certification with the open nature of the
test suite enabling anyone claiming conformance to have their claims put to the test (literally) in the most open way
possible.  This is something that I agree with.  "Public shaming" is the most effective means of enforcement
( I am amused by this idea since the politicians in the UK are undergoing just such a process at the moment with
regard to the expenses that they have been claiming over the past few years - the tool is simply the open publication
of their expenses claims - the pain that is being inflicted is a glory to behold )

So, let me make some of my views on these matters clear:

1) Making it a mandatory requirement (ie "MUST") for a conforming implementation to pass the test suite is of equal force
to making any other mandatory requirement in the spec itself.  I think that we should put a normative statement into the
Assembly spec that requires a conforming implementation to pass the tests in the Test Suite.  It is of no more or less
power than any other MUST statement in the spec.

In my opinion, the need to pass the test suite is no extra burden - if the spec is to have any force at all, then all of its
mandatory requirements MUST be met by a runtime.  The test suite gives a (public) way to show that a runtime has
credibility in making a claim to conform.  This does not prevent someone implementing the spec only partially -
but the test suite can be seen as a tool for publicly unmasking such behaviour and for undermining anyone making
false claims.

Concern has been expressed regarding potential conflicts between the Test Suite and the Spec.  Clearly this is possible.
However, I think that what is required here is along these lines:

   a) Make a statement in the spec that if there is a disagreement between test suite and the spec, the spec takes
precedence (even when it is the spec that is wrong ;-) - just go look at a couple of the recent issues raised against the
spec, which derive from work done on the test suite)

   b) Ensure a maintenance process for the test suite (just as there must be a maintenance process for the spec itself)
which provides for things like exclusion of tests known to be in error and for publication of updated versions of the test
suite.  The current charters of the TCs allow for this.

2) Concern has been expressed about the ability of the test suite to adapt to the nature of the runtimes under test.
I believe that we can structure the test suite in such a way to permit the test "harness" (in reality the client test loader/
test runner) to be extended or modified while being rigid about the SCA artifacts under test and the expected results.

So, we could envisage a test harness being implemented in a different language, or (potentially) using a different
means of communication from client to SCA runtime under test.

One thing that I think that we should insist on however, is that any modified or reimplemented harness should be made
publicly available. I'd prefer an approach where all such assets are made available from the OASIS web site.  My goal
in this is that ANYONE should be able to run the test suite against any SCA runtime that claims conformance.

3) The point is well made that the test suite is no panacea - merely passing the test suite DOES NOT guarantee
conformance to the spec.  It is hard to write testcases that cover all aspects of the spec (boy, do I know that now that I've
written a few testcases).  Plus the testcase writers are fallible humans (some more fallible than others, in my experience!!)
and leave holes.

So, it will always be possible that a runtime will have its conformance challenged even if it does pass the test suite.
However, I also view such an occurrence as a challenge to the test suite writers to prepare an enhanced version of the
test suite which does cover the area in question.

Yours,  Mike.

Strategist - Emerging Technologies, SCA & SDO.
Co Chair OASIS SCA Assembly TC.
IBM Hursley Park, Mail Point 146, Winchester, SO21 2JN, Great Britain.
Phone & FAX: +44-1962-818014    Mobile: +44-7802-467431  
Email:  mike_edwards@uk.ibm.com

From: David Booz <booz@us.ibm.com>
To: sca-assembly@lists.oasis-open.org
Date: 03/06/2009 20:25
Subject: Re: [sca-assembly] Normative language for conformance testing requirement

I told myself yesterday that I wasn't going to get into this debate....oh well.

I don't see what's wrong with "....MUST pass all the tests." This does not say that one MUST run the tests, only that the runtime MUST pass them if someone runs them against your implementation. I'm playing with words. The tests need to have some teeth otherwise there's no point in doing them. Let's also keep in mind some very important points:
1) We want to enable compliance under a self certifying model. There is no enforcement mechanism, nor do I think we want one. A runtime is compliant because the implementer of the runtime says it is. I believe this is a majority view in the TC.
2) The spec is authoritative over the test suite (and even the TA and TC specs), similar to how the spec is authoritative over the XML schemas we use, so the tests don't cause any irreconcilable problems.
3) The tests are open precisely so that anyone can run them. Non-compliant liars will be found out quite publicly by the industry or users of the software. Also, bugs in the test suite can be fixed quickly, in the open so that the test suite can be brought into line with the spec.

As a result, runtime implementation providers will run the tests just to make sure they are compliant. This will happen due to human nature, not due to some RFC keyword in the spec document.

Dave Booz
STSM, BPM and SCA Architecture
Co-Chair OASIS SCA-Policy TC and SCA-J TC
"Distributed objects first, then world hunger"
Poughkeepsie, NY (845)-435-6093 or 8-295-6093

Inactive hide details for Simon Nash ---06/03/2009 09:05:34 AM---Following on from the discussion on yesterday's call, the two Simon Nash ---06/03/2009 09:05:34 AM---Following on from the discussion on yesterday's call, the two forms of language suggested for a norm


Simon Nash <oasis@cjnash.com>


OASIS Assembly <sca-assembly@lists.oasis-open.org>


06/03/2009 09:05 AM


[sca-assembly] Normative language for conformance testing requirement

Following on from the discussion on yesterday's call, the two forms
of language suggested for a normative requirement for conformance
testing were along the lines of
1) A conforming implementation MUST be able to pass all the tests....
2) A conforming implementation MUST NOT fail any of the tests....

On the call it was stated that these are logically equivalent.

My preference is for the second form.  An advantage of this form
is that the statement is clear cut and easy to verify.  To show that
an implementation is non-conforming, it is only necessary to produce
a single test that fails to run with that implementation.

The language of the first form with its use of "be able to" seems
less clear and harder to verify.  The "be able to" language means that
there is no requirement to actually run any of the tests.  Without
doing this, how could it be shown that an implementation conforms to
the testing requirement (or not)?


To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail.  Follow this link to all your TCs in OASIS at:


Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]