Re: [sca-assembly] Normative language for conformance testing requirement

[Mike Edwards <mike_edwards@uk.ibm.com>]

Folks,

This is a worthwhile debate.  And
I believe that the outcome of our deliberations here will have a wider
influence on the way that
other OASIS TCs proceed, given that
OASIS has a desire to improve the way in which conformance to its specifications
is 
defined and measured.

The W3C has clearly gone much further
down the Testing road than has OASIS, as can be seen from the following
page:

http://www.w3.org/QA/WG/2005/01/test-faq

...I believe that OASIS would benefit
greatly from building similar pages of advice and guidance for all OASIS
TCs.

I also note that W3C in that page covers
many of the points that we have been debating in the Assembly TC.
In particular they describe:

- Conformance testing = checking that
some implementation does conform to the normative requirements of the spec
- Publishing of test results (ie the
results of running the test suite against some implementation)
- Branding / Certification program
- Maintenance of the test suite including
bug fixes and versions of the test suite developed over time

I note in particular that they take
the view that certification should be self certification
with the open nature of the
test suite enabling anyone claiming
conformance to have their claims put to the test (literally) in the most
open way
possible.  This is something that
I agree with.  "Public shaming" is the most effective means
of enforcement
( I am amused by this idea since the
politicians in the UK are undergoing just such a process at the moment
with
regard to the expenses that they have
been claiming over the past few years - the tool is simply the open publication
of their expenses claims - the pain
that is being inflicted is a glory to behold )

So, let me make some of my views on
these matters clear:

1) Making it a mandatory requirement
(ie "MUST") for a conforming implementation to pass the test
suite is of equal force
to making any other mandatory requirement
in the spec itself.  I think that we should put a normative statement
into the
Assembly spec that requires a conforming
implementation to pass the tests in the Test Suite.  It is of no more
or less
power than any other MUST statement
in the spec.

In my opinion, the need to pass the
test suite is no extra burden - if the spec is to have any force at all,
then all of its
mandatory requirements MUST be met by
a runtime.  The test suite gives a (public) way to show that a runtime
has
credibility in making a claim to conform.
 This does not prevent someone implementing the spec only partially
- 
but the test suite can be seen as a
tool for publicly unmasking such behaviour and for undermining anyone making
false claims.

Concern has been expressed regarding
potential conflicts between the Test Suite and the Spec.  Clearly
this is possible.
However, I think that what is required
here is along these lines:

   a) Make a statement in
the spec that if there is a disagreement between test suite and the spec,
the spec takes
precedence (even when it is the spec
that is wrong ;-) - just go look at a couple of the recent issues raised
against the
spec, which derive from work done on
the test suite)

   b) Ensure a maintenance
process for the test suite (just as there must be a maintenance process
for the spec itself)
which provides for things like exclusion
of tests known to be in error and for publication of updated versions of
the test
suite.  The current charters of
the TCs allow for this.

2) Concern has been expressed about
the ability of the test suite to adapt to the nature of the runtimes under
test.
I believe that we can structure the
test suite in such a way to permit the test "harness" (in reality
the client test loader/
test runner) to be extended or modified
while being rigid about the SCA artifacts under test and the expected results.

So, we could envisage a test harness
being implemented in a different language, or (potentially) using a different
means of communication from client to
SCA runtime under test.

One thing that I think that we should
insist on however, is that any modified or reimplemented harness should
be made
publicly available. I'd prefer an approach
where all such assets are made available from the OASIS web site.  My
goal
in this is that ANYONE should be able
to run the test suite against any SCA runtime that claims conformance.


3) The point is well made that the test
suite is no panacea - merely passing the test suite DOES NOT guarantee

conformance to the spec.  It is
hard to write testcases that cover all aspects of the spec (boy, do I know
that now that I've
written a few testcases).  Plus
the testcase writers are fallible humans (some more fallible than others,
in my experience!!)
and leave holes.

So, it will always be possible that
a runtime will have its conformance challenged even if it does pass the
test suite.
However, I also view such an occurrence
as a challenge to the test suite writers to prepare an enhanced version
of the
test suite which does cover the area
in question.


Yours,  Mike.

Strategist - Emerging Technologies, SCA & SDO.
Co Chair OASIS SCA Assembly TC.
IBM Hursley Park, Mail Point 146, Winchester, SO21 2JN, Great Britain.
Phone & FAX: +44-1962-818014    Mobile: +44-7802-467431  
Email:  mike_edwards@uk.ibm.com

From:	David Booz <booz@us.ibm.com>
To:	sca-assembly@lists.oasis-open.org
Date:	03/06/2009 20:25
Subject:	Re: [sca-assembly] Normative language for conformance testing requirement

---

I told myself yesterday that I wasn't going to get into
this debate....oh well.

I don't see what's wrong with "....MUST pass all the tests."
This does not say that one MUST run the tests, only that the runtime MUST
pass them if someone runs them against your implementation. I'm playing
with words. The tests need to have some teeth otherwise there's no point
in doing them. Let's also keep in mind some very important points:
1) We want to enable compliance under a self certifying model. There is
no enforcement mechanism, nor do I think we want one. A runtime is compliant
because the implementer of the runtime says it is. I believe this is a
majority view in the TC.
2) The spec is authoritative over the test suite (and even the TA and TC
specs), similar to how the spec is authoritative over the XML schemas we
use, so the tests don't cause any irreconcilable problems.
3) The tests are open precisely so that anyone can run them. Non-compliant
liars will be found out quite publicly by the industry or users of the
software. Also, bugs in the test suite can be fixed quickly, in the open
so that the test suite can be brought into line with the spec.

As a result, runtime implementation providers will run the tests just to
make sure they are compliant. This will happen due to human nature, not
due to some RFC keyword in the spec document.


Dave Booz
STSM, BPM and SCA Architecture
Co-Chair OASIS SCA-Policy TC and SCA-J TC
"Distributed objects first, then world hunger"
Poughkeepsie, NY (845)-435-6093 or 8-295-6093
e-mail:booz@us.ibm.com

Simon
Nash ---06/03/2009 09:05:34 AM---Following on from the discussion on yesterday's
call, the two forms of language suggested for a norm

From:	Simon Nash <oasis@cjnash.com>
To:	OASIS Assembly <sca-assembly@lists.oasis-open.org>
Date:	06/03/2009 09:05 AM
Subject:	[sca-assembly] Normative language for conformance testing requirement

---

Following on from the discussion on yesterday's call, the two forms
of language suggested for a normative requirement for conformance
testing were along the lines of
 1) A conforming implementation MUST be able to pass all the tests....
 2) A conforming implementation MUST NOT fail any of the tests....

On the call it was stated that these are logically equivalent.

My preference is for the second form.  An advantage of this form
is that the statement is clear cut and easy to verify.  To show that
an implementation is non-conforming, it is only necessary to produce
a single test that fails to run with that implementation.

The language of the first form with its use of "be able to" seems
less clear and harder to verify.  The "be able to" language
means that
there is no requirement to actually run any of the tests.  Without
doing this, how could it be shown that an implementation conforms to
the testing requirement (or not)?

  Simon



---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

---

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number 741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU