Re: [sca-assembly] Normative language for conformance testing requirement

[David Booz <booz@us.ibm.com>]

I told myself yesterday that I wasn't going to get into this debate....oh well.

I don't see what's wrong with "....MUST pass all the tests." This does not say that one MUST run the tests, only that the runtime MUST pass them if someone runs them against your implementation. I'm playing with words. The tests need to have some teeth otherwise there's no point in doing them. Let's also keep in mind some very important points:
1) We want to enable compliance under a self certifying model. There is no enforcement mechanism, nor do I think we want one. A runtime is compliant because the implementer of the runtime says it is. I believe this is a majority view in the TC.
2) The spec is authoritative over the test suite (and even the TA and TC specs), similar to how the spec is authoritative over the XML schemas we use, so the tests don't cause any irreconcilable problems.
3) The tests are open precisely so that anyone can run them. Non-compliant liars will be found out quite publicly by the industry or users of the software. Also, bugs in the test suite can be fixed quickly, in the open so that the test suite can be brought into line with the spec.

As a result, runtime implementation providers will run the tests just to make sure they are compliant. This will happen due to human nature, not due to some RFC keyword in the spec document.


Dave Booz
STSM, BPM and SCA Architecture
Co-Chair OASIS SCA-Policy TC and SCA-J TC
"Distributed objects first, then world hunger"
Poughkeepsie, NY (845)-435-6093 or 8-295-6093
e-mail:booz@us.ibm.com

Simon Nash ---06/03/2009 09:05:34 AM---Following on from the discussion on yesterday's call, the two forms of language suggested for a norm

From:	Simon Nash <oasis@cjnash.com>
To:	OASIS Assembly <sca-assembly@lists.oasis-open.org>
Date:	06/03/2009 09:05 AM
Subject:	[sca-assembly] Normative language for conformance testing requirement

---

Following on from the discussion on yesterday's call, the two forms
of language suggested for a normative requirement for conformance
testing were along the lines of
  1) A conforming implementation MUST be able to pass all the tests....
  2) A conforming implementation MUST NOT fail any of the tests....

On the call it was stated that these are logically equivalent.

My preference is for the second form.  An advantage of this form
is that the statement is clear cut and easy to verify.  To show that
an implementation is non-conforming, it is only necessary to produce
a single test that fails to run with that implementation.

The language of the first form with its use of "be able to" seems
less clear and harder to verify.  The "be able to" language means that
there is no requirement to actually run any of the tests.  Without
doing this, how could it be shown that an implementation conforms to
the testing requirement (or not)?

   Simon



---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php