Dear colleagues,
In the last regrep TC meeting I was assigned the ACTION to send this
proposal outline for how we can rapidly deliver the "ebXML Registry 3.0
Conformance Test Suite". So here it goes...
Please share your thoughts on this proposal.
Objective
To rapidly develop and approve an automated suite of tests that can be
used validate that an implementation conforms to the ebXML Registry 3.0
specifications.
Rationale
With 3.0 our specs have matured. An important next step as a sign of
this maturity is to have an approved "ebXML Registry 3.0 Conformance
Test Suite" that can be used to validate implementations as conforming
to our specifications. This will enhance interoperability among
different ebXML Registry 3.0 implementations. This is specially
important given the "organic" ability of ebXML Registries to federate
with any other spec conformant ebXML Registry.
Data Point: Approach Used by ebMS Conformance
The following is my understanding of the approach taken by ebMS TC in
collaboration with IIC TC:
- Write a document detailing the spec assertions that MUST be
tested (Test Assertions Document)
Assertion are statements to the effect "A registry MUST do ....."
- Write a set of automated tests where a test exists to test each
assertion in the Test Assertions Document.
-The tests were written for a specific test harness (NIST testbed used
by IIC)
-The tests were written in a language neutral neutral manner to only
use XML Message protocols via SOAP/HTTP
- Test are made available to implementors to work out conformance
bugs in their implementations
- Interop tests are organized between implementors to test
interoperability between various implementations
- An independent 3rd party (e.g. Drummond Group) offers a
Conformance Certification service for a fee to certify that an
implementation conforms to the specifications.
Proposed Approach for ebXML Registry Conformance
There exists an automated Test Suite for the freebXML Registry project
that has some 300 tests.
These tests have been developed over the last 4 years to perform
automated regression tests for the freebXML Registry
implementation.
These tests already cover a large number of assertions contained within
the ebXML Registry 3.0 specs
for nearly all the features.
These tests are already available as royalty free, open source
artifacts under a liberal "Apache Style" license.
The proposal is to seed our conformance test suite with the freebXML
Registry test suite as follows:
- Remove all tests that have implementation specific assumptions
- Take the remaining "pure" spec assertion tests (I expect ~100+
tests to qualify) and refactor / sanitize them so they no longer have
ties to freebXML Registry project
- Use these refactored / sanitized tests as our initial
conformance test suite
- Create a new open source project and commit the resulting test
suite in a source control repository there. They can now be developed
collaboratively in the open source. The project would be open for
contribution from any interested party including TC members and
implementors / vendors.
- Review, improve, finalize and approve the resulting test suite
- Seek independent 3rd party(s) to offer a Conformance
Certification service for a fee. Check with OASIS if they would be
interested in offering such a service themselves.
- Organize high visibility interop events later in this year as
conforming implementations become available.
Technical Details Regarding freebXML Registry Test Suite
- Written in Java using JAXR API to talk to the registry
- This does not require that registry implementations have to be
written in Java because all communication is via ebRS protocols using
SOAP/HTTP
- The registry implementation need not do anything more than
implement the specs.
- The implementation details of the test suite should not matter.
The suite should be treated as a black box leaving it to use whatever
means are effective in developing tests (e.g. Java, JAXR, JUnit).
- Uses the pervasive and robust Apache JUnit
[http://www.junit.org/index.htm] test harness used widely within the
industry
- Can be run fully automated
- Produces high quality tests results (see attached
sample-junit-report.png). Note that test class names would be changed
to be implementation neutral as part of the proposed refactoring /
sanitization.
Benefits of Proposed Approach
- We can rapidly deliver a comprehensive Conformance Test Suite
with relatively small investment
- The test suite can grow organically over time with periodic
review and approval of modified or new tests
- TC members and Implementors can freely collaborate on an ongoing
basis to care and feed the Conformance Test Suite
- Nightly/Weekly/periodic tests can be setup to automatically tests
all participating implementations over the web and generate reports
- Versions of the Conformance Test Suite can be managed in source
control like a professional software project
- The TC is the final approver of any given version of the
Conformance Test Suite
Hopefully you will find this idea as a "thinking outside the box"
approach to meeting an important need for the TC.
Please know that I am no special ax to grind here and no hidden agenda
to promote freebXML Registry. The refactoring / sanitization process
is meant to break all ties between the proposed test suite and the
freebXML Registry project. I am however, glad to help put sweat into
the proposal
and make it a reality quickly if the TC find it to be an attractive
option. So please keep an open mind when you consider this proposal.
I hope this quick brain dump covers most things we discussed in last TC
meeting on this topic. Kathryn, please let me know what if I am missing
something.
Most importantly, please share your thoughts on this proposal.
--
Regards,
Farrukh
|
