OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

regrep message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Proposal for ebXML Registry 3.0 Conformance Test Suite

Dear colleagues,

In the last regrep TC meeting I was assigned the ACTION to send this proposal outline for how we can rapidly deliver the "ebXML Registry 3.0 Conformance Test Suite". So here it goes...

Please share your thoughts on this proposal.


To rapidly develop and approve an automated suite of tests that can be used validate that an implementation conforms to the ebXML Registry 3.0 specifications.


With 3.0 our specs have matured. An important next step as a sign of this maturity is to have an approved "ebXML Registry 3.0 Conformance Test Suite" that can be used to validate implementations as conforming to our specifications. This will enhance interoperability among different ebXML Registry 3.0 implementations. This is specially important given the "organic" ability of ebXML Registries to federate with any other spec conformant ebXML Registry.

Data Point: Approach Used by ebMS Conformance

The following is my understanding of the approach taken by ebMS TC in collaboration with IIC TC:
  1. Write a document detailing the spec assertions that MUST be tested (Test Assertions Document)
    Assertion are statements to the effect "A registry MUST do ....."
  2. Write a set of automated tests where a test exists to test each assertion in the Test Assertions Document.
    -The tests were written for a specific test harness (NIST testbed used by IIC)
    -The tests were written in a language neutral neutral manner to only use XML Message protocols via SOAP/HTTP
  3. Test are made available to implementors to work out conformance bugs in their implementations
  4. Interop tests are organized between implementors to test interoperability between various implementations
  5. An independent 3rd party (e.g. Drummond Group) offers a Conformance Certification service for a fee to certify that an implementation conforms to the specifications.

Proposed Approach for ebXML Registry Conformance

There exists an automated Test Suite for the freebXML Registry project that has some 300 tests.

These tests have been developed over the last 4 years to perform automated regression tests for the freebXML Registry

These tests already cover a large number of assertions contained within the ebXML Registry 3.0 specs
for nearly all the features.

These tests are already available as royalty free, open source artifacts under a liberal "Apache Style" license.

The proposal is to seed our conformance test suite with the freebXML Registry test suite as follows:
  • Remove all tests that have implementation specific assumptions
  • Take the remaining "pure" spec assertion tests (I expect ~100+ tests to qualify) and refactor / sanitize them so they no longer have ties to freebXML Registry project
  • Use these  refactored / sanitized tests as our initial conformance test suite
  • Create a new open source project and commit the resulting test suite in a source control repository there. They can now be developed collaboratively in the open source. The project would be open for contribution from any interested party including TC members and implementors / vendors.
  • Review, improve, finalize and approve the resulting test suite
  • Seek independent 3rd party(s) to offer a Conformance Certification service for a fee. Check with OASIS if they would be interested in offering such a service themselves.
  • Organize high visibility interop events later in this year as conforming implementations become available.

Technical Details Regarding freebXML Registry Test Suite

  • Written in Java using JAXR API to talk to the registry
    • This does not require that registry implementations have to be written in Java because all communication is via ebRS protocols using SOAP/HTTP
    • The registry implementation need not do anything more than implement the specs.
    • The implementation details of the test suite should not matter. The suite should be treated as a black box leaving it to use whatever means are effective in developing tests (e.g. Java, JAXR, JUnit).
  • Uses the pervasive and robust Apache JUnit [http://www.junit.org/index.htm] test harness used widely within the industry
  • Can be run fully automated
  • Produces high quality tests results (see attached sample-junit-report.png). Note that test class names would be changed to be implementation neutral as part of the proposed refactoring / sanitization.

Benefits of Proposed Approach

  • We can rapidly deliver a comprehensive Conformance Test Suite with relatively small investment
  • The test suite can grow organically over time with periodic review and approval of modified or new tests
  • TC members and Implementors can freely collaborate  on an ongoing basis  to care and feed the Conformance Test Suite
  • Nightly/Weekly/periodic tests can be setup to automatically tests all participating implementations over the web and generate reports
  • Versions of the Conformance Test Suite can be managed in source control like a professional software project
  • The TC is the final approver of any given version of the Conformance Test Suite
Hopefully you will find this idea as a "thinking outside the box" approach to meeting an important need for the TC.
Please know that I am no special ax to grind here and no hidden agenda to promote freebXML Registry. The refactoring / sanitization process
is meant to break all ties between the proposed test suite and the freebXML Registry project. I am however, glad to help put sweat into the proposal
and make it a reality quickly if the TC find it to be an attractive option. So please keep an open mind when you consider this proposal.

I hope this quick brain dump covers most things we discussed in last TC meeting on this topic. Kathryn, please let me know what if I am missing something.
Most importantly, please share your thoughts on this proposal.


PNG image

fn:Farrukh Najmi

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]