OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

regrep message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [regrep] 7/29/2003: Conformance Efforts and ebXML IIC

  Please see my comments below.


----- Original Message ----- 

> Mike,
> The updated format looks very good. A few questions follow:
> -What does the Requirement Level column mean? Is it saying whether the
> assertion is part of a required feature or an optional feature?

[MIKE] - Yes, Requirement level refers to the specifications use of the
MUST, SHOULD, MAY keywords.
and equates them with  REQUIRED, RECOMMENDED and OPTIONAL.  This is
important at test
reporting time, becase an implementation should not test against a
particular requirement if it has not implemented
that feature.

>  If so maybe a more obvious column header is desirable (e.g. "Require
> with value true/false).

[MIKE] - You are correct, in that a "true/false" would be more correct (
"optional" ), however using RECOMMENDED offers a finer level of detail at
test reporting time
regarding an implementation's test result against a requirement.

In our testing framework, our conformance test driver recognizes RECOMMENDED
as equivalent and "false".
> -What process, responsible parties, milestones and deliverables do you
> you forsee going forward?

[MIKE] - I see an ebXML RS conformance test suite specification document,
contains a list of conformance test requirements ( like the example table
we've been
discussing ),a  conformance test suite ( an XML document, with test cases
that validate to
the ebXML Test Suite schema as defined in the IIC Test Framework
Specification document).

Also included in the conformance test suite specification is supporting test
material ( XML files
that that will be used in LifeCycleManager and QueryManager transactions,
like those put together
by Len Gallagher in the initial ebRS conformance test suite ).

> -Is the end product an "ebXML Registry Conformance Test Specification
> (CTS)" document?

[MIKE] - Yes.

> -Who produces the initial set of assertions that would be the basis for
> the CTS?
[MIKE] - Because of IIC limited resources, we'd like each TC to contribute
toward the
collection of test assertions.  The IIC could guide and contribute, but
collection can be a tedious
process. I would suggest a "staged" approach, in which test assertions are
compiled for a selected
"profile" or "level" ( even though the RS specification does not define
them ).  Spending a year compiling
assertions would not necessarily serve the ebXML community, whereas building
upon a "core" set of
asssertions over time would.

>  -Who produces the final CTS?

[MIKE] - That would be the responsibility of the IIC.

> -Who produces automated test suite for verifying Conformance?

 The IIC is normally tasked with this effort.  Although again, due to our
own resource limitations,
we would ask for help from the registry TC in contributing tests to the

> -What are/should be some realistic milestones for various deliverables?

[MIKE] - Based upon our own experience in compiling about 200 conformance
test assertions for ebMS, this
can be a time-consuming task.  However, if you take a "divide and conquer"
approach to RS testing, you may be
able to make this easier.  For instance, choosing a single binding (e.g.
SOAP) to write test requirements, selecting LifeCycleManager
functionality as a higher priority for test requirements than QueryManager
functionality...  could get the ball rollling sooner.

Also, focusing on special areas of concern regarding the RS specification
clarification can also reap benefits sooner.
Also, setting aside testing requirement that may not yet exist in any
implementation  can allow focus on actual
real-world  issues.

I started "marking up" the ebRS 2.5 specification, pages 1-40, ( sectoins
#1 - #6) just to begin to get
a feel for what it would take to "highlight" the specification and identify
conformance test requirements.
I found that portion of the specification to be straightforward regarding
identifying conformance test requirements.

The LifeCycleManager section (section #7) appears to be straight-forward and
easy to identify testing requirements, grouped into fundamental
sub-categores of "parameter", "response",  "exception" and  "audit trail"
types of testing

The QueryManager (section #8) could be a very rigorous exercise in test
requirement annotation, specifically since each
"node" in a FilterQuery tree could be a candidate for testing... I would
suggest a "generalized" approach to FilterQuery or SQLQuery
testing, in which "duplicate" query tree node testing is eliminated to
simplify both test requirement writing, as well as actual test generation.

Section #9, Content Management Services, is an optional service, that can be
tested by creating/testing an IIC defined content
management service to the registry, then verifying its conformant
functionality.  This could be done by creation and publishing of a
"dummy" validation service, I believe.  Because this is an optional feature,
test requirements for this service could be described.. but
actual implementation of test cases "could" be deferred if time/resources
are an issue.

Section #10, Event Notification Service, is also an optional service, that
is well documented and fairly simple to generate test requirements.
Acutal test case scenarios should be easy to construct.  Again, because this
is an optional feature, it is a candidate for test requirement
generation, but actual test cases COULD be deferred for a later time.

Section #11, Cooperating Registries Support is a complex scenario requiring
federated registries for conformance testing.  The requirements
are well stated, but implementation of test cases will be complex and
time-intensive.  The specification does not eplicitly say whether this is an
optional feature, although I assume that it is.

Section #12,  Security an be a large exercise, particularly with all of the
possible XMLDSIG options.  But this is a "required" feature, and can perhaps
be described with a small initial set of test assertions and test cases, and
a limited testing scope.

So from a deliverables standpoint, if you are doing the entire ebRS
specification  I would suggest ( working in parallel if resources are
available ) :

1) "Marking up" the ebRS document to identify testing requirements - 4 weeks
2) "Prioritizing" which marked-up sections of the specification you wish to
write conformance test requirements for - done in parallel with the
3) Create  a "minimal" ebRS Conformance Test Requirements table ( using
syntax described in our example ) - 2 months
( possible if  you omit/minimize writing some requirements, such as
rigorous FilterQuery/SQLQuery test requirements,
Content Management Service, Event Notification Service and Federated
Registry test requirements )
4) To create a "full blown" conformance ebRS Test Requirements Document - 6
5) To create a "minimal" ebRS Conformance Test Suite 4 months
6) To create a "full blown" ebRS Conformance Test Suite 12 months
7) To create a Conformance Test Suite Specification - (integrating test
requirements, test cases and supporting documentation) - 1 month

This is my estimate, based upon IIC work with ebXML Messaging Services.
Currently, I am the only person working on test cases for ebMS in the IIC.
Others in the TC are editors and reviewers of our documents.  This is not
likely to change, as others do not have the time to dedicate toward such
an intensive task.  So I can assist the regrep TC in developoing test
requirements and test cases, but it will take assistance to speed up the

In the IIC, we are currently finishing up the Conformance Test Suite
Specification  document for ebMS, and will have a final version posted on
OASIS IIC web site within a month.  Currently, version 0.9 of the ebMS
Conformance Test Suite, along with the ebXML Test Framework document are
avaiable at:


Test cases developed by the IIC are being incorporated by Drake Certivo in
their e-business conformance testing tool.  Also, the K-WareSoft of Korea
has incorporated
our test cases into their ebXML MS conformance testing tool.  I believe that
tests generated for ebRS would fit well with test cases already developed by
and used in the open source registry project.

I will be finished with ebXML Messaging Services Conformance Test Suite by
the end of September.  That will be a good time for me to work closer with
theregrep TC in developing test requirements and test cases.  However, work
certainly begin before the ( for example, annotating the ebRS specification,
and beginning a "cut/paste" operation to populate the Conformance Test
Requirements Table we've been discussing.

Also, the IIC TC must vote upon which specifications it will work with.
BPSS is also
a hot item for testing now, so the IIC has to prioritize its resources.
However, working
in "parallel" may be possible.

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]