OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

ebxml-iic-conform message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Subject: [ebxml-iic-conform] handling the"spec coverage"

Hi all:
Welcome to the "conformance" list...
Here is a proposal about the notion of specification coverage, that we discussed this morning,
and that is very specific to conformance testing (but not specific to any ebXML spec in particular):
By spec coverage, we mean the degree to which our tests adress the
conformance to the specification requirements, for each feature of the spec.
By Test Requirements, we mean a test-oriented rephrasing of the
spec features, something between the spec narrative and more detailed test case definition.
(This is in other words our "contract" with the spec owners, that we both have to agree on
before doing the real work of test cases design.)
1. For each feature of the target specification (e.g. each sub-section of its document),
we need to tell people how well these are covered by our set of  Test Requirement items.
Indeed, test reqs may not cover fully a spec feature due to the constraints associated
with testing, difficulty to reproduce all situations, etc.
So Mike and I started to do that with by associating this coverage measure with the
Test Reqs themselves - but the point is, that does not work well in some cases where
 there is not a 1-to-1 mapping between test Reqs and spec features.
(e.g. the same spec sub-section may be fully covered by three Test Reqs, yet each would
participate only partially into the verification of the features inside the sub-section. 
Or, the same Test Req might actually address two different spec features, differently.)
2. One solution is to  associate this coverage indicator with  the spec features themselves.
(Please see in the latest version of ebXMLTestFramework doc
that we posted on our site, the description in Section 4.3 "Specification Coverage".)
If we agree on this, that means:
- we should remove the "coverage" field in current Test Reqs document, and
instead provide a list of all sections/subsections numbers of the MS spec,
and for each, specify the degree of coverage we estimate our Test Reqs provide.
- so, when we communicate our Test Requirements to the spec owners (or spec originators)
for review,  we provide a document with two sections:
(a) Test Reqs (without coverage element),
(b) List of spec sections / sub-sections reference numbers with, for each ,
(b1) the list of Test Reqs that cover it, and  (b2) how well the section spec requirements
are covered by these (full, partial, none).
 Let us know if any problem with that.

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [Elist Home]

Powered by eList eXpress LLC