Sorry
for the weird reformatting of previous mail... resending
this, hoping it will do better:
---------------------------
I hope all of you are still on
board, after the winter break... which leads me to wish you a happy New
Year (even if this is still premature to some of us).
The discussion list has been up for
almost 2 months, and it has clearly revealed interest in the topic. As
you know, this list is primarily intended for testing the waters for TC
creation - and is not a forum for doing the real tech work: more time
for this in a future TC assuming we could agree on a TC platform. This
becomes now the main thing to focus on - and hopefully we should all
agree this month on a charter draft.
Here is a new iteration on a charter
draft - please review and comment. For the sake of finding consensus,
in case you disagree with a statement in the draft, please indicate:
- if it is a show-stopper for your
participation (can't tolerate it)
- if you can live with it though not
happily ever after.
Of course, feel free to comment on
this even if you/your company is not sure yet of its committment to a
TC.
-Jacques
------------- Charter Draft:
Name and abbreviation
Test Assertions Guideline TC, or TAG
TC (?)
Purpose
The design of Test Assertions (TAs)
associated with a specification or standard has the following
recognized benefits: (i) it improves the quality of this specification
during its design, and (ii) it reduces the lead time necessary to
create a test suite for this specification.
A test assertion (TA), also
sometimes defined as test specification, is understood in this charter
with the following general meaning: it describes the expected output or
behavior for an implementation under some specific operation
conditions, in a way that can be measured or tested. Each test
assertion is (as far as possible) an independent, complete, statement
of how to verify that an implementation satisfies a requirement in the
target specification. Test assertions are generally different from test
cases, which are more detailed and contingent to a concrete test
framework: TAs are the basis to write test cases, and relate the latter
to the narrative of the target specification. (NOTE: this candidate definition in
a charter is only intended to reflect a general level of agreement
about what is it we work on. It may be refined and improved in a final
speciification.)
The general objective served by this
TC is to facilitate the creation and usage of test assertions by any
group involved in designing a specification or standard of which
software implementations are expected to be developed, with a focus on
OASIS technical committees.
The first step in achieving this is to
establish a common and reusable model, metadata, methodology and
representation for TAs.
This is aligned with the intent of the
former OASIS Conformance TC, although the focus in the current
initiative is not on the various aspects of quality and conformance,
but on a specific one, namely test assertions.
This TC will submit its deliverables
to the OASIS Technical Advisory Board (TAB) for recommendation to the
OASIS Board. The TC would accept feedback and recommendations from the
TAB and OASIS Board, so that its output can be considered by the OASIS
Board for inclusion in a future revision of the OASIS general TC
process, aiming at improving the quality and adoption of OASIS output.
Scope
The scope of activity for this TC must
be within the following topics:
- A model for designing Test
Assertions (TA model), that is independent from any particular target
specification or standard, but that may recognize different types of
test assertions, and may accommodate these in a specific way.
- Test assertions may be for verifying
conformance of an implementation to a specification, or
interoperability between implementations of the same specification.
- The TA model may address any useful
relationship identified between TAs, such as pre-requisites or
pre-conditions. It may include support for grouping several TAs - or
grouping entities -, but will not pretend to fully model such entities
as conformance profiles, specification modules or implementation roles.
- A methodology to make use of this
model. This may include examples derived from applying the above TA
model to particular specifications or standards.
- An XML representation for TAs and –
if appropriate – for their grouping entities. Additional notations
supporting the modeling of TAs (e.g. UML) are also within scope. The
intent of the XML representation is left at the discretion of the TC.
It could be intended as an exchange format for editing tools, or as a
source for rendering/publishing, or as an input for a test tool, or a
combination of these.
- Within scope is a characterization
of the test environment or test harness assumed by the test assertions,
stating the expected properties and mode of operation required by the
verification of the TAs. Such characterization may be seen as some of
the requirements for a real test harness intended to process test cases
based on these TAs .
- Within scope is the selection and/or
refinement of definitions of concepts expected to be related to TAs,
even if not directly targeted by the modeling and methodology work of
this TC. Such concepts may include: test case, conformance profile or
level, test environment / harness, test execution.
- It is in the scope of this TC work
to examine current practices in other OASIS TCs or other organizations
which have already written test assertions for their specifications.
- Within scope are efforts to liaise
with other organizations than OASIS, and cooperation with external
contributors and groups will be considered, as well as joint
submissions.
Documents that will be considered as
contributing material to this TC, i.e. that will deserve prime
attention from the TC assuming their IP status is compatible with the
IPR mode of the TC:
- the Test Assertion Guideline draft,
(originally initiated within OASIS TAB, 2004-2005)
- the Conformance guideline (OASIS)
- QA Framework: Specification
Guidelines (W3C, November 2004) http://www.w3.org/TR/qaframe-spec/
- Variability in Specifications, WG
note (W3C, 2005)
- Test Assertion Documents for WS-I
profiles (2003-2004).
- Others?
Explicitly Out-of-scope:
- Defining test case metadata or
representation. (Several developments are occurring in this area.)
- Modeling or methodology of
dimensions of variability such as conformance profiles / levels /
modules / discretionary items. Substantial prior work exists in these
areas. However the TC may use this prior work as a foundation and may
also accommodate these constructs when designing an XML mark-up for TAs.
Deliverables
The TC will produce the following
deliverables, in this order: (tentative date?)
(a) Test Assertions Guideline
document, including definitions, abstract model / structure not
excluding the use of a modeling notation (e.g. UML), methodology and
examples showing how to extract TAs from a target specification.
(b) A representation of TAs in XML,
that refers to and supports the TA model of (a).
IPR Mode
- Royalty-Free on limited terms
Audience
- members of any OASIS tech committees
- members already involved in defining
specifications/standards/tests in other organizations.
Language
- English
References