Sorry for the
weird reformatting of previous mail... resending
this, hoping it will do better:
---------------------------
I hope all of you are still on board, after
the winter break... which leads me to wish you a happy New Year (even if this
is still premature to some of us).
The discussion list has been up for almost 2
months, and it has clearly revealed interest in the topic. As you know, this
list is primarily intended for testing the waters for TC creation - and is not
a forum for doing the real tech work: more time for this in a future TC
assuming we could agree on a TC platform. This becomes now the main thing to
focus on - and hopefully we should all agree this month on a charter draft.
Here is a new iteration on a charter draft - please
review and comment. For the sake of finding consensus, in case you disagree
with a statement in the draft, please indicate:
- if it is a show-stopper for your participation
(can't tolerate it)
- if you can live with it though not happily ever
after.
Of course, feel free to comment on this even if
you/your company is not sure yet of its committment to a TC.
-Jacques
------------- Charter Draft:
Name and abbreviation
Test Assertions Guideline TC, or TAG TC
(?)
Purpose
The design of Test Assertions (TAs) associated with
a specification or standard has the following recognized benefits: (i) it
improves the quality of this specification during its design, and (ii) it
reduces the lead time necessary to create a test suite for this specification.
A test assertion (TA), also sometimes defined
as test specification, is understood in this charter with the following
general meaning: it describes the expected output or behavior for an
implementation under some specific operation conditions, in a way that can be
measured or tested. Each test assertion is (as far as possible) an
independent, complete, statement of how to verify that an implementation
satisfies a requirement in the target specification. Test assertions are
generally different from test cases, which are more detailed and contingent to
a concrete test framework: TAs are the basis to write test cases, and relate
the latter to the narrative of the target specification. (NOTE: this candidate definition in a charter
is only intended to reflect a general level of agreement about what is it we
work on. It may be refined and improved in a final speciification.)
The general objective served by this TC is to
facilitate the creation and usage of test assertions by any group involved in
designing a specification or standard of which software implementations are
expected to be developed, with a focus on OASIS technical
committees.
The first step in achieving this is to establish a
common and reusable model, metadata, methodology and representation for TAs.
This is aligned with the intent of the former OASIS
Conformance TC, although the focus in the current initiative is not on the
various aspects of quality and conformance, but on a specific one, namely test
assertions.
This TC will submit its deliverables to the OASIS
Technical Advisory Board (TAB) for recommendation to the OASIS Board. The TC
would accept feedback and recommendations from the TAB and OASIS Board, so
that its output can be considered by the OASIS Board for inclusion in a future
revision of the OASIS general TC process, aiming at improving the quality and
adoption of OASIS output.
Scope
The scope of activity for this TC must be within
the following topics:
- A model for designing Test Assertions (TA model),
that is independent from any particular target specification or standard, but
that may recognize different types of test assertions, and may accommodate
these in a specific way.
- Test assertions may be for verifying conformance
of an implementation to a specification, or interoperability between
implementations of the same specification.
- The TA model may address any useful relationship
identified between TAs, such as pre-requisites or pre-conditions. It may
include support for grouping several TAs - or grouping entities -, but will
not pretend to fully model such entities as conformance profiles,
specification modules or implementation roles.
- A methodology to make use of this model. This may
include examples derived from applying the above TA model to particular
specifications or standards.
- An XML representation for TAs and – if
appropriate – for their grouping entities. Additional notations supporting the
modeling of TAs (e.g. UML) are also within scope. The intent of the XML
representation is left at the discretion of the TC. It could be intended as an
exchange format for editing tools, or as a source for rendering/publishing, or
as an input for a test tool, or a combination of these.
- Within scope is a characterization of the test
environment or test harness assumed by the test assertions, stating the
expected properties and mode of operation required by the verification of the
TAs. Such characterization may be seen as some of the requirements for a real
test harness intended to process test cases based on these TAs .
- Within scope is the selection and/or refinement
of definitions of concepts expected to be related to TAs, even if not directly
targeted by the modeling and methodology work of this TC. Such concepts may
include: test case, conformance profile or level, test environment / harness,
test execution.
- It is in the scope of this TC work to examine
current practices in other OASIS TCs or other organizations which have already
written test assertions for their specifications.
- Within scope are efforts to liaise with other
organizations than OASIS, and cooperation with external contributors and
groups will be considered, as well as joint submissions.
Documents that will be considered as contributing
material to this TC, i.e. that will deserve prime attention from the TC
assuming their IP status is compatible with the IPR mode of the TC:
- the Test Assertion Guideline draft, (originally
initiated within OASIS TAB, 2004-2005)
- the Conformance guideline (OASIS)
- QA Framework: Specification Guidelines
(W3C, November 2004) http://www.w3.org/TR/qaframe-spec/
- Variability in Specifications, WG note (W3C,
2005)
- Test Assertion Documents for WS-I profiles
(2003-2004).
- Others?
Explicitly Out-of-scope:
- Defining test case metadata or representation.
(Several developments are occurring in this area.)
- Modeling or methodology of dimensions of
variability such as conformance profiles / levels / modules / discretionary
items. Substantial prior work exists in these areas. However the TC may use
this prior work as a foundation and may also accommodate these constructs when
designing an XML mark-up for TAs.
Deliverables
The TC will produce the following deliverables, in
this order: (tentative date?)
(a) Test Assertions Guideline document, including
definitions, abstract model / structure not excluding the use of a modeling
notation (e.g. UML), methodology and examples showing how to extract TAs from
a target specification.
(b) A representation of TAs in XML, that refers to
and supports the TA model of (a).
IPR Mode
- Royalty-Free on limited terms
Audience
- members of any OASIS tech committees
- members already involved in defining
specifications/standards/tests in other organizations.
Language
- English
References