I think
there could be a little bit of chicken
and egg problem here.
I am
wondering how Patrick C. sees the
to-be-created guideline different from the one in his reference [1].
Could you
highlight your thoughts. There are also a number of article out there
relating
to QA practice (looking from the references in [1])
IMHO, I
think if this TC were to create a guideline,
it should be more grounded and more concise than [1] (focuses on TA).
And one way
I can think of is to ground (and scope) it based on a data
structure/model. There
for I think that this TC should definitely propose a mark up. I think
the data
model should be drafted first b/c it will serve as a central point of
discussion and the driver for the guideline. I think 80% vote for (c)
but we
should not have to publish them in sequence. Why not release both at
the same
time. In my experience, a guideline results in much more impact with
some sort
of template (it makes much more sense to read a guideline when you have
something to fill in). Ex. The ebXML UMM has been of little use without
a
template, similarly I think the UML would not have been popular without
a
diagramming scheme.
- serm
[1]
http://www.w3.org/TR/qaframe-spec/
Patrick:
Never too late to jump in :-) I think you made a good case for a first
deliverable just being a guideline - but a broadly applicable and
authoritative
one.
- even if the first step is a TA methodology guideline without
ambition beyond that (e.g. mark-up) you don' t seem to rule out a more
formal TA representation in a next step. :
Now, trying to nail down
what is an agreeable TC charter for
people on this list, which one(s) of the following options would you
support:
(a) a
TC charter
that only targets a TA guideline as unique deliverable, and explicitly
prohibit
any work on a formal representation (e.g. XML mark-up). Any intent to
do other
deliverables beyond this guideline, after being done with the
guideline, would
require a re-chartering of the TC.
(b) a
TC charter
that targets a TA guideline as first deliverable (mandatory), and also
allows
the TC - should the TC consider it useful - to produce another
deliverable
(optional) containing a TA representation or mark-up. The TC may also
decide
(vote) at that time on the purpose of this mark-up, as it may impact
its
design. E.g. may decide that requirements for this mark-up are just for
publishing concerns.
(c)
same as (b),
but the 2nd deliverable - the TA mark-up or whatever representation -
is not an
option, I.e. it is understood the TC is supposed to produce it.
In all 3 above, the TC must focus on a guideline in a first phase, that
excludes a mark-up or formal TA representation. It may work on a
mark-up in
second phase, and this deliverable will be distinct (a different
document) from
the guideline deliverable. Only, it is easier to do so when going from
(a) to
(c):
-
in (a), Re-chartering the TC to extend its scope is a bit more
paperwork, and
requires 2/3 majority to pass. Also needs be done with same IPR mode.
-
in (b), TC may decide either to NOT do the mark-up (simple majority
vote), or
to do it, after producing the guideline deliverable.
-
in (c), TC is committed to work on a mark-up after being done with 1st
deliverable. (I know that some folks have mentioned they consider this
an
important output)
NOTE:
everyone welcome to " vote" on this too... or add alternative
if you really can' t live with any above option...
-----Original Message-----
From: Patrick.Curran@Sun.COM [mailto:Patrick.Curran@Sun.COM]
Sent: Tuesday, December 05, 2006 6:24 PM
To: tag-discuss@lists.oasis-open.org
Subject: [tag-discuss] Keep It Simple
Hello, everyone:
I've lurked for too long - sorry about that. I've been following the
discussion
with great interest, and it's time for me to jump in. I'll start by
offering
some general opinions and suggestions, and then hopefully I'll be able
to
respond in more detail to some individual messages.
In the interests of making progress in a reasonable period of time, and
of
defining something that is likely to be adopted by a significant number
of spec
writers and test developers, I propose that we Keep It Simple.
Our primary goal should be to encourage and promote the use of test
assertions
by specification writers. The W3C QA Working Group briefly discussed
the value
of test assertions in the QA Framework:
Specification Guidelines [1], but much remains to be done to explain
and
justify their use. Publication of a "guidelines" document, even
without any definition of structure or markup would be extremely
valuable. (My
test developers spend a considerable amount of time identifying
assertions when
this could have been done by the spec authors.)
While it is theoretically possible to write specifications in formal
languages,
and to write (or even automatically derive) detailed, structured test
assertions from which test cases can be automatically generated, most
spec
authors would not use these approaches. (With some notable exceptions -
for
example the Java Language and Virtual Machine specs - most specs in the
Java
world are written in English by people whose primary job is not the
creation of
formal specifications, but rather of creating software which they
document as
best they can.)
I therefore suggest that we declare these topics out of scope, at least
for the
first phase of our work.
A topic of lively discussion within the W3C Working Group was whether
it's
possible to classify "testable sentences" within a specification as
test assertions, or whether assertions must necessarily be "derived"
from the specification and formulated separately in some kind of
supplement to
the spec. I believe that we must allow for both possibilities. (Here in
the
Java world we usually identify text within the specs as assertions, but
I know
that other approach is preferred
elsewhere.)
The QAWG eventually agreed that both approaches were valid, and
defined a
test assertion as "a measurable or testable statement of behavior,
action,
or condition [that is] contained within or derived from the
specification's
requirements and provides a normative foundation from which one or more
test
cases can be built."
In addition to promoting the identification/creation of test
assertions, a
simple XHTML markup would be very helpful. For example, when an
assertion is
contained within the specification we might mark it up like
this:
<assertion id="123">The horizontal and vertical sides of a
square are equal in length.</assertion> This means that you can
flip the
square by 90 degrees and it will look just the same.
Alternatively, if the text of an assertion is not found directly within
the
specification but instead is derived from it and recorded separately,
it might
be marked up like this (note that we now need to provide a pointer into
the
specification to indicate where the assertion was derived from):
<assertion id="123"
specref="http://www.foo.com/docs/geometry/square.html#sides">
The horizontal and vertical sides of a square must be equal in
length.</assertion>
If the TC achieved no more than this (encouraging spec authors to
create test
assertions and defining a simple markup) I would be very pleased.
A well-defined list of assertions is key to much of the test
development
process, and a great deal can be built from this simple foundation -
for
example, coverage maps and reports, "marked-up" specifications that
link assertions to test cases, etc. (See "How Many Tests are Enough?"
in the Test Development FAQ [2] for my thoughts on coverage
measurement.) For
example, here at Sun we have a tool that we use to identify and mark
test
assertions within Javadoc, to associate assertions with test cases, and
to
generate reports that tell us how extensively we've covered various
areas of
the specifications. Industry standards in this area would allow us to
create a
more widely useful version of this tool.
Of course, once we've defined a markup for assertions there will always
be a
great temptation to add more structure and additional elements.
What version(s) of the spec does this assertion apply to? Is it
associated with
a Profile or another "Dimension of Variability" [3]? How important is
it that it be tested? Is it tested, and if so how thoroughly?
Some additional structure is probably appropriate for test assertions,
but we
might want to consider starting with a "Basic Profile" that defines
the minimum necessary elements and expand on this in a later phase to
create
an"Advanced Profile".
However, we should beware of conflating test (case) metadata with test
assertions. They don't necessarily belong together. (An assertion
states what
is to be tested, whereas test metadata typically helps to define how to
test.)
Given a simple assertion ID as suggested above, test metadata stored
separately
from the specification or assertion list could easily be integrated
with it. As
an author of the Test Metadata document [4] David referred to in
another
thread, I encourage those who wish to work on test metadata to do so
through
the W3C QA Interest Group rather than here.
In conclusion, I suggest limiting our phase one activities to a
"guidelines" document that explains the value and promotes the use of
test assertions, and the definition of a simple markup that allows them
to be
identified within text specifications and also in separate "assertion
lists". As the QA Framework: Specification Guidelines [1]
explains:
"Test assertions facilitate the development of consistent, complete
specifications and promote the early development of test cases.
Developing or extracting test assertions helps uncover inconsistencies,
ambiguities, gaps, and non-testable statements in the specification. It
can
provide early feedback to the editors regarding areas that need
attention. An
added benefit is that the assertions are usable as input to test
development
efforts."
For the record, I do believe we should form a TC and I don't believe
that we
need a workshop before we can do so, though I think there would be
great
benefit to some kind of meeting once we've made a start.
I'm looking forward to participating, and will encourage other members
of my
team to do so too.
Patrick Curran
Manager, Java SE Conformance Test Development, Sun Microsystems
========
[1] QA Framework: Specification Guidelines:
http://www.w3.org/TR/qaframe-spec/
[2] Test Development FAQ: http://www.w3.org/QA/WG/2005/01/test-faq
[3] Variability in Specifications: http://www.w3.org/TR/spec-variability/
[4] Test Metadata: http://www.w3.org/TR/test-metadata/
---------------------------------------------------------------------
To unsubscribe, e-mail: tag-discuss-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: tag-discuss-help@lists.oasis-open.org