[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: [tag-discuss] Keep It Simple: some options
Patrick:
Never too late to jump
in :-) I think you made a good case for a first deliverable just being a
guideline - but a broadly applicable and authoritative one.
- even if the
first step is a TA methodology guideline without ambition beyond
that (e.g. mark-up) you don' t seem to rule out a more formal TA
representation in a next step. :
(a) a TC charter that only targets a TA guideline as unique
deliverable, and explicitly prohibit any work on a formal representation (e.g.
XML mark-up). Any intent to do other deliverables beyond this guideline, after
being done with the guideline, would require a re-chartering of the
TC.
(b) a TC charter that targets a TA guideline as first deliverable
(mandatory), and also allows the TC - should the TC consider it useful - to
produce another deliverable (optional) containing a TA representation or
mark-up. The TC may also decide (vote) at that time on the purpose of this
mark-up, as it may impact its design. E.g. may decide that requirements for this
mark-up are just for publishing concerns.
(c) same as (b), but the 2nd
deliverable - the TA mark-up or whatever representation - is not an option, I.e.
it is understood the TC is supposed to produce it.
In all 3
above, the TC must focus on a guideline in a first phase, that excludes a
mark-up or formal TA representation. It may work on a mark-up in second phase,
and this deliverable will be distinct (a different document) from the guideline
deliverable. Only, it is easier to do so when going from (a) to (c):
- in (a), Re-chartering the TC to extend its scope is a bit more paperwork, and requires 2/3 majority to pass. Also needs be done with same IPR mode.
- in (b), TC may decide either to NOT do the mark-up (simple majority vote), or to do it, after producing the guideline deliverable.
- in (c), TC is committed to work on a mark-up after being done with 1st deliverable. (I know that some folks have mentioned they consider this an important output)
NOTE: everyone welcome to " vote" on this too... or add alternative if you really can' t live with any above option...
-----Original Message-----
From: Patrick.Curran@Sun.COM [mailto:Patrick.Curran@Sun.COM]
Sent:
Tuesday, December 05, 2006 6:24 PM
To:
tag-discuss@lists.oasis-open.org
Subject: [tag-discuss] Keep It
Simple
Hello, everyone:
I've lurked for too long - sorry about
that. I've been following the discussion with great interest, and it's time for
me to jump in. I'll start by offering some general opinions and suggestions, and
then hopefully I'll be able to respond in more detail to some individual
messages.
In the interests of making progress in a reasonable period of
time, and of defining something that is likely to be adopted by a significant
number of spec writers and test developers, I propose that we Keep It
Simple.
Our primary goal should be to encourage and promote the use of
test assertions by specification writers. The W3C QA Working Group briefly
discussed the value of test assertions in the QA Framework:
Specification
Guidelines [1], but much remains to be done to explain and justify their use.
Publication of a "guidelines" document, even without any definition of structure
or markup would be extremely valuable. (My test developers spend a considerable
amount of time identifying assertions when this could have been done by the spec
authors.)
While it is theoretically possible to write specifications in
formal languages, and to write (or even automatically derive) detailed,
structured test assertions from which test cases can be automatically generated,
most spec authors would not use these approaches. (With some notable exceptions
- for example the Java Language and Virtual Machine specs - most specs in the
Java world are written in English by people whose primary job is not the
creation of formal specifications, but rather of creating software which they
document as best they can.)
I therefore suggest that we declare these
topics out of scope, at least for the first phase of our work.
A topic of
lively discussion within the W3C Working Group was whether it's possible to
classify "testable sentences" within a specification as test assertions, or
whether assertions must necessarily be "derived"
from the specification and
formulated separately in some kind of supplement to the spec. I believe that we
must allow for both possibilities. (Here in the Java world we usually identify
text within the specs as assertions, but I know that other approach is
preferred
elsewhere.)
The QAWG eventually agreed that both
approaches were valid, and defined a test assertion as "a measurable or testable
statement of behavior, action, or condition [that is] contained within or
derived from the specification's requirements and provides a normative
foundation from which one or more test cases can be built."
In addition
to promoting the identification/creation of test assertions, a simple XHTML
markup would be very helpful. For example, when an assertion is contained within
the specification we might mark it up like
this:
<assertion
id="123">The horizontal and vertical sides of a square are equal in
length.</assertion> This means that you can flip the square by 90 degrees
and it will look just the same.
Alternatively, if the text of an
assertion is not found directly within the specification but instead is derived
from it and recorded separately, it might be marked up like this (note that we
now need to provide a pointer into the specification to indicate where the
assertion was derived from):
<assertion id="123"
specref="http://www.foo.com/docs/geometry/square.html#sides">
The
horizontal and vertical sides of a square must be equal in
length.</assertion>
If the TC achieved no more than this
(encouraging spec authors to create test assertions and defining a simple
markup) I would be very pleased.
A well-defined list of assertions is key
to much of the test development process, and a great deal can be built from this
simple foundation - for example, coverage maps and reports, "marked-up"
specifications that link assertions to test cases, etc. (See "How Many Tests are
Enough?" in the Test Development FAQ [2] for my thoughts on coverage
measurement.) For example, here at Sun we have a tool that we use to identify
and mark test assertions within Javadoc, to associate assertions with test
cases, and to generate reports that tell us how extensively we've covered
various areas of the specifications. Industry standards in this area would allow
us to create a more widely useful version of this tool.
Of course, once
we've defined a markup for assertions there will always be a great temptation to
add more structure and additional elements.
What version(s) of the spec does
this assertion apply to? Is it associated with a Profile or another "Dimension
of Variability" [3]? How important is it that it be tested? Is it tested, and if
so how thoroughly?
Some additional structure is probably appropriate for
test assertions, but we might want to consider starting with a "Basic Profile"
that defines the minimum necessary elements and expand on this in a later phase
to create an"Advanced Profile".
However, we should beware of conflating
test (case) metadata with test assertions. They don't necessarily belong
together. (An assertion states what is to be tested, whereas test metadata
typically helps to define how to test.) Given a simple assertion ID as suggested
above, test metadata stored separately from the specification or assertion list
could easily be integrated with it. As an author of the Test Metadata document
[4] David referred to in another thread, I encourage those who wish to work on
test metadata to do so through the W3C QA Interest Group rather than
here.
In conclusion, I suggest limiting our phase one activities to a
"guidelines" document that explains the value and promotes the use of test
assertions, and the definition of a simple markup that allows them to be
identified within text specifications and also in separate "assertion lists". As
the QA Framework: Specification Guidelines [1]
explains:
"Test
assertions facilitate the development of consistent, complete specifications and
promote the early development of test cases.
Developing or extracting test
assertions helps uncover inconsistencies, ambiguities, gaps, and non-testable
statements in the specification. It can provide early feedback to the editors
regarding areas that need attention. An added benefit is that the assertions are
usable as input to test development efforts."
For the record, I do
believe we should form a TC and I don't believe that we need a workshop before
we can do so, though I think there would be great benefit to some kind of
meeting once we've made a start.
I'm looking forward to participating,
and will encourage other members of my team to do so too.
Patrick
Curran
Manager, Java SE Conformance Test Development, Sun
Microsystems
========
[1] QA Framework: Specification
Guidelines:
http://www.w3.org/TR/qaframe-spec/
[2]
Test Development FAQ: http://www.w3.org/QA/WG/2005/01/test-faq
[3]
Variability in Specifications: http://www.w3.org/TR/spec-variability/
[4]
Test Metadata: http://www.w3.org/TR/test-metadata/
---------------------------------------------------------------------
To
unsubscribe, e-mail: tag-discuss-unsubscribe@lists.oasis-open.org
For
additional commands, e-mail:
tag-discuss-help@lists.oasis-open.org
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]