My responses are interspersed below, prefixed "PC>".
Durand, Jacques R. wrote:
some $0.02 considerations related to
tagging " assertions" vs. writing " test assertions" :
(a plea in favor of more than just
tagging...)
Regards,
-Jacques
As I suggested in a recent post, I don't believe that it is
always necessary to "reword" spec requirements in order to turn them
into test assertions. If the spec is written in appropriately precise
and normative language, it ought to be possible to treat text within
the spec itself as assertions.
in some cases, the spec may appear crystal-clear on what
needs to be tested and gives enough hints on how to verify the
requirement But shouldn't we still recommend to extract a test
assertion for the sake of consistency with the rest of TAs that are
associated with this spec?
I am aware that this is more a methodology argument than
a technical one, but my concern would be: often what is " crystal
clear" for spec writers, may not be so for test suite writers, and
spec writers may be prone to be content with " assertions" tags where
a little more structured " test assertion" would really be more
helpful.
For example WS-I " profile definitions" already seemed
to be stated quite clearly (each requirement is a separate, numbered
item etc.), but when we wrote TAs for these in a methodical way (with a
test approach in mind, e.g. stating more clearly which were the
artifacts under test, etc.), we were surprised of how non-obvious that
exercise was, of the questioning and challenges this exercise caused
to the original statements (helping increase quality) and ultimately of
the difference it made for people writing ultimate test cases.
PC> Interesting... I wasn't intending to do more than caution
against "busy work", in which slight changes in wording are introduced.
(For example, changing "the two sides of a square are equal in length"
to "the two sides of a square must be equal in length".) In many cases,
the text of the assertion can be identified within the specification
itself, and when this is so, it should be sufficient - for the purpose
of identifying assertions - to simply mark up the spec with "assertion
tags". If the text is tagged as an assertion, then it will obviously be
possible to automatically extract assertions from the spec into a
separate list.
I think it ought to be possible to reconcile the two approaches: where
the "assertion text" can be identified within the spec itself, and
where it's necessary to re-word the text, or even to "derive" the text
of the assertion from the spec. In either case, we have "assertion
text" and we have a "spec reference". The rest is implementation :)
As for whether we want or need more "structure", that's something that
the TC will need to decide. If we agree that more structure (in
addition to a unique ID) is required, then obviously "just tagging"
will not be sufficient. In this case it will obviously be necessary to
edit the assertion list - whether this was extracted from the spec or
composed separately - to add the additional elements. Once again,
however, I would caution against incorporating too much "test metadata"
into an assertion. There really is a difference. However, this too is
implementation.
A data-point: when the W3C QA Working Group was working on the
Specification Guidelines [1] we wanted to make sure that this document
(which was itself a specification) conformed to itself (that is, that
this document conformed to the recommendations contained within it).
" eat your own dog food" as some say... I guess we' ll
have to, for our credibility...
PC> I didn't think of this, but of course we also will need to do so.
Since one of our recommendations was to use test assertions, we made
the effort to create an assertion list. It turned out to be a largely
cosmetic exercise, involving making slight but non-critical changes to
text that was already present within the spec.
This is why we defined a test assertion as "a measurable or testable
statement of behavior, action, or condition [that is] contained within
or derived from the specification's requirements". (In retrospect, I
think "the specification's requirements" is redundant, and we could
simply have said "the specification".)
[1] QA Framework: Specification Guidelines: http://www.w3.org/TR/qaframe-spec/
Patrick Curran
Manager, Java SE Conformance Test Development, Sun Microsystems
--------------------------
|