OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

tag-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [tag-discuss] Re: Test case metadata...

My responses are interspersed below, prefixed "PC>".

Durand, Jacques R. wrote:
some $0.02 considerations related to tagging " assertions" vs. writing " test assertions"  :
 (a plea in favor of more than just tagging...)


From: Patrick.Curran@Sun.COM [mailto:Patrick.Curran@Sun.COM]
Sent: Wednesday, December 06, 2006 2:58 PM
To: Durand, Jacques R.
Cc: tag-discuss@lists.oasis-open.org
Subject: Re: [tag-discuss] Re: Test case metadata...

As I suggested in a recent post, I don't believe that it is always necessary to "reword" spec requirements in order to turn them into test assertions. If the spec is written in appropriately precise and normative language, it ought to be possible to treat text within the spec itself as assertions. 
in some cases, the spec may appear crystal-clear on what needs to be tested and gives enough hints on how to verify the requirement But shouldn't  we still recommend to extract a test assertion for the sake of consistency with the rest of TAs that are associated with this spec?
I am aware that this is more a methodology argument than a technical one, but my concern would be: often what is " crystal clear"  for spec writers, may not be so for test suite writers, and spec writers may be prone to be content with " assertions"  tags where a little more structured  " test assertion"  would really be more helpful.
For example WS-I  " profile definitions"  already  seemed to be stated quite clearly (each requirement is a separate, numbered item etc.), but when we wrote TAs for these in a methodical way (with a test approach in mind, e.g. stating more clearly which were the artifacts under test, etc.), we were surprised of how non-obvious that exercise was,  of the questioning and challenges this exercise caused to the original statements (helping increase quality) and ultimately of the difference it made for people writing ultimate test cases.
PC> Interesting... I wasn't intending to do more than caution against "busy work", in which slight changes in wording are introduced. (For example, changing "the two sides of a square are equal in length" to "the two sides of a square must be equal in length".) In many cases, the text of the assertion can be identified within the specification itself, and when this is so, it should be sufficient - for the purpose of identifying assertions - to simply mark up the spec with "assertion tags". If the text is tagged as an assertion, then it will obviously be possible to automatically extract assertions from the spec into a separate list.

I think it ought to be possible to reconcile the two approaches: where the "assertion text" can be identified within the spec itself, and where it's necessary to re-word the text, or even to "derive" the text of the assertion from the spec. In either case, we have "assertion text" and we have a "spec reference". The rest is implementation :)

As for whether we want or need more "structure", that's something that the TC will need to decide. If we agree that more structure (in addition to a unique ID) is required, then obviously "just tagging" will not be sufficient. In this case it will obviously be necessary to edit the assertion list - whether this was extracted from the spec or composed separately - to add the additional elements. Once again, however, I would caution against incorporating too much "test metadata" into an assertion. There really is a difference. However, this too is implementation.

A data-point: when the W3C QA Working Group was working on the Specification Guidelines [1] we wanted to make sure that this document (which was itself a specification) conformed to itself (that is, that this document conformed to the recommendations contained within it). 
 " eat your own dog food"  as some say... I guess we' ll have to, for our credibility...
PC> I didn't think of this, but of course we also will need to do so.

Since one of our recommendations was to use test assertions, we made the effort to create an assertion list. It turned out to be a largely cosmetic exercise, involving making slight but non-critical changes to text that was already present within the spec.

This is why we defined a test assertion as "a measurable or testable statement of behavior, action, or condition [that is] contained within or derived from the specification's requirements". (In retrospect, I think "the specification's requirements" is redundant, and we could simply have said "the specification".) 

[1] QA Framework: Specification Guidelines: http://www.w3.org/TR/qaframe-spec/

Patrick Curran
Manager, Java SE Conformance Test Development, Sun Microsystems


[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]