Re: [tag-discuss] Re: Test case metadata...

[Kyle Grucci <Kyle.Grucci@Sun.COM>]

Patrick Curran wrote:
> My responses are interspersed below, prefixed "PC>".
> 
> Durand, Jacques R. wrote:
>> some $0.02 considerations related to tagging " assertions" vs. writing 
>> " test assertions"  :
>>  (a plea in favor of more than just tagging...)
>>  
>> Regards,
>> -Jacques
>>
>>  
>> ------------------------------------------------------------------------
>> *From:* Patrick.Curran@Sun.COM [mailto:Patrick.Curran@Sun.COM]
>> *Sent:* Wednesday, December 06, 2006 2:58 PM
>> *To:* Durand, Jacques R.
>> *Cc:* tag-discuss@lists.oasis-open.org
>> *Subject:* Re: [tag-discuss] Re: Test case metadata...
>>
>> As I suggested in a recent post, I don't believe that it is always 
>> necessary to "reword" spec requirements in order to turn them into 
>> test assertions. If the spec is written in appropriately precise and 
>> normative language, it ought to be possible to treat text within the 
>> spec itself as assertions. 
>>  
>> in some cases, the spec may appear crystal-clear on what needs to be 
>> tested and gives enough hints on how to verify the requirement But 
>> shouldn't  we still recommend to extract a test assertion for the sake 
>> of consistency with the rest of TAs that are associated with this spec?
>> I am aware that this is more a methodology argument than a technical 
>> one, but my concern would be: often what is " crystal clear"  for spec 
>> writers, may not be so for test suite writers, and spec writers may be 
>> prone to be content with " assertions"  tags where a little more 
>> structured  " test assertion"  would really be more helpful.
>> For example WS-I  " profile definitions"  already  seemed to be stated 
>> quite clearly (each requirement is a separate, numbered item etc.), 
>> but when we wrote TAs for these in a methodical way (with a test 
>> approach in mind, e.g. stating more clearly which were the artifacts 
>> under test, etc.), we were surprised of how non-obvious that exercise 
>> was,  of the questioning and challenges this exercise caused to the 
>> original statements (helping increase quality) and ultimately of the 
>> difference it made for people writing ultimate test cases.
> PC> Interesting... I wasn't intending to do more than caution against 
> "busy work", in which slight changes in wording are introduced. (For 
> example, changing "the two sides of a square are equal in length" to 
> "the two sides of a square must be equal in length".) In many cases, the 
> text of the assertion can be identified within the specification itself, 
> and when this is so, it should be sufficient - for the purpose of 
> identifying assertions - to simply mark up the spec with "assertion 
> tags". If the text is tagged as an assertion, then it will obviously be 
> possible to automatically extract assertions from the spec into a 
> separate list.

I agree;  and if, for some reason, the text of an assertion is not able
to be identified within the prose of a specification, then the
specification must clearly state and tag the assertion in some other
part of the specification (i.e. an appendix).  For more of my thoughts 
on this topic, please see my recent blog on the subject...

http://weblogs.java.net/blog/kgrucci/archive/2006/09/compatibility_a_1.html

-Kyle


> 
> I think it ought to be possible to reconcile the two approaches: where 
> the "assertion text" can be identified within the spec itself, and where 
> it's necessary to re-word the text, or even to "derive" the text of the 
> assertion from the spec. In either case, we have "assertion text" and we 
> have a "spec reference". The rest is implementation :)
> 
> As for whether we want or need more "structure", that's something that 
> the TC will need to decide. If we agree that more structure (in addition 
> to a unique ID) is required, then obviously "just tagging" will not be 
> sufficient. In this case it will obviously be necessary to edit the 
> assertion list - whether this was extracted from the spec or composed 
> separately - to add the additional elements. Once again, however, I 
> would caution against incorporating too much "test metadata" into an 
> assertion. There really is a difference. However, this too is 
> implementation.
> 
>> A data-point: when the W3C QA Working Group was working on the 
>> Specification Guidelines [1] we wanted to make sure that this document 
>> (which was itself a specification) conformed to itself (that is, that 
>> this document conformed to the recommendations contained within it). 
>>  
>>  " eat your own dog food"  as some say... I guess we' ll have to, for 
>> our credibility...
> PC> I didn't think of this, but of course we also will need to do so.
>>
>> Since one of our recommendations was to use test assertions, we made 
>> the effort to create an assertion list. It turned out to be a largely 
>> cosmetic exercise, involving making slight but non-critical changes to 
>> text that was already present within the spec.
>>
>> This is why we defined a test assertion as "a measurable or testable 
>> statement of behavior, action, or condition [that is] contained within 
>> or derived from the specification's requirements". (In retrospect, I 
>> think "the specification's requirements" is redundant, and we could 
>> simply have said "the specification".) 
>>
>> [1] QA Framework: Specification Guidelines: 
>> http://www.w3.org/TR/qaframe-spec/
>>
>> Patrick Curran
>> Manager, Java SE Conformance Test Development, Sun Microsystems
>>
>> --------------------------

-- 
Kyle Grucci
CTS Architect
Java EE Compatibility
Sun Microsystems, Inc.

Email - kyle.grucci@sun.com




> --------------------------------------------------------------------- To 
> unsubscribe, e-mail: tag-discuss-unsubscribe@lists.oasis-open.org For 
> additional commands, e-mail: tag-discuss-help@lists.oasis-open.org