Re: [tag] Rationale/Best Practices merged - doc/odt files (v02)

[Kevin Looney <Kevin.T.Looney@Sun.COM>]

Hi Stephen,

     I've been reading and thinking about the Best Practice draft, and I wanted to pass on a few comments.

General
 
Somehow / Somewhere - it seems like we should draw attention that care should be taken in distinguishing a 'Test Assertion' from 'Meta Data' that can be derived from the TA. 

Example: At Sun, we often assign various Meta Data to assertions, such as whether an assertion is testable/not-testable.  It is possible to assert behaviors that for pragmatic reasons can not be fully tested, and it becomes important to know (for coverage analysis) 'of the assertions that can be tested, what is the percentage of assertions for which tests have been written'. This is an example of something that should be treated purely as Meta Data, and seems out-of-domain for a TA specification.


Guidelines for creating test assertions

[8] "Just using references to the specification may avoid adding interpretations but then care needs to 
be taken to make the reference sufficiently specific (such as by specifying the boundary of which 
part is referenced or to reference a sentence)."

I would suggest to drop the last few words about a "reference a sentence".  The problem is that identifying an assertion by referring to a sentence is problematic. It is probably sufficient to say "... such as by specifying text boundaries of the parts being referenced".

The problem that I find is quite often, the normative text for an assertion can be a subset of a sentence, a superset of many sentences, or combined and disjointed text fragments from multiple sentences.  Also, multiple assertions might even share the same text fragments.   These problems illustrate why a sentence boundary is a 'bad' granularity for specifying an assertion reference.

Referring to specifications

Perhaps you are already saying this in [3] ....

When describing Umbrella Specifications (specifications that inherit assertions from other reference specifications):

2 situations occur:
1.)  A specification and it's assertions are inherited completely and unchanged by the Umbrella
2.)  A specification is partially inherited, or inherited with modifications
(eg This Uber-Spec conforms to all portions of this sub-spec, except only single window operations are supported)

In situation 1, what you said seems true (eg.  delegate behavior and testing to the sub-spec singly)
In situation 2, there is a qualified meaning to inheritance, that somehow changes the meaning of the spec, or set of assertions within a spec.  In this situation, statements of behavior for the umbrella might somehow be different than the stated behavior within the sub-spec.

I would recommend acknowledging how spec references might be complex (eg modified inheritance), and recommend that assertion behaviors might be altered, and should be evaluated in the context of the inherited spec relationship.

[I'm not sure if I explained this clearly enough.   Our Korean colleagues alluded to this form of complex spec relationships at the last face-to-face]


Sequences of Test Assertions

"Make it the rule to keep each test assertion singular and atomic but recognize that 
there are exceptions when an assertion depends on another."

I think you are trying to say something stronger here:

1.) Always keep assertions singular and atomic
2.) There may be implicit ordering of assertions to accomplish [1].

I'm guessing this is where the optional "pre-requisites" field may be useful.  With
pre-requisites, ordering is explicit (rather than relying on ordering of textual 
descriptions within a spec)

Thanks, Regards,
Kevin L

stephen.green@systml.co.uk wrote:

As I've been constantly editing the wiki Rationale page I thought
I'd send a snapshot as a .doc file and .odt file at this point in
case it needs to be discussed on a coming call. (attached)

[Note: I've copied and pasted these from the wiki so the formatting
is as results from that. Maybe if we take these formats as standard
while editing is going on we will save a lot of reformatting work
as we go.]

I'm minded to drop maintenance of the Best Practice wiki page for
now until later to avoid constant duplication since the Rationale
now incorporates the existing Best Practice statements.

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail.  You may a link to this group and all your TCs in OASIS
at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php