Review of Section 3.5 (TA Grouping)

[Kevin Looney <Kevin.T.Looney@Sun.COM>]

Hi Everyone,

   I had a chance to do my AI, and review section 3.5 from the current Wiki.

   My comments are below - and I will update the wiki with these soon too.

Thanks, Regards,
Kevin L

3.5 Test Assertion Grouping

[Kevin: Perhaps there should be some 'introductory' text before jumping into this -

eg.  "The Grouping of assertions allows a person analyzing a specification a mechanism for annotating a "similar" set of assertions carrying some common characteristic.  Typically, grouping can be accomplished either explicitly (via lists, as in 3.5.1), or implicitly (via tags, as in 3.5.2)"]

3.5.1 Lists (Dimensions of Variability)

A common requirement is to be able to group a set of test assertions in a tightly controlled way such that there is certainty that the group contains a specified set of test assertions and no more than that set.

[Kevin: I'm not sure that there is any more certainty that an assertion is explicitly in some group - because it explicitly belongs in a list, or implicitly in some group (because of it's tags).  I suppose there could be some error where a given assertion has conflicting tags - but likewise, an assertion might incorrectly appear in two lists with opposite meanings.

From what I can see, there really isn't too much difference in the functionality of Lists vs. Tags. 

I also don't think "open vs. closed" is a meaningful way to differentiate the two types of lists -
the difference is better described as 'whether the logic used to describe the list is "implicit" or "explicit"'.

A list created from tags is one where the logic is "explicitly stated",  yielding a series of TAs when the logic is applied to all TAs in the spec.

eg   list =  (TAs with <Tag A> && <Tag B> && !<Tag C>)
             =    [ TA001, TA002, ....., TA00N]

As opposed to an  list of (explicitly enumerated) TAs, in which the logic of inclusion is implicit.  As such, the logic is "Frozen" into some abstract definition of the list.

eg.    list = (all assertions describing 'Size' requirements)
             =    [ TA001, TA002, ....., TA00N]

Perhaps "frozen logic" is what is meant by 'closed' grouping.]

This can be termed 'closed' grouping (as distinct from 'open' grouping facilitated by tagging - see Section 3.5.2). It is a particularly important method of grouping when test assertions are used for defining conformance requirements.

This designation of a set of test assertions may be accompanied by a header construct which allows the declaration of any variables whose scope is defined as comprising this same set (see section 3.8). A special but ubiquitous case of this is with the grouping of a whole set of test assertions into a test assertion document. This may be a crucial part of defining test assertions which are designated to a conformance clause for a specification or conformance profile (as a key aspect of defining dimensions of variability [VAR]).

[Kevin: The paragraph above describes a 'test assertion document'  - Is this a concept in the glossary?  Should it be?

I think there probably is an "<output>" of the specification analysis process, and it's format includes what we are specifying in this document. 

Some choices, we could describe this "output" as:

    Test assertion document (vague)
    Assertion List (this is a term used in Sun, but it also seems vague)
    Specification Analysis (this term is more descriptive of the process, and not the output)
     ???  (other)

Perhaps it is not our job to formalize what this "output" is, and it is our job to describe that Test Assertions are the 'key documentation of specification analysis' that must belong in an "<output>" document.

]

In some cases the tests may be allowed to inherit shared elements such as prerequisites or variables from the header of the listed group. However there is a need for careful integrity considerations in case a test assertion is listed in more than one list. This is to avoid contradictions between inherited elements.

An important aspect of test assertion listing is version control (see Section 3.7).

[Kevin: We could expand on version control a bit - version control could be applied to various levels of granularity.
Here, I think the implication is that a "assertion grouping list" may change independently from the assertions in a spec.


Or perhaps version control is only applied at the topmost level of a "<output>" (specification analysis document, as described above).

In any case, the statement above about the relevance of version control seems indescript.]

3.5.2 Tags (Test Assertion Metadata)

A technique for defining more 'open' groups of test assertions is to use "tags". The groups creating by assigning tags to test assertions can be described as open because the set of test assertions having a particular tag may change as the same tag can be assigned to a newly created test assertion or by changing an existing test assertion to add or remove the same tag.

[Kevin: As I observed above, the two types of lists differ in how implicit/explicit the logic is that is used when creating the list.

eg.    Closed list:
               [ TA001, TA002, ....., TA00N]
     (these are individually enumerated TAs, where there is some implicit logic used to determine inclusion

        Open list:
              list = TAs annotated with (<Tag A> && <Tag B> && !<Tag C>)
                   =    [ TA001, TA002, ....., TA00N]
      (in this case, the list is derived from an explicit logical expression applied to all TAs in a spec.)]

A tag is an optional test assertion element that is given a specific name and in some cases (but not always) a value. It allows tests assertions to be subsequently grouped according to the tags or tag combinations, effectively filtering the test assertions into subsets of the whole. Several such filters can be applied to the same set of assertions and any given assertion can appear in more than one grouping.

This is not a suitable method when it is necessary to know with certainty the complete set of test assertions which will ever have a particular tag or set of tags: In such cases the better practise is to list the test assertions in the group (see Section 3.5.1).

[Kevin:   I'm not sure I agree with this statement. 

          The 'certainty' of the inclusion of a TA within a grouping seems to comes from appying some logic to the spec, producing the group.  It doesn't matter so much whether that logic is applied explicitly (and possibly dynamically), or through some predescribed enumeration.

The things that describe certainty (or lack of certainty) of inclusion within a group is:
    a.) whether the TA changes
    b.) whether the logic describing the grouping changes.

]