Re: [tag] AI: Grouping

["Stephen Green" <stephen.green@documentengineeringservices.com>]

Hi Kevin

Hey, this very good, helpful detail on the metadata. Thanks.

The metadata at Sun, then, is mainly added *after* the test assertions
are written and therefore there seems less of a need for tags as part
of actual TA - more that tags are added to an existing TA and groups
facilitated after TAs are written. I guess you might not have examples
where TAs include grouping constructs when written, then. How about
profiles, etc. Any time when groups are used for profiles of existing
specs to designate TAs belonging to a profile which is a subset of an
existing spec? Is that more the domain of WS-I, etc? Would tags ever
be used to identify which TAs belong to some aspect of a profile, say
as part of a conformance requirement?

Thanks again

Steve

2008/9/22 Kevin Looney <Kevin.T.Looney@sun.com>:
> Hi Stephen,
>
>     Thanks for the re-write on this section.  It seems to address most of my
> concerns - regarding the 'explicitness' of list logic, the closed nature of
> lists for adding TAs, and Test Assertion Document.  This section is longer
> than previous, but it is much more explicit, contains examples, and has good
> glue to flow between the subsections -- So all of this is goodness.
>
>     I am hoping that a definition for Test Assertion Document will also be
> placed in the glossary.
>
>     Below is a discussion about 'workflow of Spec Analysis' here at Sun --
> you were asking for a description of this last week.
>
> Thanks, Regards,
> Kevin L
>
>
> Regarding Workflows (background purposes)
>
> You were previously asking about workflow for analysis (in Java Specs at
> Sun).   There are a few different workflows, typically depending on a few
> variables:
>
> 1.) whether a specification is original or in revision
> 2.) Size of a spec (sometimes groups break apart specs and analyze different
> sections at different times).
> .... (a few other variables)
> Paul and Victor can comment further about this, as their groups are actually
> doing this.
>
>
> In the workflow for an Original Spec:
>
> Typically, a specification (or portion) of an original spec will carry some
> hierarchy (eg the 'VM portion of the Java Specification', or the 'Util
> portion of the Java Language Specification').  Within these portions, there
> also is some hierarchy, typically aligned with the natural package
> organization of the Java Language, then further by class/method
> organization.  TAs (as Sun sees them) are currently associated with this
> organization - so part of describing a TA is identifying the context where
> it belongs in the Spec.
>
> Second step is typically to describe (Sun's version of) TAs by identifying
> all of the relevant fragments of text per TA.
>
> Third step is to annotate these TAs with appropriate descriptors
> (meta-data).  The types of Meta-Data may vary between different teams
> (associated with analyzing different portions of Specs).   Typically,
> 'testable/nontestable', 'ambiguous', ...  these meta-data are
> identified/attributed in this step.   Usually, this type of Meta-Data
> describes whether a test can be written from the TA.    Other types of
> Meta-Data can be identified later - this can be used for grouping tests (for
> data analysis) or for configuration/runtime control of tests.
>
> If the workflow is for a Spec Revision:
>
> The first step is typically to track differences in assertions within the
> original Spec, and the Revision Spec
>     a.) Identify existing TAs (unchanged)
>     b.) Identify TAs that have changed (significantly) - these become new
> TAs
>     c.) Identify missing TAs (for tracking purposes)
>     d.) Identify new TAs
>
> For (a/b), Meta-Data is typically inherited from the analysis of the
> original specification.  This may be updated however
> For (d) New TAs, the process is the same as the workflow for an original
> spec.
>
> This analysis is often fedback to the original specification writers, and
> specifications can improve based on the analysis.
>
> Sun typically works further with these TAs/Analysis  when test-suites are
> created.  Tests are associated with TAs, and coverage metrics are reported.
> This gives Sun an indication of 'depth' of testing in a technology area, as
> well as 'completeness' of testing.   Meta-data associated with TAs can help
> this reporting as well.
>
>
>
> So in relation to the discussion on Grouping,  there is a natural grouping
> of TAs by location (hierarchy) that happens when assertions are identified.
> There is also a minimal set of Meta-Data that is typically assigned when
> assertions are identified as well (for purposes of test identification).
> And, there is also some set of Meta-Data that can be assigned either for
> control or analysis purposes.  Grouping is useful for these purposes.
>
>
>
>
>
>
> Stephen Green wrote:
>
> Here is some candidate text for the Grouping sections (see also in
> wiki). It doesn't include yet the AI of providing text for a new
> glossary item 'Test Assertion Document' but I've added a subsection by
> that title to the Lists section.
>
> <start/>
>
> 3.5 Test Assertion Grouping
>
> When writing test assertions, as the specification is being analysed,
> it is usual to group certain test assertions together, either as
> having a special status, such as all accredited test assertions for a
> given specification, or as sharing a particular characteristic, such
> as a common category of test assertion target.
>
> A special kind of grouping is the container of all test assertions
> which belong to a particular specification or profile. Here the
> container may be the specification document itself if it includes
> within it the test assertions to be associated with it.
>
> Ways to group test assertions assertions include two of special note:
> - explicit listing of test assertions by their identifiers (section
> 3.5.1) and a more implicit grouping by a common but not unique
> property such as the test tags names or tag values assigned to the
> test assertions (section 3.5.2).
>
>
> 3.5.1 Lists (Dimensions of Variability)
>
> To explicitly identify a group of test assertions they can be listed
> explictly by the unique test assertion identifiers. This makes it
> clear once and for all which assertions belong to that particular
> group and which do not. In addition to such a list, the logical reason
> which determines whether a test assertion is a member or not of that
> list will need to be stated, at least to help with understanding and
> maintenance of the list.
>
> For example:
>
> Test Assertion List Id: A001
> List Description: all assertions describing 'Size' requirements
> List members: TA001, TA002, ....., TA008
>
> Note that although, in this example, we have avoided enumerating each
> and every test assertion identifier by using an ellipsis ('...'), such
> methods introduce a possible weakness. It might be overlooked during
> maintenance of the test assertions or the test assertion list if a
> later test assertion is given an ID of TA002a and therefore is
> implicitly rather than explictly made a member of this particular
> list. This may be a mistake since the new test assertion TA002a might
> not relate to the list description of 'Size' requirements. A list is
> completely explicit about every test assertion member when every
> member test assertion is listed explicitly by its test assertion
> identifier.
>
> For example:
>
> Test Assertion List Id: A001
> List Description: all assertions describing 'Size' requirements
> List members: TA001, TA002, TA003, TA004a TA004b, TA005, TA006, TA007, TA008
>
> Such a list may be regarded as 'fixed', 'frozen' or 'closed'. A test
> assertion added later with identifier TA005a, if it is to be included
> in this list, would require a change to the list (with possible
> version or change control implications) or the creation of a new list
> (with a new list identifier, if a list identifier is included with the
> list).
>
>
> Test Assertion Document
>
> A list of test assertions related to either conformance or
> interoperability testing will need special care with respect to
> version control and change management and therefore it will need to be
> clear what criteria are being used to determine which test assertions
> are members of the list and which are not. The special case of a
> container for all test assertions related to a given specification or
> profile, say, is a special example of a most explicit list, although
> here the method used to define such a list may involve the use of
> inclusion of the test assertion itself (rather than just its
> identifier) within a special document or package. One way to create
> such a list is to include all such related test assertions within a
> document, which we might call a 'Test Assertion Document'. Other
> synonymous terms might be 'Test Assertion List'. 'Specification
> Analysis' or 'Test Assertion Set'. Note that the container of this
> complete set of test assertions might instead be the document of the
> specification or conformance profile itself, when test assertions are
> included, say, within the text of the actual specification or profile.
>
>
> 3.5.2 Tags (Test Assertion Metadata)
>
> Another way to define a group of test assertions is to use a
> non-unique property of such assertions rather than just using their
> unique identifiers in a list or containing the test assertions
> themselves in a document. To this end, test assertions may be assigned
> metadata in the form of non-unique 'tags' or 'labels'.
>
> For example, test assertion 'widget-TA100-7' might be tagged as
> 'Size-Property-Descriptive':
>
> TA id: widget-TA100-7
> Target: widget
> Normative Source: specification requirement 104
> Predicate: [the widget] is from 5 to 15 centimeters long in its longer
> dimension.
> Prescription Level: medium-size:mandatory
> Tag: Size-Property-Descriptive
>
> Then it might be included in a list of test assertions related to
> 'Medium Size' requirements, along with other assertions, say, tagged
> 'Size-Related' but 'not' say with test assertions
> 'Small-Size-Related'.
>
> Test Assertion List Id: A002
> List Description: all assertions involved in describing 'Medium Size
> Widget' requirements
> List members: All test assertions with Tag 'Size-Property-Descriptive'
> AND Tag 'Size-Related' AND NOT Tag 'Small-Size-Related'
>
> This we have called a 'List' but it is in fact defined rather more
> implicitly than if every member were listed by its identifier, as
> described in the previous section (section 3.5.1). In fact a more
> explicit and well-defined list might combine both tags and identifiers
> to group the assertions:
>
> For example:
>
> Test Assertion List Id: A002
> List Description: all assertions involved in describing 'Medium Size
> Widget' requirements
> List Description: All test assertions with Tag
> 'Size-Property-Descriptive' AND Tag 'Size-Related' AND NOT Tag
> 'Small-Size-Related'
> List members: TA001, TA002, TA003, TA004a TA004b, TA005, TA006, TA007, TA008
>
> So a tag is a further, optional test assertion element useful in
> grouping test assertions. It may sometimes be useful to create tags as
> name-value pairs. For example, tagging a test assertion:
>
> Tag: WidgetSize=Medium
>
> This would allow all test assertions related to requirements for
> medium sized widgets to be grouped to facilitate, say, testing of just
> the medium sized widgets or a conformance profile relating just to
> medium-sized widgets.
>
> Several such filters can be applied to the same set of assertions and
> any given assertion can appear in more than one grouping.
>
> Special consideration when using tags for grouping is to be given to
> the stages in the workflow of test assertion authoring and maintenance
> and subsequent use at which changes might be made to tags and their
> values. Specially there may be the addition of new tags, perhaps by
> adding metadata which is separate to the documented test assertion. If
> metadata for test assertions is defined and maintained separately from
> the test assertions it may be subject to an entirely different set of
> version and change control rules and methodologies. In this case, a
> distinction might need to be made between lists based tags which were
> part of the original test assertion and those whose list membership
> might be different to that which was known or expected at the tiome
> the list was defined.
>
> For example, consider a list defined using tags but without explictly
> listing test assertion identifiers:
>
> Test Assertion List Id: A002
> List Description: all assertions involved in describing 'Medium Size
> Widget' requirements
> List Description: All test assertions with Tag
> 'Size-Property-Descriptive' AND Tag 'Size-Related' AND NOT Tag
> 'Small-Size-Related'
>
> If TA004a is originally tagged 'Size-Related' but the workflow allows
> it to be subsequently tagged 'Small-Size-Related', then there will
> need to be rules defined which determine whether the test assertion is
> still a member of List 'A002'.
>
>
>
> <end/>
>
>
>
>



-- 
Stephen D. Green

Partner
SystML, http://www.systml.co.uk
Tel: +44 (0) 117 9541606
Associate Director
Document Engineering Services
http://www.documentengineeringservices.com

http://www.biblegateway.com/passage/?search=matthew+22:37 .. and voice