Notes of Best Practice comments

[stephen.green@systml.co.uk]

Notes taken about Best Practice sections from today's half of TC meeting:
(elaborations or amendments welcome)


BEST PRACTICE GUIDANCE FOR WRITING TEST ASSERTIONS

-GENERAL PRINCIPLES

--THE VALUE OF WRITING TEST ASSERTIONS

  Add that the writing of TAs provides for a tighter spec
  Add that the TA is ideally written in parallel with spec rather than  
merely early on in writing the spec
  This is all part of the foundation rather than best practice

--WHO WRITES TEST ASSERTIONS

  Clarify mention of TA mandating in relation to mandating of  
conformance clauses
  (Relates to the way a conformance clause can refer to a set of TAs)
  The mandating aspect refers to mandating of the inclusion of TA  
creation in the process of spec writing

--THE RELATIONSHIP BETWEEN TEST ASSERTIONS AND CONFORMANCE

  Conformance clauses are at a higher level.
  The conformance clauses will likely include lots more than the TAs,  
such as specifying which parts of the
   specification are normative or informative, the use of keywords  
such as MUST, SHOULD, MAY, etc.
  In contrast the TAs are a refinement to of parts of specification to  
provide the foundation for building tests
  TAs singular / Conformance clauses groups
  TAs have a narrower purpose
  Conformance clauses may make ref to TAs but are distinct from them
  Conformance clauses may specify conformance to levels, say, in  
relation to different TAs
  TAs like leafnodes in branching structure
  For examaple, Sun's conformance clauses (called compatibility  
requirements) run into pages
   and include details about which parts (or all) of the specifcation  
to implement, whether
   subsets or supersets may be implemented whether the Sun test  
harness must be used.
  Patrick not sure about just adding this section as a definition to  
the Glossary
   - more of a fundamental matter to be defined up front

--TEST ENVIRONMENT CONSIDERATIONS

  TA should NOT say how to test (the TA writers may not know how to  
test at time of writing)
   but should give enough detail so can tests can be constructed
  Again, include this in fundamentals section up front

  Debatable whether the IUT should be required at TA level - either  
can be part of conformance
   or may be generalized for a set of TAs
  For example, the conformance clauses might list classes of  
implementations or IUTs and this
   might result in production of an enumerated list or controlled  
vocabulary of IUTs (either in
   the conformance clause or test assertion suite introduction or  
both) which can be referred to,
   perhaps by coded values, in the individual TAs
  A best practice technique is to list the IUTs to which a TA applies  
using words like 'applies to ...'
  Consensus that IUT should not be required as part of the individual TA
  There may be granularity issues

  Use of 'how': not to be used regarding control of test but more  
appropriate in the behaviour aspect of a TA

  Harness: metadata bridges the gap between the TA and a test case in  
deterministic way

--COVERAGE

  Patrick will cover this

-WRITING SIMPLE STRUCTURE TEST ASSERTIONS

  Care needs to be excercised when converting a requirement to a TA  
not to be adding interpretation
   or subtly changing the meaning
  Just using references to spec avoid adding interpretations but then  
care needs to be taken to
   make the reference sufficiently specific such as by specifying the  
boundary of which part is
   referenced or to reference a sentence
  There is value in including text outside the spec in the TA does but  
there is also a danger of
   divorcing the text from its context, formatting, etc
  One best practice is to use some technique to highlight text  
actually in the spec itself and add
   a TA Id to it there
  Note that a the relationship between TA and spec may be  
many-to-many; there may even be more than
   one spec to which references need to be made


-MORE COMPLEX STRUCTURE TEST ASSERTIONS
[look at later]


-AUTOMATION
  Note: generation from schema can just go straight to test by-passing TA


-IMPACT OF VARIABILITY IN SPECIFICATIONS ON TEST ASSERTIONS
  e.g. 'applies to'

-GROUPING

-SEQUENCING
  Patrick still nervous  - prob if modular
  specs on top of other specs


-
Stephen Green

Partner
SystML, http://www.systml.co.uk
Tel: +44 (0) 117 9541606

http://www.biblegateway.com/passage/?search=matthew+22:37 .. and voice