Re: [tag] comments on TAG 0.4

[lynne rosenthal <lynne.rosenthal@nist.gov>]

My comments are also in line.  Thanks for your thoughts Stephen.

At 05:21 PM 1/28/2008, stephen.green@systml.co.uk wrote:
>Thanks Lynne, my own comments in line below
>
>Quoting Lynne Rosenthal <lynne.rosenthal@nist.gov>:
>
>>Comments on TAG draft 0.4
>>
>>1.  Please add Tim Boland (NIST) to list of Contributors.  Serm
>>Kulvatunyou is no longer at NIST
>
>We will start an 'Issues'/change request list and add this to it

An Issues list is a good idea



>>2.  Abstract:  "...mandatory and optional components..."
>>Perhaps we need a conformance clause to explain that if you want to
>>implement the guideline and create a TA then you need to implement the
>>'mandatory' components.
>
>Last meeting it was agreed we'd not consider the guidelines to
>be a spec but if we finish a TA markup then that would be candidate
>for a spec,in which case this distinction becomes important I guess.

I remember the discussion.  We said we wouldn't have a conformance 
clause since its not a spec. But, if we are going to say something is 
mandatory, then what does that mean?  Mandatory for what?



>>3.  Section 1.3 Coverage Analysis
>>What is the relationship between the TA and coverage?  I have lots of
>>problems with this section - it seems more about test suites and test
>>than TAs.  Much of what is here relates coverage and test suites.
>>Typically, if you don't have the TAs, you don't have tests.  What would
>>help is a statement of what this section is trying to achieve (relate).
>
>This is just outstanding, to be partly removed and partly merged into
>the previous section(s) (Patrick's action item)

O.K.



>>4.  Contradiction? - Sec 1.1 What is a Test Assertion and  Section
>>called Testability
>>In Sec. 1.1 "...Each is an independent testable statement..." and
>>"sometimes an assertion is included which is not testable"
>>It seems that we define TA to be testable and then say it sometimes
>>isn't testable.
>>Also, in Testability  "The test assertion should be testable".
>>So, is a TA testable or not. I believe that the whole purpose of
>>writing the TA is that it is testable.
>
>I think we run into this as an issue especially when we consider
>'prerequisites' (or preconditions) when a TA #B2 depends on a condition
>being met which is described in another assertion #A1. #A1 might not
>be subject to testing (even if it is testable), say. I would call it
>an 'assertion' but not necessarily a testable assertion. I would say
>that a test assertion is a subclass of an assertion. A prerequisite
>or precondition may also be an assertion or a subclass of an assertion
>separate to a test assertion. Also, the spec might include things which
>are candidate assertions but whether or not they are testable might
>depend on factors beyond the spec (perhaps implementation specific).
>However it would not be desirable to exclude them as assertions from an
>assertion list and so preclude them from any testing when it IS possible.
>Making them assertions means it can be delegated to metadata to
>state whether or not they are considered testable in a particular case.
>It might be that they are not meant to be test assertions at all, as
>when they are included only to provide logic to cater for an option
>(such as Assertion #1: widget exists; TA #2: if #1 then ...).
>Also a test assertion might be theoretically testable (might be written
>as if there were a proper way to do testing) but not necessarily actually
>testable. E.g. a requirement to 'understand' is theoretically testable
>but unlikely to be testable (as when we cannot say that something which
>is 'a SHOULD' can be tested as such). So we either say a TA might not be
>testable (but SHOULD be testable, perhaps) or add another type of
>assertion/predicate which isn't (necessarily) itself to be tested, or both.
>I think we'll end up with both: both TAs which might not actually be
>testable AND assertions which aren't necessarily even intended to be
>tested but are probably testable (such as assertions which
>correspond to optional statements rather than normative statements).

I agree with your discussion.  Perhaps testability is a desirable 
attribute, but not a necessary one.  For now, lets know that we need 
to be careful and consistent with how we describe and define a TA.


--Lynne