Re: [tag] comments on TAG 0.4

[stephen.green@systml.co.uk]

Thanks Lynne, my own comments in line below

Quoting Lynne Rosenthal <lynne.rosenthal@nist.gov>:

> Comments on TAG draft 0.4
>
> 1.  Please add Tim Boland (NIST) to list of Contributors.  Serm
> Kulvatunyou is no longer at NIST

We will start an 'Issues'/change request list and add this to it

>
> 2.  Abstract:  "...mandatory and optional components..."
> Perhaps we need a conformance clause to explain that if you want to
> implement the guideline and create a TA then you need to implement the
> 'mandatory' components.

Last meeting it was agreed we'd not consider the guidelines to
be a spec but if we finish a TA markup then that would be candidate
for a spec,in which case this distinction becomes important I guess.

>
> 3.  Section 1.3 Coverage Analysis
> What is the relationship between the TA and coverage?  I have lots of
> problems with this section - it seems more about test suites and test
> than TAs.  Much of what is here relates coverage and test suites.
> Typically, if you don't have the TAs, you don't have tests.  What would
> help is a statement of what this section is trying to achieve (relate).

This is just outstanding, to be partly removed and partly merged into
the previous section(s) (Patrick's action item)

>
> 4.  Contradiction? - Sec 1.1 What is a Test Assertion and  Section
> called Testability
> In Sec. 1.1 "...Each is an independent testable statement..." and
> "sometimes an assertion is included which is not testable"
> It seems that we define TA to be testable and then say it sometimes
> isn't testable.
> Also, in Testability  "The test assertion should be testable".
> So, is a TA testable or not. I believe that the whole purpose of
> writing the TA is that it is testable.

I think we run into this as an issue especially when we consider
'prerequisites' (or preconditions) when a TA #B2 depends on a condition
being met which is described in another assertion #A1. #A1 might not
be subject to testing (even if it is testable), say. I would call it
an 'assertion' but not necessarily a testable assertion. I would say
that a test assertion is a subclass of an assertion. A prerequisite
or precondition may also be an assertion or a subclass of an assertion
separate to a test assertion. Also, the spec might include things which
are candidate assertions but whether or not they are testable might
depend on factors beyond the spec (perhaps implementation specific).
However it would not be desirable to exclude them as assertions from an
assertion list and so preclude them from any testing when it IS possible.
Making them assertions means it can be delegated to metadata to
state whether or not they are considered testable in a particular case.
It might be that they are not meant to be test assertions at all, as
when they are included only to provide logic to cater for an option
(such as Assertion #1: widget exists; TA #2: if #1 then ...).
Also a test assertion might be theoretically testable (might be written
as if there were a proper way to do testing) but not necessarily actually
testable. E.g. a requirement to 'understand' is theoretically testable
but unlikely to be testable (as when we cannot say that something which
is 'a SHOULD' can be tested as such). So we either say a TA might not be
testable (but SHOULD be testable, perhaps) or add another type of
assertion/predicate which isn't (necessarily) itself to be tested, or both.
I think we'll end up with both: both TAs which might not actually be
testable AND assertions which aren't necessarily even intended to be
tested but are probably testable (such as assertions which
correspond to optional statements rather than normative statements).

>
>
> regards
> Lynne
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail.  You may a link to this group and all your TCs in OASIS
> at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php