RE: Section 3.7 Specification Versions

[david_marston@us.ibm.com]

I'm glad to see that the VersionAdd/VersionDrop
mechanism is useful. Here are a few comments:

<JD> The above paragraph
could be made easier to read by showing snippet(s) of example:
 tag:
VersionAdd = rev15
 tag: VersionDrop
= rev19
 means that the TA applies
from rev15 up to rev18 included.

More specifically, it means that rev18.1,
rev18.2, etc. ARE included.

   tag: VersionAdd: the lowest numerical version
to which the test assertion applies.
   tag: VersionDrop: the lowest numerical version number to which
the test assertion does NOT apply.
 
<JD> could we have two or
more Add-Drop intervals in the same TA?

That would be messy. I think the other
line of thinking, to use a separate but "related" TA, is probably
better. More on that after I cite Kevin's comment.

KL>Much of my discussion is based on
the need to maintain metrics, metrics that show (to specification writers,
and testing leads) that the amount of a specification being analyzed and
tested is increasing.

Here is how I would apply the metrics:
Take the test suite at any given snapshot. (With TAs and sufficient test
case metadata, you can do this for various points in time.) Take a given
implementation to be tested, and note which version (number) of the spec
it implements. Use that version number and the VersionAdd/VersionDrop properties
to filter the TAs to just the relevant ones, then fetch the test cases
that apply to those TAs. Now you can have a count of the number of relevant
TAs and the number of relevant test cases, and if you apply the test cases
against the implementation, you can count pass, fail, and other outcomes.
Depending on how you structured the relationship between TAs and test cases,
you can choose another point in time and either re-fetch the set of cases
associated with the first set of relevant TAs, or you can fetch the set
of TAs as of that point-in-time and filter down to what's relevant using
VersionAdd/VersionDrop.

Again: with sufficient metadata, you
could answer questions such as how many tests did we apply against that
implementation back then vs. how many relevant tests can we apply today?
If today's suite is richer even for a prior spec version, then you can
quantify how many bugs can be caught now that would have been missed before.

KL>Arguably, significant meaning changes
should be accompanied by drop/add of TAs, where the TAs should have a different
ID.
KL>Still, it is important to change the tests (implementation) that
may be associated with changed assertion, therefor, it is important to
maintain a history trail of where an assertion (change) may have derived
from.
KL>One idea that came up at the last meeting was to use another "Version"
event, "VersionReplaceID".

That's fine. I wonder if it has enough
universal value to be in the TA spec at this early stage. There are probably
lots of other things that would be nice to have in the TA spec, but have
been excluded "for now" and open to later discussion.