OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

tag-comment message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: RE: Section 3.7 Specification Versions

I'm glad to see that the VersionAdd/VersionDrop mechanism is useful. Here are a few comments:

<JD> The above paragraph could be made easier to read by showing snippet(s) of example:
 tag: VersionAdd = rev15
tag: VersionDrop = rev19
 means that the TA applies from rev15 up to rev18 included.

More specifically, it means that rev18.1, rev18.2, etc. ARE included.

   tag: VersionAdd: the lowest numerical version to which the test assertion applies.
  tag: VersionDrop: the lowest numerical version number to which the test assertion does NOT apply.

<JD> could we have two or more Add-Drop intervals in the same TA?

That would be messy. I think the other line of thinking, to use a separate but "related" TA, is probably better. More on that after I cite Kevin's comment.

KL>Much of my discussion is based on the need to maintain metrics, metrics that show (to specification writers, and testing leads) that the amount of a specification being analyzed and tested is increasing.

Here is how I would apply the metrics: Take the test suite at any given snapshot. (With TAs and sufficient test case metadata, you can do this for various points in time.) Take a given implementation to be tested, and note which version (number) of the spec it implements. Use that version number and the VersionAdd/VersionDrop properties to filter the TAs to just the relevant ones, then fetch the test cases that apply to those TAs. Now you can have a count of the number of relevant TAs and the number of relevant test cases, and if you apply the test cases against the implementation, you can count pass, fail, and other outcomes. Depending on how you structured the relationship between TAs and test cases, you can choose another point in time and either re-fetch the set of cases associated with the first set of relevant TAs, or you can fetch the set of TAs as of that point-in-time and filter down to what's relevant using VersionAdd/VersionDrop.

Again: with sufficient metadata, you could answer questions such as how many tests did we apply against that implementation back then vs. how many relevant tests can we apply today? If today's suite is richer even for a prior spec version, then you can quantify how many bugs can be caught now that would have been missed before.

KL>Arguably, significant meaning changes should be accompanied by drop/add of TAs, where the TAs should have a different ID.
KL>Still, it is important to change the tests (implementation) that may be associated with changed assertion, therefor, it is important to maintain a history trail of where an assertion (change) may have derived from.
KL>One idea that came up at the last meeting was to use another "Version" event, "VersionReplaceID".

That's fine. I wonder if it has enough universal value to be in the TA spec at this early stage. There are probably lots of other things that would be nice to have in the TA spec, but have been excluded "for now" and open to later discussion.

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]