handling optional (but normative) statements

["Durand, Jacques R." <JDurand@us.fujitsu.com>]

Revisiting the proposals for handling optional statements:

Examples:

"a package MAY contain a widget X "

"While the Sequence is not closed or terminated, the RM Source SHOULD retransmit unacknowledged messages."

The approach we were leaning toward at F2F, is to say: there is no TA to write here, as it is impossible to violate such statements. Or at least, one should wait for someone to write a conformance profile that "upgrades" such optional statements to MUST statements,  before worrying about writing a TA for them.

 

After more thoughts, I believe we need to reconsider this for two main reasons:

 

1- We agreed that TAs can be written with other goal in mind than conformance. Here are two of these extra goals:

- improve specification quality at the time it is written.

By making sure that the (for now optional) feature is specified clearly enough to be testable, we ensure that it will be testable later on, should a conformance profile require it.

- another use of TA that has been reported by others, e.g. the Voting standard group: "implementation statements".

See http://www.eac.gov/vvsg/part1/chapter02.php/, in section 2.3.  The idea is that one may want to know what an implementation is capable of, independently of any conformance concerns. E.g. several options exist, and some test suite will simply reveal which options are supported and which are not, without any conformance profile associated with these. In that case, clearly some TA will need be written for all the SHOULD and MAY.

 

 

2- Even with conformance in mind, the TA is only about measuring whether a specification feature is supported or not (true/false), and NOT into making a conformance assessment. Support for specification feature is a true or false proposition which is independent from the requirement level MUST/SHOULD/MAY: these keywords matter when deciding of (pass/fail), not of (true/false).

And different test cases could be derived from the same TA, one will flag a "false" as a "pass with warning", the other as a "fail".

There is no point to wait for a conformance profile to be fully defined before writing the TA: the TA only depends on the feature to be tested and does not vary with the conformance level. What to do with the result of the TA expression (true/false), is precisely the job of a test suite, and behind it of a conformance evaluation. Not for the TA to be concerned about.

 

Of course, I am not saying our guideline doc should mandate that folks *always* write TAs for optional statements. Only that such TAs are useful, and should they decide to write such TAs they should do it the following way:

 

Treat any of the 3 following statements in exactly the same way:

"A package MAY contain a widget X "

"A package SHOULD contain a widget X "

"A package MUST contain a widget X "

 

which means:

- the same TA will equally address any one of these 3 normative statements.

- The assertion expression derived from each one of these normative statements is then exactly the same, and proceeds from applying the same rule: just ignore the <MAY/SHOULD/MUST> keyword, and give the normative statement an assertive form: "the package contains a widget X". (TA target = package) (the "bold statement" of UniSoft, see http://www.unisoft.com/glossary.html)

 

Immediate benefit:

All the byzantine distinctions considered in F2F about the different handling of use case #3 and use case #5 (see previous mail) go away.

 

I suggest also to update the definition of TA as follows:

 

replace:

"A testable expression for evaluating adherence of an implementation (or part of) to some normative statement or requirement in the specification."

 

with:

"A testable expression for evaluating whether or not an implementation (or part of) exhibits a feature or behavior subject to a normative statement or requirement in the specification."

 

rationale:

- this definition more clearly distanciates the role of a TA from conformance evaluation. "Adherence" is still too close to conformance, which we carefully avoid to mention in anything concerning TA.

- It puts the focus on measuring a feature /behavior, not on how well the specification is followed (again too close to a conformance job).

- "adherence to a normative statement" is meaningless in case of an optional statement, if we agree a TA makes sense for such cases. Need be more specific.

 

Jacques