Re: [tag] handling optional (but normative) statements

[stephen.green@systml.co.uk]

I agree we could state in a TA "a package contains a widget X" when
the spec says "a package MAY contain a widget X".

However there are strange side effects for the scenario I brought up
earlier:

What if the spec says:

"a package MAY contain a widget X but widget X MUST
  have property Y otherwise there MUST NOT be a widget X"

(I agree better wording would be possible like
"a package MAY contain a widget X if widget X
  contains property Y otherwise there MUST NOT be a widget X")

Two TAs might be

#TA1 "package contains a widget X"
#TA2 "widget X has property Y"

Then there might be a third TA which seems to contradict the first
because it has to negate the first
#TA3 "if NOT #TA2 then NOT #TA1"

How else could the TA(s) be written?

To me it all makes more sense to the test writer when something is
added to include the MAY, the MUST and the MUST NOT, etc. I wouldn't
want to rule that as out of scope metadata. It seems it is the only
way to pass on from the spec how the TAs relate to eachother and
to their outcomes (their purposes, that is). I would see the MAY,
MUST, etc as qualifying something - perhaps the verb as in the prose
or perhaps the TA as a whole. Maybe it is metadata but if so I think
it is still so much part of the spec that it deserves to go with
the TA, included by the TA writers.

- Steve


Quoting "Durand, Jacques R." <JDurand@us.fujitsu.com>:

> Revisiting the proposals for handling optional statements:
>
> Examples:
>
> "a package MAY contain a widget X "
>
> "While the Sequence is not closed or terminated, the RM Source SHOULD
> retransmit unacknowledged messages."
>
> The approach we were leaning toward at F2F, is to say: there is no TA to
> write here, as it is impossible to violate such statements. Or at least,
> one should wait for someone to write a conformance profile that
> "upgrades" such optional statements to MUST statements,  before worrying
> about writing a TA for them.
>
> After more thoughts, I believe we need to reconsider this for two main
> reasons:
>
> 1- We agreed that TAs can be written with other goal in mind than
> conformance. Here are two of these extra goals:
> - improve specification quality at the time it is written.
> By making sure that the (for now optional) feature is specified clearly
> enough to be testable, we ensure that it will be testable later on,
> should a conformance profile require it.
> - another use of TA that has been reported by others, e.g. the Voting
> standard group: "implementation statements".
> See http://www.eac.gov/vvsg/part1/chapter02.php/, in section 2.3.  The
> idea is that one may want to know what an implementation is capable of,
> independently of any conformance concerns. E.g. several options exist,
> and some test suite will simply reveal which options are supported and
> which are not, without any conformance profile associated with these. In
> that case, clearly some TA will need be written for all the SHOULD and
> MAY.
>
>
> 2- Even with conformance in mind, the TA is only about measuring whether
> a specification feature is supported or not (true/false), and NOT into
> making a conformance assessment. Support for specification feature is a
> true or false proposition which is independent from the requirement
> level MUST/SHOULD/MAY: these keywords matter when deciding of
> (pass/fail), not of (true/false).
> And different test cases could be derived from the same TA, one will
> flag a "false" as a "pass with warning", the other as a "fail".
> There is no point to wait for a conformance profile to be fully defined
> before writing the TA: the TA only depends on the feature to be tested
> and does not vary with the conformance level. What to do with the result
> of the TA expression (true/false), is precisely the job of a test suite,
> and behind it of a conformance evaluation. Not for the TA to be
> concerned about.
>
> Of course, I am not saying our guideline doc should mandate that folks
> *always* write TAs for optional statements. Only that such TAs are
> useful, and should they decide to write such TAs they should do it the
> following way:
>
> Treat any of the 3 following statements in exactly the same way:
> "A package MAY contain a widget X "
> "A package SHOULD contain a widget X "
> "A package MUST contain a widget X "
>
> which means:
> - the same TA will equally address any one of these 3 normative
> statements.
> - The assertion expression derived from each one of these normative
> statements is then exactly the same, and proceeds from applying the same
> rule: just ignore the <MAY/SHOULD/MUST> keyword, and give the normative
> statement an assertive form: "the package contains a widget X". (TA
> target = package) (the "bold statement" of UniSoft, see
> http://www.unisoft.com/glossary.html
> <http://www.unisoft.com/glossary.html> )
>
> Immediate benefit:
> All the byzantine distinctions considered in F2F about the different
> handling of use case #3 and use case #5 (see previous mail) go away.
>
> I suggest also to update the definition of TA as follows:
>
> replace:
> "A testable expression for evaluating adherence of an implementation (or
> part of) to some normative statement or requirement in the
> specification."
>
> with:
> "A testable expression for evaluating whether or not an implementation
> (or part of) exhibits a feature or behavior subject to a normative
> statement or requirement in the specification."
>
> rationale:
> - this definition more clearly distanciates the role of a TA from
> conformance evaluation. "Adherence" is still too close to conformance,
> which we carefully avoid to mention in anything concerning TA.
> - It puts the focus on measuring a feature /behavior, not on how well
> the specification is followed (again too close to a conformance job).
> - "adherence to a normative statement" is meaningless in case of an
> optional statement, if we agree a TA makes sense for such cases. Need be
> more specific.
>
> Jacques
>