RE: [tag] handling optional (but normative) statements

["Durand, Jacques R." <JDurand@us.fujitsu.com>]

 inline <JD>

-----Original Message-----
From: stephen.green@systml.co.uk [mailto:stephen.green@systml.co.uk] 
Sent: Monday, December 10, 2007 11:46 AM
To: tag@lists.oasis-open.org
Subject: Re: [tag] handling optional (but normative) statements

I agree we could state in a TA "a package contains a widget X" when the
spec says "a package MAY contain a widget X".

However there are strange side effects for the scenario I brought up
earlier:

What if the spec says:

"a package MAY contain a widget X but widget X MUST
  have property Y otherwise there MUST NOT be a widget X"

(I agree better wording would be possible like "a package MAY contain a
widget X if widget X
  contains property Y otherwise there MUST NOT be a widget X")

Two TAs might be

#TA1 "package contains a widget X"
#TA2 "widget X has property Y"

Then there might be a third TA which seems to contradict the first
because it has to negate the first
#TA3 "if NOT #TA2 then NOT #TA1"

How else could the TA(s) be written?

<JD> The problem with modeling TA3 as you did, is that it is not obvious
that the target of TA2 (the widget) is implicitly reused in TA1 which
has another target (the package). Now if you use a predicative form,
this link canbe made more explicit through variables: 
TA1(P,X) =  package(P) and widget(X) and contains(P, X)
TA2(W) = widget(W) and propertyof(W, Y) (note that Y is here assumed to
be known, not a variable)
Then you could say unambiguously: TA3(P) = for any widget X, not(TA2(X))
implies not(TA1(P, X)) 
(note:  here, I use the expression TA(x,y,z...) for the assertion
predicate)
</JD>

<JD> another approach is to consider we have two (not completely
equivalent) statements here, that do not try to break down the "package
part"  from the "widget part":
S1: "a package MAY contain a widget X with property Y"
S2: "a package MUST NOT contain a widget X that does not have property
Y"

#TA1a is now: "package contains a widget X with property Y"
#TA1b is now:  "package does not contain a widget X that does not have
property Y"
In a conformance profile  that preserves the optionality described in
S1, there would simply be no test case written for TA1a. But there could
be more demanding conformance profiles that will require TA1a to be true
for package targets. Both conformance profiles will however need a test
case to be derived from TA1b, that will map "false" to a "fail". Besides
all this, TA1a could be used in a test case that only verifies the
capabilities of an implementation (no testing against a specific
conformance profile), as in the "implementation statements" described in
the voting standard (see section 2.4 in
http://www.eac.gov/vvsg/part1/chapter02.php/)
However, you have a point that both #TA1a and #TA1b reuse the same #TA2
expression: "widget has property Y". It becomes then a convenience issue
to rewrite #TA1a and #TA1b so that they don't repeat this expression,
which is a hint to test case writers that they can reuse the same test
case derived from TA2:
 #TA1a' is : "package contains a widget X  AND TA2(X)"
#TA1b' is :  "not (package contains a widget X AND  not(TA2(X)))"
</JD>

To me it all makes more sense to the test writer when something is added
to include the MAY, the MUST and the MUST NOT, etc. I wouldn't want to
rule that as out of scope metadata. It seems it is the only way to pass
on from the spec how the TAs relate to eachother and to their outcomes
(their purposes, that is). I would see the MAY, MUST, etc as qualifying
something - perhaps the verb as in the prose or perhaps the TA as a
whole. Maybe it is metadata but if so I think it is still so much part
of the spec that it deserves to go with the TA, included by the TA
writers.

<JD> Stephen: initially, I wanted the result of a TA to be pass/fail as
I found it more intuitive (I believe so did you) and as a matter of
fact, TAs I have written in the past were of the pass/fail kind. But
like a few others in the TC I now agree that this may be too close to
test metadata, and that it is the job of a test case to actually decide
pass/fail - which is a conformance assessment - based on the true/false
of the TA this test case is derived from, and possibly in case of
MAY/SHOULD statement, interpreting true/false in a different way (e.g.
"false"  may map to "fail with warning". Or yet, no test case at all
will be written for this TA in case the conformance profile does not
care about this option). I believe the optional statements are a serious
challenge for pass/fail TAs... While they are not anymore if we scope
the TA role to only state a testable behavior/property (true/false). If
we leave the profile-dependent conformance decisions to derived test
cases, doesn't that simplify the job of writing Tas, while keeping these
more "reusable"?
</JD>. 

- Steve


Quoting "Durand, Jacques R." <JDurand@us.fujitsu.com>:

> Revisiting the proposals for handling optional statements:
>
> Examples:
>
> "a package MAY contain a widget X "
>
> "While the Sequence is not closed or terminated, the RM Source SHOULD 
> retransmit unacknowledged messages."
>
> The approach we were leaning toward at F2F, is to say: there is no TA 
> to write here, as it is impossible to violate such statements. Or at 
> least, one should wait for someone to write a conformance profile that

> "upgrades" such optional statements to MUST statements,  before 
> worrying about writing a TA for them.
>
> After more thoughts, I believe we need to reconsider this for two main
> reasons:
>
> 1- We agreed that TAs can be written with other goal in mind than 
> conformance. Here are two of these extra goals:
> - improve specification quality at the time it is written.
> By making sure that the (for now optional) feature is specified 
> clearly enough to be testable, we ensure that it will be testable 
> later on,

> should a conformance profile require it.
> - another use of TA that has been reported by others, e.g. the Voting 
> standard group: "implementation statements".
> See http://www.eac.gov/vvsg/part1/chapter02.php/, in section 2.3.  The

> idea is that one may want to know what an implementation is capable 
> of, independently of any conformance concerns. E.g. several options 
> exist, and some test suite will simply reveal which options are 
> supported and which are not, without any conformance profile 
> associated with these. In that case, clearly some TA will need be 
> written for all the SHOULD and MAY.
>
>
> 2- Even with conformance in mind, the TA is only about measuring 
> whether a specification feature is supported or not (true/false), and 
> NOT into making a conformance assessment. Support for specification 
> feature is a true or false proposition which is independent from the 
> requirement level MUST/SHOULD/MAY: these keywords matter when deciding

> of (pass/fail), not of (true/false).
> And different test cases could be derived from the same TA, one will 
> flag a "false" as a "pass with warning", the other as a "fail".
> There is no point to wait for a conformance profile to be fully 
> defined before writing the TA: the TA only depends on the feature to 
> be tested and does not vary with the conformance level. What to do 
> with the result of the TA expression (true/false), is precisely the 
> job of a test suite, and behind it of a conformance evaluation. Not 
> for the TA to be concerned about.
>
> Of course, I am not saying our guideline doc should mandate that folks
> *always* write TAs for optional statements. Only that such TAs are 
> useful, and should they decide to write such TAs they should do it the

> following way:
>
> Treat any of the 3 following statements in exactly the same way:
> "A package MAY contain a widget X "
> "A package SHOULD contain a widget X "
> "A package MUST contain a widget X "
>
> which means:
> - the same TA will equally address any one of these 3 normative 
> statements.
> - The assertion expression derived from each one of these normative 
> statements is then exactly the same, and proceeds from applying the 
> same
> rule: just ignore the <MAY/SHOULD/MUST> keyword, and give the 
> normative statement an assertive form: "the package contains a widget 
> X". (TA target = package) (the "bold statement" of UniSoft, see 
> http://www.unisoft.com/glossary.html
> <http://www.unisoft.com/glossary.html> )
>
> Immediate benefit:
> All the byzantine distinctions considered in F2F about the different 
> handling of use case #3 and use case #5 (see previous mail) go away.
>
> I suggest also to update the definition of TA as follows:
>
> replace:
> "A testable expression for evaluating adherence of an implementation 
> (or part of) to some normative statement or requirement in the 
> specification."
>
> with:
> "A testable expression for evaluating whether or not an implementation

> (or part of) exhibits a feature or behavior subject to a normative 
> statement or requirement in the specification."
>
> rationale:
> - this definition more clearly distanciates the role of a TA from 
> conformance evaluation. "Adherence" is still too close to conformance,

> which we carefully avoid to mention in anything concerning TA.
> - It puts the focus on measuring a feature /behavior, not on how well 
> the specification is followed (again too close to a conformance job).
> - "adherence to a normative statement" is meaningless in case of an 
> optional statement, if we agree a TA makes sense for such cases. Need 
> be more specific.
>
> Jacques
>




---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail.  You may a link to this group and all your TCs in
OASIS
at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php