RE: [tag] some thoughts and comments from reading a month worthofmessages

[stephen.green@systml.co.uk]

Quoting "Durand, Jacques R." <JDurand@us.fujitsu.com>:
>
> How to convey this test logic in a "prose" TA?
> (a)- either force a rewriting of the prose without the keyword, e.g. "
> When API function F(x) is called with argument value greater than MAX,
> then the API implementation PASS this test if F(x) = F(MAX), and FAIL
> otherwise."
> (b)- or add a meta statement to the specification statement: "the test
> outcome is PASS if the expected behavior is observed, FAIL otherwise."
>
> In (b) the RFC keywords are still there, but the TA text is clear as
> when pass/fail.
>

Yes, this is like how I was thinking with the metadata attribute like
'isTrueFalse'. There would be a problem with 'isPassFail' since it
might be more than just pass and fail - it might be pass fail or error
or it might be pass fail error or inconclusive or even pass fail error
inconclusive or inapplicable. It is in some cases determined by the
logic of the TA (e.g. depending on whether there are clear conditions
to mean a test can be inapplicable/inappropriate) and in most cases
determined by the test methodology, to which the TA should be agnostic.

Maybe the guideline could be that if the value is not boolean (isTrueFalse
= false) then it is to be assumed that the outcomes will include pass
and fail but not exclusively (other values may be dependent on factors
out of scope).

But if we can model something satisfactory then this seems an essential
feature to allow automation by those who use suitable expressions without
over-complicating things for those who don't want the automation as such
(or for TAs which don't allow automation).

> NOTE: when there are many spec requirements (either MUST or SHOULD) of
> the same style for a large number of API functions, they could all share
> the same additional meta statement (b), so as to avoid repetition.

I agree

>
> Granted, we do not have this ambiguity in case we have MUST instead of
> SHOULD, but I'd still prefer to handle these spec requirements in a
> similar way: apply (a) or (b) above, regardless of the RFC keyword used.
> Would that work for you?

Yes, thanks. I would understand SHOULD as a MUST with possible exceptions.
I think we can do better for allowing formalization (for potential
automation, say, of a SHOULD when the spec wording can be rewritten) if
we add 0..n exception elements to the model

-TA
--prose (0..1)
--pointer (0..n)
--flow (0..n)
---pre (0..n)
---trigger (0..?)
---result (1..n)
---exception (0..n)
---post (0..n)

Not all SHOULD keyword uses would be expected to be handled this way
since sometimes the exceptions are not explicit and no information is
given in the spec about implied exceptions but this would be one way
a spec architect would benefit from thinking about the TAs - make it
more clear what the exceptions are which make it SHOULD rather than
MUST and perhaps then make it just a MUST with those exceptions :-)

Best regards

Steve

>
> Regards,
> Jacques
>
> -----Original Message-----
> From: stephen.green@systml.co.uk [mailto:stephen.green@systml.co.uk]
> Sent: Thursday, September 20, 2007 11:42 PM
> To: tag@lists.oasis-open.org
> Subject: Re: [tag] some thoughts and comments from reading a month worth
> ofmessages
>
> Hi Serm,
>
> Quoting and itemizing your breakdown of three main parts of the TA
>
> A, > - Prose - normative for human consumption B, > - Pointer to
> specification - normative for human consumption and analysis C, > - TA
> Model - detail break down of TA components
>
> I understand that when speaking of the 'TA' you are really referring to
> the flow part and calling that the TA Model, C. So I get the sense you
> may mean that C (above) should not include the RFC keywords (MUST, etc).
> I gather then that there might not be an objection to the likelihood
> that the prose and pointer (A and B) would by almost necessity include
> the RFC keywords. That I would agree with.
>
> --
> Stephen Green
>
> Partner
> SystML, http://www.systml.co.uk
> Tel: +44 (0) 117 9541606
>
> http://www.biblegateway.com/passage/?search=matthew+22:37 .. and voice
>
>
>
> Quoting Serm Kulvatunyou <serm@nist.gov>:
>
>> I would like the recall the discussion I had with JD when we sketching
>
>> the TA model. We didn't like the pre-test and post-test conditions
>> because the antecedent and consequences encompasses more than just
>> conditions as it could also be event and/or action. So we tried to
>> make it general.  One of the reason we didn't want to go into the
>> specific of separating the pre-condition, action, and event because
>> one form could be written using another form and event and action may
>> contain conditions. For example, the condition "the light is on" could
>
>> be written as an event "the system turns the light on", so is this
>> consequence a test affect or a post-condition or both. Given the
>> recent version of the TA model, maybe we can clearly distinguish the
>> test affect and post-condition in that 1) post-conditions indicate the
>
>> state the IUT and/or environment need to be in in order to evaluate
>> the test affect which leading to the pass/fail outcome. The
>> post-condition may be empty, but if it is not it is a prerequisite the
>
>> the test affect, i.e., if post-condition is not met the IUT cannot be
>> evaluated to fail hence the outcome is "cannot be determined" (or
>> "technical fail" may be is a common term used in IIC). 2) test affect
> is everything else beyond the post-condition that is necessary &
> sufficient for determining the "pass"
>> or "fail" outcome. The point is we need more guideline.
>>
>> There were some discussions about TA testability. There is a proposed
>> TA defintion which implies that all TAs must be testable. It is common
>
>> that some specification requirements are not testable or partially
>> testable, does it mean that we need no TA for them?
>>
>> I think we just use UML as non-normative for illustration and
> explanation.
>> We don't want to deal with the issue to map UML to XML/marks up.
>>
>> I think as far as spec dependency, it seems like if writing a TA for a
>
>> specific specification, conformances to other dependent specs can be
>> easily stated. However, if writing TA for something like a SID
>> (standard integration deployment) profile, TA needs to be sort of
>> cross-spec. In that case, a TA for SID may need to ref individual TA
> in each related spec.
>>
>> I generally lik the Rationale for Test Assertion is by Patrick Curran
>> but I think some workdings need to be revised. It can be used for
>> motivation/background section of the TAG doc.
>>
>> What is the definitive point of what is an implicit vs. explicit
> reference?
>>
>> Concerning one of the Dave Pawson comments, I think the separation of
>> the TA from MUST, SHOULD, MAY in the specification is the right
>> statement. It is true that TA always indicate the MUST conditions, but
>
>> I think we are making an argument that we want to separate TA from
>> determining the level of conformance. That is what JD is saying is
>> that regardless of MUST, SHOULD, MAY that is associated with the
>> requirements in the spec, TA only indicates condition to determine
> whether the requirement is met.
>>
>> It seems like the current annatomy of TA includes (in order of
>> increasing formality/machine friendliness):
>> - Prose - normative for human consumption
>> - Pointer to specification - normative for human consumption and
>> analysis
>> - TA Model - detail break down of TA components
>>
>> - Serm
>>
>
>
>
>