RE: [tag] Non Testable Normative Sources

[Jacques Durand <JDurand@us.fujitsu.com>]

OK - see if the updated text below would encourage this:
"Finally, it may be the case that the Normative Source is considered to be non-testable, to the best of the knowledge of the test assertion writer. For example, it may concern a property or behavior that is not easily observable by a test environment, or that remains privy to an implementation. 
Such test assertions still serve a purpose: 
- if the specification work is still ongoing, such test assertions may alert specification authors that some of the normative statements may not be testable. This may lead to refining the specification to avoid this.
- at the time an implementation is developed if not tested: they provide guidance to developers by calling out more explicitly the requirements for conformance, and also indicate what kind of feature should be made observable (and testable) when possible, during implementation. 
In all cases it is desirable that every test assertion be usable to derive executable test cases. If the test assertion predicate is to remain "un-testable", the test assertion could be tagged as "not-testable" so that test suite writers can ignore it. Note that as a test environment evolves, non-testability may be temporary."
-jacques
-----Original Message-----
From: Boland Jr, Frederick E. [mailto:frederick.boland@nist.gov] 
Sent: Tuesday, April 16, 2013 3:49 AM
To: kevin.looney@oracle.com; Jacques Durand
Cc: TAG TC List
Subject: RE: [tag] Non Testable Normative Sources

agreed, particularly point 3 - we should encourage assertions to be testable wherever/whenever possible..  Thx Tim ________________________________________
From: tag@lists.oasis-open.org [tag@lists.oasis-open.org] On Behalf Of Kevin Looney [kevin.looney@oracle.com]
Sent: Monday, April 15, 2013 11:22 PM
To: Jacques Durand
Cc: TAG TC List
Subject: Re: [tag] Non Testable Normative Sources

Hi Jacques,

     My apologies - I've been crazy busy - and couldn't draft the verbage last week.

That said:
     1.) I was going to recommend a retitle of the section, to account for the extended description - so you read my mind there.
     2.) the text you wrote covers most of my concerns - thanks, it looks good.
     3.) Finally, I think we briefly discussed last week that environments might change, and non-testable assertions may evolve into testable ones.  In this case, the non-testable assertion acts as a 'placeholder' to be corrected as the environment changes.

Thanks,
Kevin


On 4/15/13 6:53 PM, Jacques Durand wrote:
Kevin and all:
Would the addition below at the end of 4.6 address the "non-testable" case?
4.6 Partially or Non Testable Normative Sources ....

"Finally, it may be the case that the Normative Source is known to be non-testable, to the best of the knowledge of the test assertion writer. For example, it may concern a property or behavior that will likely not be observable by a test environment, or that will likely remain privy to an implementation. Such test assertions still serve a purpose in guiding implementors to develop in conformance to a specification, and also indicate what kind of feature should be made observable when possible, at the time an implementation is developed. If the test assertion predicate is to remain "untestable", the test assertion could be tagged as "not-testable" so that test suite writers can ignore it."



-jacques




--
kevin.looney@oracle.com<mailto:kevin.looney@oracle.com>