Re: TAG Proposal on weak predicates

[Patrick Curran <Patrick.Curran@Sun.COM>]

We discussed this briefly during today's teleconference. This allowed me to collect my thoughts on the matter.

As I understand the arguments, you're suggesting that because it may be difficult to test a particular assertion with the tools or test framework that's available, it is therefore appropriate to "weaken" the assertion such that it is testable. Having done so, you argue that it is therefore necessary to tag the assertion as "weak".

I strongly disagree with this approach. Whether or not one is able (in the example you give, "willing" seems a more appropriate term) to test a normative requirement in the specification should not influence the derivation/identification of the appropriate assertion. Assertions should reflect the spec exactly. (Of course, a spec that contains a significant number of requirements that are untestable or difficult to test is a poor specification, since in practice implementations will tend to differ from each other in these areas. The very process of developing test assertions can help to identify such cases, and if performed early enough in the development cycle, feedback can be provided to improve the spec.)

As I said, the list of assertions should exactly match the normative requirements in the spec. When it comes to testing, judgment calls are always made. Some assertions are not tested at all. Others are partially tested. Some are "completely" tested. In the example you give, the assertion would be partially tested. There is clearly value in annotating an assertion list with information about what is and what is not tested, and about the thoroughness of the testing that is performed. Such annotations seem to me to be a perfect example of "test metadata", and therefore out of scope for our document.

In conclusion, I see no need for the interpretation qualifier, at least in this case.

Durand, Jacques R. wrote:

Patrick:

 

for your review:

 

3.4 Weak Predicates (proposal)

In an ideal situation, the outcome of the TA predicate serves as an indicator of both fulfillment and violation of the normative statement addressed by the TA:

• An outcome of “True” for a target, means the target fulfills the normative statement.
• An outcome of “False” for a target, means the target violates – or at the very least does not fulfill - the normative statement.

It may not always be possible to design such a predicate, due to the “testability” requirement. In some cases, the normative statement is only partially testable, given the test environment and constraints that are assumed by the Test Assertion writer.

For example, consider:

Requirement 120: “If the message body is signed with an XML digital signature that does not also sign the header, then there must be another signature that signs both the header and the body signature, and both signatures must use the same key.”

The presence and scope of XML signatures can be verified just by “looking” at the message structure (signature elements explicitly refer to the parts they sign). The only requirement that cannot be verified by examining the message, is the use of the same key. Given a test environment that is not assumed to have any knowledge of the digital certificates related to the set of messages exchanged or logged, the following predicate is the best that can be designed:

• Predicate: [the message] has a signature covering both the header and the body signature.

Note that the "if" part of Requirement 120 is not part of the predicate. It will map to the prerequisite element (see below). The outcome of such predicate can be interpreted as:

• “True” means undetermined, because the normative statement is only partially tested: it is still possible that two different keys were used, in which case the normative statement is violated.
• “False” means the target violates the normative statement.

Because only one outcome – here the “false” outcome - is conclusive about the normative statement, the predicate is said to be “weak”. When a TA has a weak predicate, one of its outcomes must explicitly map to the value "undetermined". The recommended notation is to add an Interpretation item in the predicate specification, that maps each outcome to one of the three values: {fulfillment, violation, undetermined} :

 

TA id: widget-TA120

Target: widget

Normative Source: specification requirement 120

Prerequisite: [the message] has a signed body and the body signature does not cover the header.

Predicate: [the message] has a signature covering both the header and the body signature. Interpretation: true=undetermined, false=violation.

Prescription Level: mandatory