RE: [tag] requirements = test assertion

["Durand, Jacques R." <JDurand@us.fujitsu.com>]

Forwarding this to the TC as it is clearly yet another prior art of interest.

 

My comments:

 

- I like the "class" approach, in section 2.5.2. that identifies classes of implementations, with the purpose of targeting a set of conformance statements to each class. By knowing the relationship between classes, you can infer which requirements apply, e.g. if a class A inherits from a class B, the requirements targeting A also include requirements targeting B.

To me, that says that it helps to think of targets (IUTs) in tems of classes of items.

 

Specific cases we could reuse in a guideline doc:

 

-Requirement that is written is such a precise and testable way that nothing more need be added to a related TA

(3...)The system SHALL achieve a Perfect Ballot Index of at least 2.33 as measured by the VPP.

- Requirement that would require a more precise expression when writing a TA, as in itself it does not provide any measurable or testable behavior. A TA woudl for example narrow the notion of "help" (or define the forms help can take) in a way that can be evaluated.

 

(3...) The voting system SHALL provide a means for the voter to get help directly from the system at any time during the voting session.

 

- Requirement that poses the interesting question of "composed" targets. In the following case: how should a TA for assessing the fulfillment of "Voting systems" to this requirement below, relate to the subtargets that "EMS" and "vote-capture device"  are? Shouldn't we say that a voting system that contains exactly one EMS instance which is not conforming to the EMS requirements, is itself NOT conforming? (since a non-conforming EMS actually might not be considered to be an EMS?)  And if it contains one conforming EMS and one non-conforming EMS, shouldn't the voting system be considered conforming to the requirement below?

 

(6.1) Voting systems SHALL contain at least one EMS and at least one vote-capture device.

leading to similar to the notion

 

 

Jacques

---

From: Lynne Rosenthal [mailto:lynne.rosenthal@nist.gov]
Sent: Thursday, December 06, 2007 4:07 PM
To: Durand, Jacques R.
Subject: RE: [tag] requirements = test assertion

This is the standard I spoke about today, where we wrote the requirements as testable statements – so basically, the requirement = a test assertion.

 

http://www.eac.gov/vvsg/part1

 

Look at chapters 3-7.  chapter 2 explains conformance which is in terms of conforming to a ‘class of voting system’.

 

lynne

 

---

From: Durand, Jacques R. [mailto:JDurand@us.fujitsu.com]
Sent: Thursday, December 06, 2007 6:59 PM
To: tag@lists.oasis-open.org
Subject: [tag] a quick summary of "anatomy of TA" discussions so far

 

Quick summary of current status of discussions on Anatomy of a TA: (this is not exhaustive of what has been said this afternoon!)

 

- The TA "model" (what are the elements of a TA, whether optional or mandatory,

and what they mean) is to be distinguished from the representation structure(s) of

a TA. In other words, saying that the "target" is a mandatory element does not mean

that it must be defined explicitly in every TA representation: as long as a TA

representation provides means to access this element (URL, inheritance, context info...)

then it is still a representation compliant with our model.

 

- The assertion target identifies actually a class of items to which the TA

applies. There is a rough agreement that the "target" is more than just a tag

or classification means. It helps identify the [part of an] implementation that

will be under test, for a test case derived from this TA.

 

- As a normative specification statement may concern different targets depending

on the user perspective, it is OK to arbitrarily use any of these targets,

or even several.

 

- The target of a TA may be (and in general is) a finer grained item than the

target of a conformance clause, which usually has an intuitive user-level

business function. The latter corresponds to current definition of "implementation"

(product / service / document...)

 

- The ID of a TA must be globally unique, and should reflect spec ID and version #.

 

- Testability is the prime quality a test assertion writer should be concerned with.

If the addressed normative statement is unambiguously testable, it is sufficient

to precisely refer to it, in order to have a valid TA (no need to write a separate

"assertion expression" or so.)

 

- Optionality in a specification (SHOULD, MAY...) is NOT an obstacle to testability,

as the optional quality does not affect the testability of the feature described,

if described precisely enough.

The test assertion may however need to be worded as [for derived test cases] to

verify that "WHEN <the feature is used > THEN <it MUST be used accordingly to spec>".

Normally, any optional feature can be interpreted that way and *in the case it

is used* is subject to clear normative statements of mandatory character,

in a well written specification. The WHEN part could be defined in a qualifier /

precondition part of the TA. To be more specific: when a spec says:

"the widget X MAY be present in the package. The widget X MUST satisfy A,B and C."

The first sentence in itself is not material for a TA. The two sentences together

are. Should the "package" be chosen as target, the presence of X is a precondition

or qualifier of the TA.

 

- "pre-requisites" of a TA, as references to other TAs that a target is supposed

to satisfy for this TA to apply, are just some particular kind of qualifier, i.e.

are part of the "qualifier" (or precondition) for this TA.