OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

tag-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [tag-discuss] RE: TA definition (formerly: surviving the winterbreak)

Your proposed addition:

"A TA may refer to an abstract test harness architecture that characterizes test components in terms of their interaction with the implementation under test. A TA may specify which interactions between implementation and test harness are expected to be observed or exercised."

helps to alleviate the concerns I expressed earlier about test assertions straying into the area of test requirements.


Durand, Jacques R. wrote:
OK let us try one more time...
In order to see if we can say "something more"  in the TA def and how far, I would start from this statement in the charter draft - which may be clumsy in its expression, but convey some consensus I believe:
" Within scope [of the TC] is a characterization of the test environment or test harness assumed by the test assertions, stating the expected properties and mode of operation required by the verification of the TAs. Such characterization may be seen as some of the requirements for a real test harness intended to process test cases based on these TAs."
This assumes that TAs explicitly (or implicitly) (1) refer to an implementation and its behavior, (2) assume a test harness that does not need be precisely described, but needs identify its relation to the implementation under test (IUT) i.e. the kind of  control it can exercise on the IUT, and/or the kind of IUT output that the harness must be able to observe.
So for example a TA could say "if a message M1 with content A but no X is sent out by implementation [implicitly sent to test harness], and if the implementation had previously sent out a message MC [captured by test harness] that shows it was operating under condition C, then the implementation fails to satisfy the spec requirement." 
NOTE: test harness is broadly understood here as  operational "context" for the IUT: it may include a mix of test software components (e.g. a "sniffer') and application components - e.g. a partner system.
So here, should the TA also take care of telling how to cause the IUT to send M1 out ? (e.g. as a response to a message M0 sent by the test harness)? I would argue it is at the discretion of the TA writer, depending on how "testable" the TA would be.  Obviously, in some deployed system you may not have the liberty to send what you want to the IUT, but the IUT will be stimulated enough to exhibit the behavior you want to verify. But if the TA is written with conformance in mind, you may assume greater control from the test harness and the TA may be more explicit on how to prod the IUT.
Now lets assume that the operation conditions C require the IUT to encrypt the message, and that the only sure way to observe this encryption, is the content of some security header, that would unfortunately not be visible in a test harness that only "sits" on top of an opaque security infrastructure that strips away this header before handing the message to higher-level components.  So we face an issue here: either we assume a test component that "sniffs" the raw message onthe wire and see the header, but cannot check the encrypted content A+X, or we assume a component that checks A+X in clear but can't get the header anymore...
In that case the TA could suggest a sequence of operations - controls and observations - on the message (i.e. capture "on the wire", then decryption, then checking A+X), which in turn assume a particular test harness, e.g. stating:
"if the transport envelope of a message M1 sent by the IUT contains a security header showing the use of encryption for its payload, then after decryption the payload must contain A+X."
So I would say a TA can go as far as the operations / interactions - as defined in the test harness referred by the TA - allow to express the "how to verify". To the extent the test harness referred by TAs remains abstract enough, the TA expression would not degenerate into a test case...
So I would propose to add to the TA def something like:
"A TA may refer to an abstract test harness architecture that characterizes test components in terms of their interaction with the implementation under test. A TA may specify which interactions between implementation and test harness are expected to be observed or exercised."

From: Serm Kulvatunyou [mailto:serm@nist.gov]
Sent: Thursday, January 18, 2007 1:58 PM
To: tag-discuss@lists.oasis-open.org
Subject: RE: [tag-discuss] RE: TA definition (formerly: surviving the winter break)



I generally agree with your charter. I guess the difficulty is at qualifying the ‘how to verify’ in terms of how deep it goes. In your example, the ‘how to qualify’ only covers the process. However, one could also looks for/interpret that the TA also should indicate how to verify that the message M does not use A or when the Implementation is under condition C. It would be better if we could qualify the definition further to make it more obvious how deep the TA has to go.


- Serm


From: david_marston@us.ibm.com [mailto:david_marston@us.ibm.com]
Sent: Tuesday, January 16, 2007 1:41 PM
To: tag-discuss@lists.oasis-open.org
Subject: [tag-discuss] RE: TA definition (formerly: surviving the winter break)


Jacques Durand wrote:
>In any case and to avoid controversy at charter level, I would propose to replace the def of TA as:

A test assertion (TA), also sometimes defined as test specification, is understood in this charter with the following general meaning: it describes the expected output or behavior for an implementation under some specific operation conditions, in a way that can be measured or tested. Each test assertion is an independent, complete, testable statement for requirements in the specification. Test assertions are generally different from test cases, which are more detailed and contingent to a concrete test framework: TAs are the basis to write test cases, and relate the latter to the narrative of the target specification."

>Anyone disagrees?

I like this definition because it avoids the discussion of preconditions vs. stimulus, relying only on the very bland "operation conditions" to describe the situation in which a behavior may be expected. It also avoids stating whether that behavior is always the end result of a test case (that could be evaluated as pass/fail/whatever) or might sometimes be a condition that can be used for compounding TAs. I suspect that the TC would want to discuss those issues deeply, so it's best if the charter does not presume any design philosophy at that level.

In other words, the TC would probably want to have lively discussions about how many different kinds of contingencies and results should be defined. These discussions will affect sequencing of the parts of a test and how compounding of TAs might affect the sequence. Is a state model always needed? It's best if the definition of a TA (and indeed, the whole charter) makes no assumptions on these topics.
.................David Marston

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]