OK let us try one more time...
In order to see if we can say
"something more" in the TA def and how far, I would start from this
statement in the charter draft - which may be clumsy in its expression,
but convey some consensus I believe:
" Within scope [of the
TC] is a characterization of the test environment or test harness
assumed by the test assertions, stating the expected properties and
mode of operation required by the verification of the TAs. Such
characterization may be seen as some of the requirements for a real
test harness intended to process test cases based on these TAs."
This assumes that TAs explicitly
(or implicitly) (1) refer to an implementation and its behavior, (2)
assume a test harness that does not need be precisely described, but
needs identify its relation to the implementation under test (IUT) i.e.
the kind of control it can exercise on the IUT, and/or the kind of IUT
output that the harness must be able to observe.
So for example a TA could say "if
a message M1 with content A but no X is sent out by implementation
[implicitly sent to test harness], and if the implementation had
previously sent out a message MC [captured by test harness] that shows
it was operating under condition C, then the implementation fails to
satisfy the spec requirement."
NOTE: test harness is broadly
understood here as operational "context" for the IUT: it may include a
mix of test software components (e.g. a "sniffer') and application
components - e.g. a partner system.
So here, should the TA also take
care of telling how to cause the IUT to send M1 out ? (e.g. as a
response to a message M0 sent by the test harness)? I would argue it is
at the discretion of the TA writer, depending on how "testable" the TA
would be. Obviously, in some deployed system you may not have the
liberty to send what you want to the IUT, but the IUT will be
stimulated enough to exhibit the behavior you want to verify. But if
the TA is written with conformance in mind, you may assume greater
control from the test harness and the TA may be more explicit on how to
prod the IUT.
Now lets assume that the
operation conditions C require the IUT to encrypt the message, and that
the only sure way to observe this encryption, is the content of some
security header, that would unfortunately not be visible in a test
harness that only "sits" on top of an opaque security infrastructure
that strips away this header before handing the message to higher-level
components. So we face an issue here: either we assume a test
component that "sniffs" the raw message onthe wire and see the header,
but cannot check the encrypted content A+X, or we assume a component
that checks A+X in clear but can't get the header anymore...
In that case the TA could
suggest a sequence of operations - controls and observations - on the
message (i.e. capture "on the wire", then decryption, then checking
A+X), which in turn assume a particular test harness, e.g. stating:
"if the transport envelope
of a message M1 sent by the IUT contains a security header showing the
use of encryption for its payload, then after decryption the payload
must contain A+X."
So I would say a TA can go as
far as the operations / interactions - as defined in the test harness
referred by the TA - allow to express the "how to verify". To the
extent the test harness referred by TAs remains abstract enough, the TA
expression would not degenerate into a test case...
So I would propose to add to the
TA def something like:
"A TA may
refer to an abstract test harness architecture that characterizes test
components in terms of their interaction with the implementation under
test. A TA may specify which interactions between implementation and
test harness are expected to be observed or exercised."
Comments?
Jacques
Jacques,
I generally
agree with your charter. I guess the difficulty is at qualifying the
‘how to verify’ in terms of how deep it goes. In your example, the ‘how
to qualify’ only covers the process. However, one could also looks
for/interpret that the TA also should indicate how to verify that the
message M does not use A or when the Implementation is under condition
C. It would be better if we could qualify the definition further to
make it more obvious how deep the TA has to go.
- Serm
Jacques Durand wrote:
>In any case and to avoid
controversy at charter level, I would propose to replace the def of TA
as:
>"A test assertion (TA),
also sometimes defined as test specification, is understood in this
charter with the following general meaning: it describes the expected
output or behavior for an implementation under some specific operation
conditions, in a way that can be measured or tested. Each test
assertion is an independent, complete, testable statement for
requirements in the specification. Test assertions are generally
different from test cases, which are more detailed and contingent to a
concrete test framework: TAs are the basis to write test cases, and
relate the latter to the narrative of the target specification."
>Anyone disagrees?
I like this
definition because it avoids the discussion of preconditions vs.
stimulus, relying only on the very bland "operation conditions" to
describe the situation in which a behavior may be expected. It also
avoids stating whether that behavior is always the end result of a test
case (that could be evaluated as pass/fail/whatever) or might sometimes
be a condition that can be used for compounding TAs. I suspect that the
TC would want to discuss those issues deeply, so it's best if the
charter does not presume any design philosophy at that level.
In other words, the
TC would probably want to have lively discussions about how many
different kinds of
contingencies and results should be defined. These discussions will
affect sequencing of the parts of a test and how compounding of TAs
might affect the sequence. Is a state model always needed? It's best if
the definition of a TA (and indeed, the whole charter) makes no
assumptions on these topics.
.................David
Marston