[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: RE: [tag-discuss] RE: TA definition (formerly: surviving the winter break)
OK let us try one more time...
In order to see if we can say "something more" in the
TA def and how far, I would start from this statement in the charter draft -
which may be clumsy in its expression, but convey some consensus I
believe:
"
Within scope [of the TC] is a characterization of the test environment or test
harness assumed by the test assertions, stating the expected properties and mode
of operation required by the verification of the TAs. Such characterization may
be seen as some of the requirements for a real test harness intended to process
test cases based on these TAs."
This assumes that TAs explicitly (or implicitly) (1) refer
to an implementation and its behavior, (2) assume a test harness that does not
need be precisely described, but needs identify its relation to the
implementation under test (IUT) i.e. the kind of control it can exercise
on the IUT, and/or the kind of IUT output that the harness must be able to
observe.
So for example a TA could say "if a message M1 with
content A but no X is sent out by implementation [implicitly sent to test
harness], and if the implementation had previously sent out a message MC
[captured by test harness] that shows it was operating under condition
C, then the implementation fails to satisfy the spec
requirement."
NOTE: test harness is broadly understood here as
operational "context" for the IUT: it may include a mix of test software
components (e.g. a "sniffer') and application components - e.g. a partner
system.
So here, should the TA also take care of telling how to
cause the IUT to send M1 out ? (e.g. as a response to a message M0 sent by the
test harness)? I would argue it is at the discretion of the TA writer, depending
on how "testable" the TA would be. Obviously, in some deployed system you
may not have the liberty to send what you want to the IUT, but the IUT will be
stimulated enough to exhibit the behavior you want to verify. But if the TA is
written with conformance in mind, you may assume greater control from the test
harness and the TA may be more explicit on how to prod the
IUT.
Now lets assume that the operation conditions C require the
IUT to encrypt the message, and that the only sure way to observe this
encryption, is the content of some security header, that would unfortunately not
be visible in a test harness that only "sits" on top of an opaque security
infrastructure that strips away this header before handing the message to
higher-level components. So we face an issue here: either we assume a test
component that "sniffs" the raw message onthe wire and see the header, but
cannot check the encrypted content A+X, or we assume a component that checks A+X
in clear but can't get the header anymore...
In that case the TA could suggest a sequence of operations
- controls and observations - on the message (i.e. capture "on the wire",
then decryption, then checking A+X), which in turn assume a particular test
harness, e.g. stating:
"if the transport envelope of a message M1
sent by the IUT contains a security header showing the use of encryption for its
payload, then after decryption the payload must contain
A+X."
So I would say a TA can go as far as the operations /
interactions - as defined in the test harness referred by the TA - allow to
express the "how to verify". To the extent the test harness referred by TAs
remains abstract enough, the TA expression would not degenerate into a test
case...
So I would propose to add to the TA def something
like:
"A TA may refer to an
abstract test harness architecture that characterizes test components in
terms of their interaction with the implementation under test. A TA may specify
which interactions between implementation and test harness are expected to be
observed or exercised."
Comments?
Jacques
From: Serm Kulvatunyou [mailto:serm@nist.gov] Sent: Thursday, January 18, 2007 1:58 PM To: tag-discuss@lists.oasis-open.org Subject: RE: [tag-discuss] RE: TA definition (formerly: surviving the winter break) Jacques, I generally agree with
your charter. I guess the difficulty is at qualifying the ‘how to verify’ in
terms of how deep it goes. In your example, the ‘how to qualify’ only covers the
process. However, one could also looks for/interpret that the TA also should
indicate how to verify that the message M does not use A or when the
Implementation is under condition C. It would be better if we could qualify the
definition further to make it more obvious how deep the TA has to go.
-
Serm From:
david_marston@us.ibm.com [mailto:david_marston@us.ibm.com]
|
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]