|
I think Serm
has pointed at something we have been debating for a while,
and for which
we have settle on a solution that could still evolve later.
In the
current design, there is indeed the notion of "master" test
suite,
from which we
can derive other test suites that respond to particular test profiles (e.g.
mapping to conformance profiles).
So a Test
Suite identifies a set of Test Cases, but this set is (today) reprsented
by the combination:
[master] test
suite doc +
test
requirements doc +
test profile
doc
as Mike points out..
So a "Test Suite " is actually more than the Test Suite
document.
(there
is also other material in a test suite, regarding context of execution, manual,
etc.)
The TD
currently needs all these docs to execute a test suite:
takes the
test profile doc
which points to a subset of test requirements, which in turn are referred
to
by a subset of Test Cases. That is how the Test
Cases to be executed for a test suite are
identified.
Now, you
could define your own Test Suite, but in the current design it has to be
accompained by a
test
requirements doc and a test profile doc.
The idea
behind the current design, is that a test suite is rooted in a "master" test
suite doc,
so that there
is no duplication of test cases in derived test suties.
if changes or
corrections need be made to a test case,
there is a
single place where this change needs be made: it will automatically reflect in
all derived profiles.
The drawback
of defining test profil;es (or derived test suites) as a subset of Test
Requirements,
is that some
profiles may not cut a clean partition over test requirements: a new profile
may
for example
require a same test req to be tested only partially, or in some
way.
It could then
be that two test profiles refer to exactly the same set of test requirements,
but using different test cases.
We can't do
that today , but it could be done in the future by defining test suite
as:
Test Suite =
[master] test
cases doc (just a renaming of "test suite doc" previous) +
test profile
doc (which would refer directly to a subset of test cases, not to test
reqs)
We will
record proposed updates in an issue list, for future versions. That one might be
a candidate,
let us your
opinion...
Regards,
jacques
Serm,
Please see my comments
below
Regards,
Mike
----- Original Message -----
Sent: Monday, October 04, 2004 2:59
PM
Subject: Re: [ebxml-iic] Test Framework
Question
Thanks Mike,
2 questions. What is the conceptual definition
of the TS?
[MIKE]
"The combination of test software, test
documentation and test procedurese that check an implementation for
conformance to a standard."
This definition fits nicely with
how we describe a Conformance Test Suite in section 4.1 of Test Framework
V1.0 and in section 5.1 of Test Framework V1.1 :
A conformance test suite is composed
of:
-
One or more Test Profile documents (XML). Such
documents represent the level or profile of conformance to the
specification, as verified by this Test
Suite.
-
Design of a Test
Harness for the Candidate Implementation that is based on
components of the ebXML IIC Test Framework.
-
A Test
Requirements document. This document contains a list of conformance
test assertions that are associated with the test profile to be
tested.
-
An annotation
of the target specification, that indicates the degree of Specification
Coverage for each specification feature or section, that this set of Test
Requirements provides.
-
A Test
Suite document. This document
implements the Test Requirements, described using the Test Framework
material (XML mark-up, etc.)
I think what I am asking here is what are the
factors grouping TC together in a TS.
TC's in the IIC Framework are currently defined
in a single XML document instance, with each one assigned a unique ID.
The TCs including packaging tests, security tests, reliability tests,
multi-hop tests and others.
They are only "grouped" together in some
meaningful fasion at run-time, when a user selects a particular profile of
TRs they wish to test against. At that point, the Test Driver selects
TCs that "point-to" those particular
TRs and executes them.
Secondly, your last comment below says that
Test Driver (TD) searches through TS after receiving TP..... That is not
what I read in the TF doc. The TF doc says that the TD searchs for necessary
TCs. I have been wondering for so long about the role of TS in IIC. I
remembered one time talking with Jacques and got an answer that it is a
functionally related tests.
The Test Driver does search for TCs.. but it
looks for them in the "master" TS. So I guess the point is what do you
define a TS as? The superset of TCs from which you select, or
the functionally related subset of TC's executed at runtime.
I think the answer could be "both" , but the better one would be the
"subset" of TCs selected at run-time.
- serm
----- Original Message -----
Sent: Monday, October 04, 2004 12:08
PM
Subject: Re: [ebxml-iic] Test
Framework Question
Serm,
Regards,
Mike
----- Original Message -----
Sent: Monday, October 04, 2004
10:31 AM
Subject: Re: [ebxml-iic] Test
Framework Question
Thanks Mike.
I was reading Section 5-7 (TR, TP, and TC)
of version 1 dated May 03. I think here is why I was
confused.
Section 5 says that TR is test harness
independent. Section 6 says "A Test Case is the translation of a TR (or
a part of a TR), in an executable form, for a particular Test Harness".
[MIKE] - This is true. However, the
word "executable" has a slightly different meaning in Conformance
Testing semantics. "Executable" means just that, something that
can be executed by a Test Harness. It does not imply that it is
"compiled", using different languages. What form a Test Case is in
(binary or XML.. or other) is not relevant in conformance testing
jargon.
Due to my problem context, I might have too
concrete interpretation to the "Test Harness". I interpreted it as
perhaps different languages, for example, I may be writing the TC in
Java or in C++. That's why I was asking how the TD knows which TC to use
b/c it does say which Test Harness it is associated with (maybe that's
the configurationGroup element).
[MIKE] - The "native" Test Case format in
the IIC Test Framework is XML. It is assumed that an underlying
message enveloping and transport mechanism is in place in the Test
Harness to construct, send, receive and store and evaluate
messages by interpreting the XML instructions using the
appropriate underlying messaging API calls.
It seems like TC in IIC is more abstract
b/c it is descriptively defined using XML and then Test Harness
implemented in whatever languages reads this XML to execute the test. In
my problem context, a TC is a rule written in either schematron or
another rule script language. I think I understand better
now.
[MIKE] - Yes, your interpretation (above)
of a TC is different from the IIC Test Framework definition.
It sound like the schematron engine is the Test Harness, and the
rule is the TC.
I think I understand better
now.
One more question though, why do you have
Test Suite reference TCs instead of TRs. In my mind, TS is a functional
grouping of tests, hence I should point to TRs.
[MIKE] - The Test Suite is a collection (or
grouping) of TCs. Each TC points to a particular TR through its
"requirementReferenceId" attribute, so I think that we agree
there.
On the other hand, I see TP as arbitrary
grouping of tests, so it could point to TCs. I think I am missing a lot
things b/c IIC's TRs and TCs are much more complicated.
[MIKE] - TPs are (in the Test
Framework view), an arbitrary grouping of TRs that satisfy a particular
profile. The Test Harness would read that TR list, then
search through the TS, and execute all TC's that satisfy that list of
TRs, and therefore satisfy the testing requirements of the
TP.
Thank you,
Serm
----- Original Message -----
Sent: Monday, September 27, 2004
12:44 PM
Subject: Re: [ebxml-iic] Test
Framework Question
Serm,
What particular sectio
of the spec are you reading? The answer is: yes, one Test
Requirement may have multip;e Test Cases to verify that an
implemenatation satisfies that requirement. For instance,
a Test Requirement that states that all
ebXML extension elements MUST have a "soap:mustUnderstand" attribute
present. This may require multip;e Test Cases to verify that an
implementation does this.
(one Test Case for SyncReply, one Test
Case for Acknowledgment testing, one Test Case for MessageStatus
testing...etc...) So, each of those Test Cases would point
to that single Test Requirement, and ALL of them must
pass successfully for an implementation
to say that it satisfies that requirement.
A Test Driver identifies
which TCs satisfy a requirement by examining the TC's
"requirementReferenceID".. which is a pointer (via ID) to that
particular TR. A Test Driver then executes all TC's that "point"
to a particular TR.
Regards,
Mike
----- Original Message -----
Sent: Monday, September 27,
2004 11:24 AM
Subject: [ebxml-iic] Test
Framework Question
Jacques and Mike et.al,
I looked over the TF doc and have a
question. In the test material, Test Profile is an aggregation of
Test Requirements. Test Driver takes a Test Profile and
searches for associated Test Cases and execute them.
However, in one section the it is
describing that a TR may have multiple TCs associated as one may
have multiple ways of validating the TR. The question is how the
Test Driver can identify a specific TC for a TR that has multiple
TCs associated or I am missing something?
Thank you,
Serm
|