OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

ebxml-iic message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [ebxml-iic] Test Framework Question


Serm,
 
   Please see my comments below
 
 
Regards,
Mike
 
 
----- Original Message -----
Sent: Monday, October 04, 2004 2:59 PM
Subject: Re: [ebxml-iic] Test Framework Question

Thanks Mike,
 
2 questions. What is the conceptual definition of the TS?
 
[MIKE]
 
   The conceptual definition of a TS is quite large.. quoting the OASIS Glossary of Terms 
 
"The combination of test software, test documentation and test procedurese that check an implementation for conformance to a standard."
 
   This definition fits nicely with how we describe a Conformance Test Suite in section 4.1 of Test Framework V1.0 and in section 5.1 of Test Framework V1.1 :
 
 

A conformance test suite is composed of: 

  • One or more Test Profile documents (XML). Such documents represent the level or profile of conformance to the specification, as verified by this Test Suite.
  • Design of a Test Harness for the Candidate Implementation that is based on components of the ebXML IIC Test Framework.
  • A Test Requirements document. This document contains a list of conformance test assertions that are associated with the test profile to be tested.
  • An annotation of the target specification, that indicates the degree of Specification Coverage for each specification feature or section, that this set of Test Requirements provides.
  • A Test Suite document. This document implements the Test Requirements, described using the Test Framework material (XML mark-up, etc.)
 
 
I think what I am asking here is what are the factors grouping TC together in a TS.
 
TC's in the IIC Framework are currently defined in a single XML document instance, with each one assigned a unique ID.  The TCs including packaging tests, security tests, reliability tests, multi-hop tests and others.
They are only "grouped" together in some meaningful fasion at run-time, when a user selects a particular profile of TRs they wish to test against.  At that point, the Test Driver selects TCs that "point-to" those particular
TRs and executes them.
 
 
Secondly, your last comment below says that Test Driver (TD) searches through TS after receiving TP..... That is not what I read in the TF doc. The TF doc says that the TD searchs for necessary TCs. I have been wondering for so long about the role of TS in IIC. I remembered one time talking with Jacques and got an answer that it is a functionally related tests.
 
The Test Driver does search for TCs.. but it looks for them in the "master" TS.  So I guess the point is what do you define a TS as?  The superset of TCs from which you select, or the functionally related subset of TC's executed at runtime.  I think the answer could be "both" , but the better one would be the "subset" of TCs selected at run-time.
 
- serm
----- Original Message -----
Sent: Monday, October 04, 2004 12:08 PM
Subject: Re: [ebxml-iic] Test Framework Question

Serm,
 
   My comments below: (also note that we have version 1.1 available to IIC members at: http://www.oasis-open.org/apps/org/workgroup/ebxml-iic/  )  We are hoping to have a final vote on it soon. 
 
Regards,
Mike
 
----- Original Message -----
Sent: Monday, October 04, 2004 10:31 AM
Subject: Re: [ebxml-iic] Test Framework Question

Thanks Mike.
 
I was reading Section 5-7 (TR, TP, and TC) of version 1 dated May 03. I think here is why I was confused.
 
Section 5 says that TR is test harness independent. Section 6 says "A Test Case is the translation of a TR (or a part of a TR), in an executable form, for a particular Test Harness".
 
[MIKE] - This is true.  However, the word "executable" has a slightly different meaning in Conformance Testing semantics.  "Executable" means just that, something that can be executed by a Test Harness.  It does not imply that it is "compiled", using different languages. What form a Test Case is in (binary or XML.. or other) is not relevant in conformance testing jargon.
 
Due to my problem context, I might have too concrete interpretation to the "Test Harness". I interpreted it as perhaps different languages, for example, I may be writing the TC in Java or in C++. That's why I was asking how the TD knows which TC to use b/c it does say which Test Harness it is associated with (maybe that's the configurationGroup element).
 
[MIKE] - The "native" Test Case format in the IIC Test Framework is XML.  It is assumed that an underlying message enveloping and transport mechanism is in place in the Test Harness to construct, send, receive and store and evaluate  messages by interpreting the XML instructions using the appropriate underlying messaging API calls. 
 
It seems like TC in IIC is more abstract b/c it is descriptively defined using XML and then Test Harness implemented in whatever languages reads this XML to execute the test. In my problem context, a TC is a rule written in either schematron or another rule script language. I think I understand better now.
 
[MIKE] - Yes, your interpretation (above)  of a TC is different from the IIC Test Framework definition.  It sound like the schematron engine is the Test Harness, and the rule is the TC.
 
I think I understand better now.
 
One more question though, why do you have Test Suite reference TCs instead of TRs. In my mind, TS is a functional grouping of tests, hence I should point to TRs.
 
[MIKE] - The Test Suite is a collection (or grouping) of TCs.  Each TC points to a particular TR through its "requirementReferenceId" attribute, so I think that we agree there.
 
On the other hand, I see TP as arbitrary grouping of tests, so it could point to TCs. I think I am missing a lot things b/c IIC's TRs and TCs are much more complicated.
 
[MIKE] -   TPs are (in the Test Framework view), an arbitrary grouping of TRs that satisfy a particular profile.  The Test Harness would read that TR list,  then search through the TS, and execute all TC's that satisfy that list of TRs, and therefore satisfy the testing requirements of the TP.
 
 
Thank you,
Serm
----- Original Message -----
Sent: Monday, September 27, 2004 12:44 PM
Subject: Re: [ebxml-iic] Test Framework Question

Serm,
 
    What particular sectio of the spec are you reading?  The answer is:  yes, one Test Requirement may have multip;e Test Cases to verify that an implemenatation satisfies that requirement.  For instance,
a Test Requirement that states that all ebXML extension elements MUST have a "soap:mustUnderstand" attribute present.  This may require multip;e Test Cases to verify that an implementation does this.
(one Test Case for SyncReply, one Test Case for Acknowledgment testing, one Test Case for MessageStatus testing...etc...)   So, each of those Test Cases would point to that single Test Requirement, and ALL of them must
pass successfully for an implementation to say that it satisfies that requirement.
 
   A Test Driver identifies which TCs satisfy a requirement by examining the TC's "requirementReferenceID".. which is a pointer (via ID) to that particular TR.  A Test Driver then executes all TC's that "point" to a particular TR.
 
Regards,
Mike
 
  
----- Original Message -----
Sent: Monday, September 27, 2004 11:24 AM
Subject: [ebxml-iic] Test Framework Question

Jacques and Mike et.al,
 
I looked over the TF doc and have a question. In the test material, Test Profile is an aggregation of Test Requirements. Test Driver takes a Test Profile and searches for associated Test Cases and execute them.
 
However, in one section the it is describing that a TR may have multiple TCs associated as one may have multiple ways of validating the TR. The question is how the Test Driver can identify a specific TC for a TR that has multiple TCs associated or I am missing something?
 
Thank you,
Serm


[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]