[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: IIC minutes, and more
All:
Minutes of two last meetings attached.
Also, here is a proposed set of definitions for Test Reqs / Abstract test suite / concrete test suite,
as discussed this morning. Candidate for inclusion in the ebMS conf suite.
Jacques
<<IIC_August_18_03_Minutes.txt>> <<IIC_August_25_03_Minutes.txt>>
1. Test Requirements (also called Test Assertions in literature)
-----------------------------------------------------------------------------------------
(See Test Framework Section 5.)
A statement more formal than the spec narrative about conformance requirements, with more structure.
The Test req identifies more precisely the material to be tested (e.g. the message content, the state of the
implentation under test, or some other specific behavior of the impl under test: message sending, timing.
It also identifies formally under which conditions the requirement applies.
When creating test requirements, no assumption or restriction is made based on testing methodology (e.g. black box).
Authors of a specification should be able to draft test requirements without knowledge of the test environment
that will verify these.
2. Abstract test Cases:
---------------------------------
An abstract test case more precisely verifies a test requirement, or parts of it (e.g. by defining a subset of
situations where the test requirement applies.).
It defines detailed test material (e.g. message content, or test conditions), defines test steps.
It is precise enough so that the testing could be executed by an agent (human or processor) and repeated with similar results.
It is more restricted than a test requirement, because focusing (1) on the observable behavior (generally black-box),
and (2) on the available controls for the implementation under test.
For example, for ebMS, (1) is primarily the messages generated on the wire, the message data passed to the application,
plus errors and notifications to application. (2) is limited to the following material and actions: input (received) messages,
configuration with CPA data, MSH shutting down / starting. Any test requirement involving some state or material for which the
original specification does not describe any standard input/output, interface or control, cannot be expressed as is by
an abstract test case, as it would be at the discretion of each implementation.
This is the case about requirements about the state of the persistence store that is supposed to be part of an
MSH implementation: persistence of a message cannot be directly observed, only its effect on the overall behavior
of the MSH (e.g. resending the messsage after a shutdown/recovery) can be observed in an abstract test case.
3. Concrete test cases:
---------------------------------
A concrete test case uses and assumes an instance of the Test Framework, i.e. a specific test harness.
It is a translation of an abstract test case into the material specified by this Test Framework version.
Therefore, depending on the version of the Test Framework that is referred to, the same abstract test case can have
different concrete expressions. Some test harness restrictions may cause an abstract test case to not be translatable
into a concrete test case: e.g. for ebMS and V1.0 of the Test Framework, inability to generate some kind of application input,
and inability to carry out shutdown and restart in an automated way.
In the case of ebMS test suites, as the Test Framework is not normative (its use is not required), the concrete test suite notation
is not normative for the ebMS conformance test suite.
IIC Conference Call Minutes: Monday, August 18, 2003 Attendance: Monica Martin (Sun) Serm Kulvatunyou (NIST) Mike Kass (NIST) Jacques Durand (Fujitsu) excused: Pete Wenzel (SeeBeyond) Steve Yung (Sun) Tim Sakach (DC) Minutes taker: Jacques Durand CALL DATE: AUG-18-2003 (Monday) CALL TIME: 02:00 PM EASTERN TIME DURATION: 1 hr LEADER: MR MICHAEL KASS USA Toll Free Number: 888-810-5904 USA Toll Number: +1-773-756-4804 PASSCODE: 18586 The agenda will be: 1. Status of MS conformance test suite spec. (Mike Kass editor) - Gal assessment of current draft of MS conf test suite. Overview of remaining issues. - test requirements and their test coverage: comments on Pete/Monica/Mike doc. - ordering / partitioning based on config/CPA? (how to avoid too many reconfigs) - conformance profiles or levels (detailed content?) 2. BPSS test update (Serm K., Monica M.): - BSI testing definition. - what cooperation/feedback with/to the BPSS team (emphasis on testing operations). - The "BCF" concept (business collaboration Framework) .. 3. Implementers corner, and PR: - Drake Certivo test driver status. - IIC members demo (Test Framework, test suites) at XML 2003, 7-12 December 4. RegRep test requirements (compiling their test assertions) guidelines ----- 1. Status of MS conformance test suite spec. (Mike Kass editor) - Profiles docs: point to functional "groups" of test reqs, as created in test reqs doc. Such groups will not need to be broken for conf profiles. - CPA grouping: test cases already sufficiently grouped based on CPA, Mike said. - We'll review the "not testable" test reqs: there seems to be too many of them. - Pete pointed that several so -called "app-level tests" can in fact be set/simulated by the "initiator" action. E.g. req #29 (must understand) is not app level, also the tests on COntentID payload. - Jacques mentioned the RefToMessageId issue, if Test Driver needs to set it up. (will need to use an intermediate variable). Will check if that is needed in the current conf test suite. - This will be a candidate upgrade for thenext version of the Test Framework. 2. BPSS update. - Serm to get feedback from the BPSS team - for now, busy with Autotech demo. - Comments needed on "ebXML Lite" from Covisint. Seems to have serious problems of compatibility / interoperability... Monica consolidating comments. 3. Implementers corner, and PR: - IIC demo (Test Framework, test suites) at XML 2003, 7-12 December , on the agenda. Drake Certivo possible candidate, NIST/POSTECH as well. 4. RegRep conformance test requirements (compiling their test assertions) guidelines . - Mike and Jacques answering to Farrukh from RR TC. First phase is to get the TC come up with a set of test requirements. IIC to help with this.
IIC Conference Call Minutes: Monday, August 25, 2003 Attendance: Monica Martin (Sun) Tim Sakach (DrakeCertivo) Mike Kass (NIST) Jacques Durand (Fujitsu) excused: Pete Wenzel (SeeBeyond) Steve Yung (Sun) Serm Kulvatunyou (NIST) Minutes taker: Jacques Durand Time: Monday August 25, 11am PT Host: Fujitsu Toll Free - : 1-800-251-6413 Toll - : 1-913-905-1400 Participant code: 598136 Agenda: 1. Status of MS conformance test suite spec. (Mike Kass editor) - Status of current draft of MS conf test suite. Overview of remaining issues. - test requirements and their test coverage: comments on Pete/Monica/Mike doc. - ordering / partitioning based on config/CPA? (how to avoid too many reconfigs) - conformance profiles or levels 2. BPSS test update (Serm K., Monica M.): - BSI testing. - what cooperation/feedback with/to the BPSS team (emphasis on testing operations). - The "BCF" concept (business collaboration Framework) .. 3. Implementers corner, and PR: - Drake Certivo test driver status. - IIC members demo (Test Framework, test suites) at XML 2003, 7-12 December 4. RegRep test requirements (compiling their test assertions) guidelines ----- 1. Status of MS conformance test suite spec. (Mike Kass editor) - Mike Kass factored in commetns from everyone in V0.91. - the problem of "app-level" test cases. Pete advised to use "Initiator" action for many of these. - should we describe abstract test cases even for cases we know won't be testable with the Test Framework V1.0? agreement that yes: the concrete test suite will not be normative, and may evolve with the capability of the test framework. So the abstract test suite should not depend on this. E.g. test framework cannot automate shutdown/restart, for recovery tests. Yet that should be part of the conformance test suite. - Tim mentions sthat for non-testbale features, DC relies on an "honor system", where the candidate claims that it has implemented the features / req. - Jacques will draft a definition of what test reqs / abstact t.cases / concrete t.cases are, especially whenderived from each other. - Mike said that the distinction abstract/concrete test cases is not unusual in test methodologies. - Mike: overall, suite lot closer to completion. Main renmaining work: concrete cases going through the details of the cases, verifying the XPath expressions. Also, more "Initiator" actions need to be described, e.g. with proper data. - issue with Conv ID on the "Initiators" test steps? (e.g. would test driver be able to filter "noise" from previous test cases, filtering out irrelevant messages?) Jacques said that if test driver can clearly associate each test case with one or maybe two conv ids, and these are unique over the test suite, we are OK. Safe to assume this. - Plan to vote ebMS conf test suite TC spec in September. 2. BPSS test update . - Serm asked for comments from the TC, waiting. Possibly wait for 1.1 draft. Is busy on Autotech demo. - Monica: BCF unrelated to ebXML so far. - Tim: in search of a better methodology, for crafting mode generally test suites (DC working on a business process suite, focus on OAGIS BODs.) No general methodology, Mike said. 3. Implementers corner, and PR - Tim: DC expects methodology developed for ebMS condormance test suite, be reusable for other conformance test suites (e.g. bus process level). - DC implementing ebMS conformance. Considering Dec (Philly) demo. - Tim will provide some feedback on ebMS conf test suite by mid September. 4. RegRep test requirements (compiling their test assertions) guidelines - idea is that even though we focus now on ebMS, we should not wait to get theRR TC clarify its test reqs: thats useful work to be done mostly by them. - Farrukh (RR TC) happy with responses from Mike, on (staged approach on a minimal conf profile, SOAP binding, yet both LCM and QM functional areas.) - estimate in time seems reasonable for Farrukh: o "Marking up" the ebRS document to identify testing requirements - 4 weeks o To create a Conformance Test Suite Specification - (integrating test requirements, test cases and supporting documentation) - 1 month o To create a "full blown" ebRS Conformance Test Suite 12 months .
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]