IIC minutes, and more

[Jacques Durand <JDurand@fsw.fujitsu.com>]

Title: IIC minutes, and more

All:

Minutes of two last meetings attached.
Also, here is a proposed set of definitions for Test Reqs / Abstract test suite / concrete test suite,
as discussed this morning. Candidate for  inclusion in the ebMS conf suite.

Jacques

<<IIC_August_18_03_Minutes.txt>> <<IIC_August_25_03_Minutes.txt>>

1. Test Requirements (also called Test Assertions in literature)
-----------------------------------------------------------------------------------------
(See Test Framework Section 5.)
A statement more formal than the spec narrative about conformance requirements, with more structure.
The Test req identifies more precisely the material to be tested (e.g. the message content, the state of the
implentation under test, or some other specific behavior of the impl under test: message sending, timing.
It also identifies formally under which conditions the requirement applies.
When creating test requirements, no assumption or restriction is made based on testing methodology (e.g. black box).
Authors of a specification should be able to draft test requirements without knowledge of the test environment
that will verify these.

2. Abstract test Cases:
---------------------------------
An abstract test case more precisely verifies a test requirement, or parts of it (e.g. by defining a subset of
situations where the test requirement applies.).
It defines detailed test material (e.g. message content, or test conditions), defines test steps.
It is precise enough so that the testing could be executed by an agent (human or processor) and repeated with similar results.

It is more restricted than a test requirement, because focusing (1) on the observable behavior (generally black-box),
and (2) on the available controls for the implementation under test.
For example, for ebMS, (1) is primarily the messages generated on the wire, the message data passed to the application,
plus errors and notifications to application. (2) is limited to the following material and actions: input (received) messages,

configuration with CPA data, MSH shutting down / starting. Any test requirement involving some state or material for which the

original specification does not describe any standard input/output, interface or control, cannot be expressed as is by
an abstract test case, as it would be at the discretion of each implementation.
This is the case about requirements about the state of the persistence store that is supposed to be part of an
MSH implementation: persistence of a message cannot be directly observed, only its effect on the overall behavior
of the MSH (e.g. resending the messsage after a shutdown/recovery) can be observed in an abstract test case.

3. Concrete test cases:
---------------------------------
A concrete test case uses and assumes an instance of the Test Framework, i.e. a specific test harness.
It is a translation of an abstract test case into the material specified by this Test Framework version.
Therefore, depending on the version of the Test Framework that is referred to, the same abstract test case can have
different concrete expressions. Some test harness restrictions may cause an abstract test case to not be translatable
into a concrete test case: e.g. for ebMS and V1.0 of the Test Framework, inability to generate some kind of application input,

and inability to carry out shutdown and restart in an automated way.
In the case of ebMS test suites, as the Test Framework is not normative (its use is not required), the concrete test suite notation

is not normative for the ebMS conformance test suite.

IIC Conference Call Minutes: Monday, August 18, 2003
 
Attendance:


Monica Martin (Sun)
Serm Kulvatunyou (NIST)
Mike Kass (NIST)
Jacques Durand (Fujitsu)

excused: 
Pete Wenzel (SeeBeyond)
Steve Yung (Sun)
Tim Sakach (DC)


Minutes taker: Jacques Durand

CALL DATE: AUG-18-2003 (Monday)
CALL TIME: 02:00 PM EASTERN TIME
DURATION: 1 hr
LEADER: MR MICHAEL KASS 
USA Toll Free Number: 888-810-5904
USA Toll Number: +1-773-756-4804
PASSCODE: 18586 

The agenda will be: 
1. Status of MS conformance test suite spec. (Mike Kass editor) 
- Gal assessment of current draft of MS conf test suite. Overview of remaining issues. 
- test requirements and their test coverage: comments on Pete/Monica/Mike doc. 
- ordering / partitioning based on config/CPA? (how to avoid too many reconfigs) 
- conformance profiles or levels (detailed content?)

2. BPSS test update (Serm K., Monica M.): 
- BSI testing definition. 
- what cooperation/feedback with/to the BPSS team (emphasis on testing operations). 
- The "BCF" concept (business collaboration Framework) .. 

3. Implementers corner, and PR: 
- Drake Certivo test driver status. 
- IIC members demo (Test Framework, test suites) at XML 2003, 7-12 December 

4. RegRep test requirements (compiling their test assertions) guidelines 

 
 
-----

1. Status of MS conformance test suite spec. (Mike Kass editor) 

- Profiles docs: point to functional "groups" of test reqs, as created in test reqs doc.
Such groups will not need to be broken for conf profiles.
- CPA grouping: test cases already sufficiently grouped based on CPA, Mike said.
- We'll review the "not testable" test reqs: there seems to be too many of them.
- Pete pointed that several so -called "app-level tests" can in fact be set/simulated
by the "initiator" action. E.g. req #29 (must understand) is not app level,
also the tests on COntentID payload.
- Jacques mentioned the RefToMessageId issue, if Test Driver needs to set it up.
(will need to use an intermediate variable). Will check if that is needed
in the current conf test suite.
- This will be a candidate upgrade for thenext version of the Test Framework.

2. BPSS update.

- Serm to get feedback from the BPSS team - for now, busy with Autotech demo.
- Comments needed on "ebXML Lite" from Covisint. Seems to have serious problems
of compatibility / interoperability... Monica consolidating comments.

3. Implementers corner, and PR: 
- IIC demo (Test Framework, test suites) at XML 2003, 7-12 December , on the agenda.
Drake Certivo possible candidate, NIST/POSTECH as well.

4. RegRep conformance test requirements (compiling their test assertions) guidelines .
- Mike and Jacques answering to Farrukh from RR TC. First phase is to get the TC come up
with a set of test requirements. IIC to help with this.


IIC Conference Call Minutes: Monday, August 25, 2003
 
Attendance:


Monica Martin (Sun)
Tim Sakach (DrakeCertivo)
Mike Kass (NIST)
Jacques Durand (Fujitsu)

excused: 
Pete Wenzel (SeeBeyond)
Steve Yung (Sun)
Serm Kulvatunyou (NIST)

Minutes taker: Jacques Durand

Time: Monday August 25, 11am PT
Host: Fujitsu 
Toll Free - : 1-800-251-6413 
Toll - : 1-913-905-1400 
Participant code: 598136 

Agenda: 

1. Status of MS conformance test suite spec. (Mike Kass editor) 
- Status of current draft of MS conf test suite. Overview of remaining issues.
- test requirements and their test coverage: comments on Pete/Monica/Mike doc.
- ordering / partitioning based on config/CPA? (how to avoid too many reconfigs)
- conformance profiles or levels 

2. BPSS test update (Serm K., Monica M.): 
- BSI testing.
- what cooperation/feedback with/to the BPSS team (emphasis on testing operations).
- The "BCF" concept (business collaboration Framework) ..

3. Implementers corner, and PR: 
- Drake Certivo test driver status. 
- IIC members demo (Test Framework, test suites) at XML 2003, 7-12 December

4. RegRep test requirements (compiling their test assertions) guidelines
 
 
-----

1. Status of MS conformance test suite spec. (Mike Kass editor) 

- Mike Kass factored in commetns from everyone in V0.91. 
- the problem of "app-level" test cases. Pete advised to use "Initiator" action
for many of these.
- should we describe abstract test cases even for  cases we know won't be testable
with the Test Framework V1.0? agreement that yes: the concrete test suite will not
be normative, and may evolve with the capability of the test framework.
So the abstract test suite should not depend on this. E.g. test framework cannot
automate shutdown/restart, for recovery tests. Yet that should be part of the
conformance test suite.
- Tim mentions sthat for non-testbale features, DC relies on an "honor system",
where the candidate claims that it has implemented the features / req.
- Jacques will draft a definition of what test reqs / abstact t.cases / concrete t.cases
are, especially whenderived from each other.
- Mike said that the distinction abstract/concrete test cases is not unusual in test
methodologies.
- Mike: overall, suite lot closer to completion. Main renmaining work: concrete cases
going through the details of the cases, verifying the XPath expressions. Also, more "Initiator"
actions need to be described, e.g. with proper data.
- issue with Conv ID on the "Initiators" test steps? (e.g. would test driver be able to filter 
"noise" from previous test cases, filtering out irrelevant messages?) Jacques said
that if test driver can clearly associate each test case with one or maybe two conv ids,
and these are unique over the test suite, we are OK. Safe to assume this.
- Plan to vote ebMS conf test suite TC spec in September.

2. BPSS test update .
- Serm asked for comments from the TC, waiting. Possibly wait for 1.1  draft.
Is busy on Autotech demo.
- Monica: BCF unrelated to ebXML so far.
- Tim: in search of a better methodology, for crafting mode generally test suites (DC working
on a business process suite, focus on OAGIS BODs.) No general methodology, Mike said.


3. Implementers corner, and PR
- Tim: DC expects methodology developed for ebMS condormance test suite, be reusable
for other conformance test suites (e.g. bus process level).
- DC implementing ebMS conformance. Considering Dec (Philly) demo.
- Tim will provide some feedback on ebMS conf test suite by mid September.


4. RegRep test requirements (compiling their test assertions) guidelines
- idea is that even though we focus now on ebMS, we should not wait to
get theRR TC clarify its test reqs: thats useful work to be done mostly by them.
- Farrukh (RR TC) happy with responses from Mike, on (staged approach on
a minimal conf profile, SOAP binding, yet both LCM and QM functional areas.)
- estimate in time seems reasonable for Farrukh: 
o "Marking up" the ebRS document to identify testing requirements - 4 weeks
o To create a Conformance Test Suite Specification - (integrating test
requirements, test cases and supporting documentation) - 1 month
o To create a "full blown" ebRS Conformance Test Suite 12 months
.