Re: [ebxml-iic] IIC minutes, and more

[Monica Martin <monica.martin@sun.com>]

> 1. Test Requirements (also called Test Assertions in literature)
> ----------------------------------------------------------------------------------------- 
>
> (See Test Framework Section 5.)
> A statement more formal than the spec narrative about conformance 
> requirements, with more structure.
> The Test req identifies more precisely the material to be tested (e.g. 
> the message content, the state of the
> implentation under test, or some other specific behavior of the impl 
> under test: message sending, timing.
> It also identifies formally under which conditions the requirement 
> applies.
> When creating test requirements, no assumption or restriction is made 
> based on testing methodology (e.g. black box).
> Authors of a specification should be able to draft test requirements 
> without knowledge of the test environment
> that will verify these.
>
mm1: Suggest you indicate that the test requirement development is 
independent of the test environment,  Therefore test requirements
can be developed without explicit knowledge underlying infrastructure or 
environment (i.e. an implementation of the test environment
designed to support the framework).

> 2. Abstract test Cases:
> ---------------------------------
> An abstract test case more precisely verifies a test requirement, or 
> parts of it (e.g. by defining a subset of
> situations where the test requirement applies.).
> It defines detailed test material (e.g. message content, or test 
> conditions), defines test steps.
> It is precise enough so that the testing could be executed by an agent 
> (human or processor) and repeated with similar results.
>
> It is more restricted than a test requirement, because focusing (1) on 
> the observable behavior (generally black-box),
> and (2) on the available controls for the implementation under test.
> For example, for ebMS, (1) is primarily the messages generated on the 
> wire, the message data passed to the application,
> plus errors and notifications to application. (2) is limited to the 
> following material and actions: input (received) messages,
>
> configuration with CPA data, MSH shutting down / starting. Any test 
> requirement involving some state or material for which the
>
> original specification does not describe any standard input/output, 
> interface or control, cannot be expressed as is by
> an abstract test case, as it would be at the discretion of each 
> implementation.
>
mm1: Perhaps we should indicate that discretion should be used when the 
reference specification does not explicitly or specifically
describe the standard input/output, interface or control or the 
mechanisms used for specific functions.  My comment: When an abstract test
cases may need to describe functions that include an implementation or 
design component outside of the test framework (or any of our 
'specifications') responsibility, it is a design choice what mechanisms 
are used.  The persistent store is a prime example.

>
> This is the case about requirements about the state of the persistence 
> store that is supposed to be part of an
> MSH implementation: persistence of a message cannot be directly 
> observed, only its effect on the overall behavior
> of the MSH (e.g. resending the messsage after a shutdown/recovery) can 
> be observed in an abstract test case.
>
>
> 3. Concrete test cases:
> ---------------------------------
> A concrete test case uses and assumes an instance of the Test 
> Framework, i.e. a specific test harness.
> It is a translation of an abstract test case into the material 
> specified by this Test Framework version.
>
mm1: I would say that the concrete test case is a detailed or explicit 
representation(s) of the abstract test case that is fully specified and in
line with the Test Framework.  It also, even if implicitly, provides the 
confidence that the test framework and the original/reference
specification provided enough clarity to define and execute against the 
test requirements in support of the specification requirements.

I would suggest we focus on what the framework can do, and why a test 
framework may or may not be the appropriate venue
to verify reference specification requirements.  If you look at the 
example below, you see that the application level interactions, 
interfaces, processing
fit into this category. I think these are parameters we place on the 
test framework, not exclusively its limitations.  Think (+) not (- - can't).

Thanks.

>
> Therefore, depending on the version of the Test Framework that is 
> referred to, the same abstract test case can have
> different concrete expressions. Some test harness restrictions may 
> cause an abstract test case to not be translatable
> into a concrete test case: e.g. for ebMS and V1.0 of the Test 
> Framework, inability to generate some kind of application input,
>
> and inability to carry out shutdown and restart in an automated way.
> In the case of ebMS test suites, as the Test Framework is not 
> normative (its use is not required), the concrete test suite notation
>
> is not normative for the ebMS conformance test suite.
>
>------------------------------------------------------------------------
>
>IIC Conference Call Minutes: Monday, August 18, 2003
> 
>Attendance:
>
>
>Monica Martin (Sun)
>Serm Kulvatunyou (NIST)
>Mike Kass (NIST)
>Jacques Durand (Fujitsu)
>
>excused: 
>Pete Wenzel (SeeBeyond)
>Steve Yung (Sun)
>Tim Sakach (DC)
>
>
>Minutes taker: Jacques Durand
>
>CALL DATE: AUG-18-2003 (Monday)
>CALL TIME: 02:00 PM EASTERN TIME
>DURATION: 1 hr
>LEADER: MR MICHAEL KASS 
>USA Toll Free Number: 888-810-5904
>USA Toll Number: +1-773-756-4804
>PASSCODE: 18586 
>
>The agenda will be: 
>1. Status of MS conformance test suite spec. (Mike Kass editor) 
>- Gal assessment of current draft of MS conf test suite. Overview of remaining issues. 
>- test requirements and their test coverage: comments on Pete/Monica/Mike doc. 
>- ordering / partitioning based on config/CPA? (how to avoid too many reconfigs) 
>- conformance profiles or levels (detailed content?)
>
>2. BPSS test update (Serm K., Monica M.): 
>- BSI testing definition. 
>- what cooperation/feedback with/to the BPSS team (emphasis on testing operations). 
>- The "BCF" concept (business collaboration Framework) .. 
>
>3. Implementers corner, and PR: 
>- Drake Certivo test driver status. 
>- IIC members demo (Test Framework, test suites) at XML 2003, 7-12 December 
>
>4. RegRep test requirements (compiling their test assertions) guidelines 
>
> 
> 
>-----
>
>1. Status of MS conformance test suite spec. (Mike Kass editor) 
>
>- Profiles docs: point to functional "groups" of test reqs, as created in test reqs doc.
>Such groups will not need to be broken for conf profiles.
>- CPA grouping: test cases already sufficiently grouped based on CPA, Mike said.
>- We'll review the "not testable" test reqs: there seems to be too many of them.
>- Pete pointed that several so -called "app-level tests" can in fact be set/simulated
>by the "initiator" action. E.g. req #29 (must understand) is not app level,
>also the tests on COntentID payload.
>- Jacques mentioned the RefToMessageId issue, if Test Driver needs to set it up.
>(will need to use an intermediate variable). Will check if that is needed
>in the current conf test suite.
>- This will be a candidate upgrade for thenext version of the Test Framework.
>
>2. BPSS update.
>
>- Serm to get feedback from the BPSS team - for now, busy with Autotech demo.
>- Comments needed on "ebXML Lite" from Covisint. Seems to have serious problems
>of compatibility / interoperability... Monica consolidating comments.
>
>3. Implementers corner, and PR: 
>- IIC demo (Test Framework, test suites) at XML 2003, 7-12 December , on the agenda.
>Drake Certivo possible candidate, NIST/POSTECH as well.
>
>4. RegRep conformance test requirements (compiling their test assertions) guidelines .
>- Mike and Jacques answering to Farrukh from RR TC. First phase is to get the TC come up
>with a set of test requirements. IIC to help with this.
>
>
>  
>
>------------------------------------------------------------------------
>
>IIC Conference Call Minutes: Monday, August 25, 2003
> 
>Attendance:
>
>
>Monica Martin (Sun)
>Tim Sakach (DrakeCertivo)
>Mike Kass (NIST)
>Jacques Durand (Fujitsu)
>
>excused: 
>Pete Wenzel (SeeBeyond)
>Steve Yung (Sun)
>Serm Kulvatunyou (NIST)
>
>Minutes taker: Jacques Durand
>
>Time: Monday August 25, 11am PT
>Host: Fujitsu 
>Toll Free - : 1-800-251-6413 
>Toll - : 1-913-905-1400 
>Participant code: 598136 
>
>Agenda: 
>
>1. Status of MS conformance test suite spec. (Mike Kass editor) 
>- Status of current draft of MS conf test suite. Overview of remaining issues.
>- test requirements and their test coverage: comments on Pete/Monica/Mike doc.
>- ordering / partitioning based on config/CPA? (how to avoid too many reconfigs)
>- conformance profiles or levels 
>
>2. BPSS test update (Serm K., Monica M.): 
>- BSI testing.
>- what cooperation/feedback with/to the BPSS team (emphasis on testing operations).
>- The "BCF" concept (business collaboration Framework) ..
>
>3. Implementers corner, and PR: 
>- Drake Certivo test driver status. 
>- IIC members demo (Test Framework, test suites) at XML 2003, 7-12 December
>
>4. RegRep test requirements (compiling their test assertions) guidelines
> 
> 
>-----
>
>1. Status of MS conformance test suite spec. (Mike Kass editor) 
>
>- Mike Kass factored in commetns from everyone in V0.91. 
>- the problem of "app-level" test cases. Pete advised to use "Initiator" action
>for many of these.
>- should we describe abstract test cases even for  cases we know won't be testable
>with the Test Framework V1.0? agreement that yes: the concrete test suite will not
>be normative, and may evolve with the capability of the test framework.
>So the abstract test suite should not depend on this. E.g. test framework cannot
>automate shutdown/restart, for recovery tests. Yet that should be part of the
>conformance test suite.
>- Tim mentions sthat for non-testbale features, DC relies on an "honor system",
>where the candidate claims that it has implemented the features / req.
>- Jacques will draft a definition of what test reqs / abstact t.cases / concrete t.cases
>are, especially whenderived from each other.
>- Mike said that the distinction abstract/concrete test cases is not unusual in test
>methodologies.
>- Mike: overall, suite lot closer to completion. Main renmaining work: concrete cases
>going through the details of the cases, verifying the XPath expressions. Also, more "Initiator"
>actions need to be described, e.g. with proper data.
>- issue with Conv ID on the "Initiators" test steps? (e.g. would test driver be able to filter 
>"noise" from previous test cases, filtering out irrelevant messages?) Jacques said
>that if test driver can clearly associate each test case with one or maybe two conv ids,
>and these are unique over the test suite, we are OK. Safe to assume this.
>- Plan to vote ebMS conf test suite TC spec in September.
>
>2. BPSS test update .
>- Serm asked for comments from the TC, waiting. Possibly wait for 1.1  draft.
>Is busy on Autotech demo.
>- Monica: BCF unrelated to ebXML so far.
>- Tim: in search of a better methodology, for crafting mode generally test suites (DC working
>on a business process suite, focus on OAGIS BODs.) No general methodology, Mike said.
>
>
>3. Implementers corner, and PR
>- Tim: DC expects methodology developed for ebMS condormance test suite, be reusable
>for other conformance test suites (e.g. bus process level).
>- DC implementing ebMS conformance. Considering Dec (Philly) demo.
>- Tim will provide some feedback on ebMS conf test suite by mid September.
>
>
>4. RegRep test requirements (compiling their test assertions) guidelines
>- idea is that even though we focus now on ebMS, we should not wait to
>get theRR TC clarify its test reqs: thats useful work to be done mostly by them.
>- Farrukh (RR TC) happy with responses from Mike, on (staged approach on
>a minimal conf profile, SOAP binding, yet both LCM and QM functional areas.)
>- estimate in time seems reasonable for Farrukh: 
>o "Marking up" the ebRS document to identify testing requirements - 4 weeks
>o To create a Conformance Test Suite Specification - (integrating test
>requirements, test cases and supporting documentation) - 1 month
>o To create a "full blown" ebRS Conformance Test Suite 12 months
>.
>
>
>
>  
>
>------------------------------------------------------------------------
>
>You may leave a Technical Committee at any time by visiting http://www.oasis-open.org/apps/org/workgroup/ebxml-iic/members/leave_workgroup.php
>