Re: [ebxml-iic] IIC minutes, and more

["Michael Kass" <michael.kass@nist.gov>]

To all,

   My comments preceded by [MIKE]



----- Original Message ----- 
From: "Monica Martin" <monica.martin@sun.com>
To: "Jacques Durand" <JDurand@fsw.fujitsu.com>; "ebXML IIC - main list
(E-mail)" <ebxml-iic@lists.oasis-open.org>
Sent: Tuesday, August 26, 2003 9:50 AM
Subject: Re: [ebxml-iic] IIC minutes, and more


>
> > 1. Test Requirements (also called Test Assertions in literature)
>
> --------------------------------------------------------------------------
--------------- 
> >
> > (See Test Framework Section 5.)
> > A statement more formal than the spec narrative about conformance
> > requirements, with more structure.
> > The Test req identifies more precisely the material to be tested (e.g.
> > the message content, the state of the
> > implentation under test, or some other specific behavior of the impl
> > under test: message sending, timing.
> > It also identifies formally under which conditions the requirement
> > applies.
> > When creating test requirements, no assumption or restriction is made
> > based on testing methodology (e.g. black box).
> > Authors of a specification should be able to draft test requirements
> > without knowledge of the test environment
> > that will verify these.
> >
> mm1: Suggest you indicate that the test requirement development is
> independent of the test environment,  Therefore test requirements
> can be developed without explicit knowledge underlying infrastructure or
> environment (i.e. an implementation of the test environment
> designed to support the framework).

[MIKE] - I think that Jacques says that in his last sentence, and your
wording,
(a little more specific) shows that test requirements are independent of
the testing framework.  This will be a nice addition to the MS Conformance
Test Suite Spec.


>
> > 2. Abstract test Cases:
> > ---------------------------------
> > An abstract test case more precisely verifies a test requirement, or
> > parts of it (e.g. by defining a subset of
> > situations where the test requirement applies.).
> > It defines detailed test material (e.g. message content, or test
> > conditions), defines test steps.
> > It is precise enough so that the testing could be executed by an agent
> > (human or processor) and repeated with similar results.

[MIKE] - I Suggest      "It is precise enough so that test cases can be
created that
will yield a correct and repeatable test result."    As I read it right now,
it implies
that the Abstract Test Case is "executable".. and this is not the case.  It
is a
test "descriptor" only, and not necessarily precise enough that "testing
could be "executed"
by an agent (human or processor)".  That is the duty of the "concrete" test
cases,
not the abstract test cases.

> >
> > It is more restricted than a test requirement, because focusing (1) on
> > the observable behavior (generally black-box),
> > and (2) on the available controls for the implementation under test.
> > For example, for ebMS, (1) is primarily the messages generated on the
> > wire, the message data passed to the application,
> > plus errors and notifications to application. (2) is limited to the
> > following material and actions: input (received) messages,
> >
> > configuration with CPA data, MSH shutting down / starting. Any test
> > requirement involving some state or material for which the
> >
> > original specification does not describe any standard input/output,
> > interface or control, cannot be expressed as is by
> > an abstract test case, as it would be at the discretion of each
> > implementation.
> >
> mm1: Perhaps we should indicate that discretion should be used when the
> reference specification does not explicitly or specifically
> describe the standard input/output, interface or control or the
> mechanisms used for specific functions.  My comment: When an abstract test
> cases may need to describe functions that include an implementation or
> design component outside of the test framework (or any of our
> 'specifications') responsibility, it is a design choice what mechanisms
> are used.  The persistent store is a prime example.
>

[MIKE] - Following Monica's thread, I think that we should identify
assertions in the conformance testing requirements
that we recognize as "implementation guidelines", since in fact the ebMS
specification
is full of them.   For example, "storing a message in its entirety in
persistent store" is one
assertion that falls in the implemtation guideline category.  I believe that
any test requirement identified as an
implementation guideline should NOT have an abstract test case.  If, however
we recognize
a test requirement as being a valid conformance test requirement, but we
cannot write a test case for it because of constraints
on our test framework, then we SHOULD write an abstract test case for it
(e.g. MSH interrupts and
restarts ).  I think that this approach would give us a consistent test
requirements (and test suite) document.

> >
> > This is the case about requirements about the state of the persistence
> > store that is supposed to be part of an
> > MSH implementation: persistence of a message cannot be directly
> > observed, only its effect on the overall behavior
> > of the MSH (e.g. resending the messsage after a shutdown/recovery) can
> > be observed in an abstract test case.
> >

[MIKE] - This is the MSH interrupt/restart conformance test requirement,
which I believe is a valid
conformance test requirement ( not an implementation guideline ), and must
have an abstract test case.

> >
> > 3. Concrete test cases:

[MIKE] - I recommend the use of the term "executable test case" ( we can use
"concrete" to describe
its meaning.. but the correct formal testing term is "executable test
case/test suite").

> > ---------------------------------
> > A concrete test case uses and assumes an instance of the Test
> > Framework, i.e. a specific test harness.
> > It is a translation of an abstract test case into the material
> > specified by this Test Framework version.
> >
> mm1: I would say that the concrete test case is a detailed or explicit
> representation(s) of the abstract test case that is fully specified and in
> line with the Test Framework.  It also, even if implicitly, provides the
> confidence that the test framework and the original/reference
> specification provided enough clarity to define and execute against the
> test requirements in support of the specification requirements.
>
[MIIKE] - Monica's additional description I believe further clarifies the
integration of
executable test cases with the test framework, the test requirements and the
specification itself.

> I would suggest we focus on what the framework can do, and why a test
> framework may or may not be the appropriate venue
> to verify reference specification requirements.  If you look at the
> example below, you see that the application level interactions,
> interfaces, processing
> fit into this category. I think these are parameters we place on the
> test framework, not exclusively its limitations.  Think (+) not (- -
can't).

[MIKE] - I agree that, based upon all of the conformance test requirements
we have
collected from the MS specification, we have perhaps "overloaded" the test
framework
in terms of trying to make it everything to everyone.  Particularly because
many of the
test requirements are in fact implementation guidelines, or
application-level guidelines.
I suggest that our best approach to those requirements ( and they are some
of the ones that are highlighted
in "RED" in the conformance test requirements list) is to recognize them as
implementation guidelines, and not
try to "shoehorn" them in through abstract test case description.  I suggest
working within
the scope of valid conformance testing requirements, and providing abstract
test cases of ALL
valid conformance test requirements only.  If something is an
"implementation guideline", then let's call it that,
and leave it at that.

>
> Thanks.
>
> >
> > Therefore, depending on the version of the Test Framework that is
> > referred to, the same abstract test case can have
> > different concrete expressions. Some test harness restrictions may
> > cause an abstract test case to not be translatable
> > into a concrete test case: e.g. for ebMS and V1.0 of the Test
> > Framework, inability to generate some kind of application input,
> >
> > and inability to carry out shutdown and restart in an automated way.
> > In the case of ebMS test suites, as the Test Framework is not
> > normative (its use is not required), the concrete test suite notation
> >
> > is not normative for the ebMS conformance test suite.

[MIKE] - This is a good technical description of the relationship between
normative
test suites and the test framework.  Another way to further clarify might be
to say that
the abstract test suite can serve as a guide for implementers to create
their
own test framework and conformance test suite, independent of the executable
test suite described in the ebMS Conformance Test Suite specification.


> >
> >------------------------------------------------------------------------
> >
> >IIC Conference Call Minutes: Monday, August 18, 2003
> >
> >Attendance:
> >
> >
> >Monica Martin (Sun)
> >Serm Kulvatunyou (NIST)
> >Mike Kass (NIST)
> >Jacques Durand (Fujitsu)
> >
> >excused:
> >Pete Wenzel (SeeBeyond)
> >Steve Yung (Sun)
> >Tim Sakach (DC)
> >
> >
> >Minutes taker: Jacques Durand
> >
> >CALL DATE: AUG-18-2003 (Monday)
> >CALL TIME: 02:00 PM EASTERN TIME
> >DURATION: 1 hr
> >LEADER: MR MICHAEL KASS
> >USA Toll Free Number: 888-810-5904
> >USA Toll Number: +1-773-756-4804
> >PASSCODE: 18586
> >
> >The agenda will be:
> >1. Status of MS conformance test suite spec. (Mike Kass editor)
> >- Gal assessment of current draft of MS conf test suite. Overview of
remaining issues.
> >- test requirements and their test coverage: comments on Pete/Monica/Mike
doc.
> >- ordering / partitioning based on config/CPA? (how to avoid too many
reconfigs)
> >- conformance profiles or levels (detailed content?)
> >
> >2. BPSS test update (Serm K., Monica M.):
> >- BSI testing definition.
> >- what cooperation/feedback with/to the BPSS team (emphasis on testing
operations).
> >- The "BCF" concept (business collaboration Framework) ..
> >
> >3. Implementers corner, and PR:
> >- Drake Certivo test driver status.
> >- IIC members demo (Test Framework, test suites) at XML 2003, 7-12
December
> >
> >4. RegRep test requirements (compiling their test assertions) guidelines
> >
> >
> >
> >-----
> >
> >1. Status of MS conformance test suite spec. (Mike Kass editor)
> >
> >- Profiles docs: point to functional "groups" of test reqs, as created in
test reqs doc.
> >Such groups will not need to be broken for conf profiles.
> >- CPA grouping: test cases already sufficiently grouped based on CPA,
Mike said.
> >- We'll review the "not testable" test reqs: there seems to be too many
of them.
> >- Pete pointed that several so -called "app-level tests" can in fact be
set/simulated
> >by the "initiator" action. E.g. req #29 (must understand) is not app
level,
> >also the tests on COntentID payload.
> >- Jacques mentioned the RefToMessageId issue, if Test Driver needs to set
it up.
> >(will need to use an intermediate variable). Will check if that is needed
> >in the current conf test suite.
> >- This will be a candidate upgrade for thenext version of the Test
Framework.
> >
> >2. BPSS update.
> >
> >- Serm to get feedback from the BPSS team - for now, busy with Autotech
demo.
> >- Comments needed on "ebXML Lite" from Covisint. Seems to have serious
problems
> >of compatibility / interoperability... Monica consolidating comments.
> >
> >3. Implementers corner, and PR:
> >- IIC demo (Test Framework, test suites) at XML 2003, 7-12 December , on
the agenda.
> >Drake Certivo possible candidate, NIST/POSTECH as well.
> >
> >4. RegRep conformance test requirements (compiling their test assertions)
guidelines .
> >- Mike and Jacques answering to Farrukh from RR TC. First phase is to get
the TC come up
> >with a set of test requirements. IIC to help with this.
> >
> >
> >
> >
> >------------------------------------------------------------------------
> >
> >IIC Conference Call Minutes: Monday, August 25, 2003
> >
> >Attendance:
> >
> >
> >Monica Martin (Sun)
> >Tim Sakach (DrakeCertivo)
> >Mike Kass (NIST)
> >Jacques Durand (Fujitsu)
> >
> >excused:
> >Pete Wenzel (SeeBeyond)
> >Steve Yung (Sun)
> >Serm Kulvatunyou (NIST)
> >
> >Minutes taker: Jacques Durand
> >
> >Time: Monday August 25, 11am PT
> >Host: Fujitsu
> >Toll Free - : 1-800-251-6413
> >Toll - : 1-913-905-1400
> >Participant code: 598136
> >
> >Agenda:
> >
> >1. Status of MS conformance test suite spec. (Mike Kass editor)
> >- Status of current draft of MS conf test suite. Overview of remaining
issues.
> >- test requirements and their test coverage: comments on Pete/Monica/Mike
doc.
> >- ordering / partitioning based on config/CPA? (how to avoid too many
reconfigs)
> >- conformance profiles or levels
> >
> >2. BPSS test update (Serm K., Monica M.):
> >- BSI testing.
> >- what cooperation/feedback with/to the BPSS team (emphasis on testing
operations).
> >- The "BCF" concept (business collaboration Framework) ..
> >
> >3. Implementers corner, and PR:
> >- Drake Certivo test driver status.
> >- IIC members demo (Test Framework, test suites) at XML 2003, 7-12
December
> >
> >4. RegRep test requirements (compiling their test assertions) guidelines
> >
> >
> >-----
> >
> >1. Status of MS conformance test suite spec. (Mike Kass editor)
> >
> >- Mike Kass factored in commetns from everyone in V0.91.
> >- the problem of "app-level" test cases. Pete advised to use "Initiator"
action
> >for many of these.
> >- should we describe abstract test cases even for  cases we know won't be
testable
> >with the Test Framework V1.0? agreement that yes: the concrete test suite
will not
> >be normative, and may evolve with the capability of the test framework.
> >So the abstract test suite should not depend on this. E.g. test framework
cannot
> >automate shutdown/restart, for recovery tests. Yet that should be part of
the
> >conformance test suite.
> >- Tim mentions sthat for non-testbale features, DC relies on an "honor
system",
> >where the candidate claims that it has implemented the features / req.
> >- Jacques will draft a definition of what test reqs / abstact t.cases /
concrete t.cases
> >are, especially whenderived from each other.
> >- Mike said that the distinction abstract/concrete test cases is not
unusual in test
> >methodologies.
> >- Mike: overall, suite lot closer to completion. Main renmaining work:
concrete cases
> >going through the details of the cases, verifying the XPath expressions.
Also, more "Initiator"
> >actions need to be described, e.g. with proper data.
> >- issue with Conv ID on the "Initiators" test steps? (e.g. would test
driver be able to filter
> >"noise" from previous test cases, filtering out irrelevant messages?)
Jacques said
> >that if test driver can clearly associate each test case with one or
maybe two conv ids,
> >and these are unique over the test suite, we are OK. Safe to assume this.
> >- Plan to vote ebMS conf test suite TC spec in September.
> >
> >2. BPSS test update .
> >- Serm asked for comments from the TC, waiting. Possibly wait for 1.1
draft.
> >Is busy on Autotech demo.
> >- Monica: BCF unrelated to ebXML so far.
> >- Tim: in search of a better methodology, for crafting mode generally
test suites (DC working
> >on a business process suite, focus on OAGIS BODs.) No general
methodology, Mike said.
> >
> >
> >3. Implementers corner, and PR
> >- Tim: DC expects methodology developed for ebMS condormance test suite,
be reusable
> >for other conformance test suites (e.g. bus process level).
> >- DC implementing ebMS conformance. Considering Dec (Philly) demo.
> >- Tim will provide some feedback on ebMS conf test suite by mid
September.
> >
> >
> >4. RegRep test requirements (compiling their test assertions) guidelines
> >- idea is that even though we focus now on ebMS, we should not wait to
> >get theRR TC clarify its test reqs: thats useful work to be done mostly
by them.
> >- Farrukh (RR TC) happy with responses from Mike, on (staged approach on
> >a minimal conf profile, SOAP binding, yet both LCM and QM functional
areas.)
> >- estimate in time seems reasonable for Farrukh:
> >o "Marking up" the ebRS document to identify testing requirements - 4
weeks
> >o To create a Conformance Test Suite Specification - (integrating test
> >requirements, test cases and supporting documentation) - 1 month
> >o To create a "full blown" ebRS Conformance Test Suite 12 months
> >.
> >
> >
> >
> >
> >
> >------------------------------------------------------------------------
> >
> >You may leave a Technical Committee at any time by visiting
http://www.oasis-open.org/apps/org/workgroup/ebxml-iic/members/leave_workgroup.php
> >
>
>
>
> You may leave a Technical Committee at any time by visiting
http://www.oasis-open.org/apps/org/workgroup/ebxml-iic/members/leave_workgroup.php
>
>
>