Re: [ebxml-iic] 10/13/2003: Executable Test Case Package

["Michael Kass" <michael.kass@nist.gov>]

Monica,

   Thanks for your input. My comments below:


----- Original Message ----- 
From: "Monica Martin" <monica.martin@sun.com>
To: "ebXML IIC - main list (E-mail)" <ebxml-iic@lists.oasis-open.org>
Sent: Monday, October 13, 2003 6:37 PM
Subject: [ebxml-iic] 10/13/2003: Executable Test Case Package


Mike, it appears that looking at your draft executable test case
document that you have a fairly natural checklist for:

    * Profile
          o Test profile levels - core, core + reliable messaging +
            acknowledgements (core +1),
            (core + 1) + multi-hop
          o References
    * Requirements - Test requirements
          o Pre-Conditions
          o Assertions
          o Mapping
                + Specification requirements
                + Test coverage
          o References
    * Test and test driver configuration
          o Test Driver
                + ConfigurationGroup: Default definition for Test Driver
                      # Default or recommended configurations mapped to
                        test profile levels
          o Test
                + CPA configuration
          o References
    * Message Payloads
          o Normative values for message content
          o References
    * Executable Test Suite
          o References to minimum set of schemas / components (see above)

Secondarily, n 1.x/2.0 enhancements for test framework:
============================================================================
========
Issue: “Point to” Test Cases that test a particular
<FunctionalRequirement> in the Test Requirements document proposed by
Tim Sakach

Proposed Solution: Optimize Test Driver performance by pointing directly
at Executable Test Cases that verify a particular <FunctionalRequirement>

Action: Update specification to explain meaning of the “testCaseRef”
attribute on a <FunctionalRequirement>, and update the Test Requirements
schema
Specification section 5.2 and Test Requirements schema in Appendix B.
Modified ebXMLTestRequirements.xsd schema, must still modify spec

Comment: What if there is not an executable test case?

[MIKE] - In version 1.X or 2.0 of the Test Framework, it will be documented
that if an Executable Test Case does not
exist for a requirement, then the Test Driver simply moves on to the next
Test Requirement defined in the Profile list, and flags
the requirement as "not testable"

What if it is
specialized so it is tested? How can we verify the executable if we
don't have a clear mapping between the abstract and executable?

[MIKE] - The important linkage is to have  a clear mapping between
an Executable Test Case ( or Cases ) and a single Test Requirement.  If an
Abstract Test Case
is represented as multiple "specialized"  Executable Test Cases, then the
important mapping is really between
the Executable Test Case and (its/their) Test Requirement.  An Abstract Test
Case can be generalized,
then implemented by multiple "specialized" Executable Test Cases.


============================================================================
========
Envelope Element
Comment: It seems with the inclusion of SOAP, RNIF, other to Envelope we
may have parameters that dictate different security, reliability and
packaging requirements on the test framework, and ultimately any
conformance or interoperability specifications developed thereof. How do
we maintain the integrity here or do we have a superset test framework
that has different profiles in and of itself, or is this only allowed at
the conformance or interoperability level? For example, taking the
superset approach, how can we expect that the CPA ID will be a part of
the configuration group? Does it change according to the conformance or
interoperability being sought?

[MIKE] - This is a very good question, and reflects the current issue of an
"ebXML-centric"
design scheme.  Perhaps a better schema for a ConfigurationGroup would have
a "generic"
parameter naming scheme, useable by any type of messaging:


<ebTest:ConfigurationGroup id="abc">
    <ebTest:ConfigurationItem>
        <ebTest:Name>myName1</ebTest:Name>
        <ebTest:Value>myValue1</ebTest:Value>
        <ebTest:Type>string</ebTest:Type>
    </ebTest:ConfigurationItem>
    <ebTest:ConfigurationItem>
        <ebTest:Name>myName2</ebTest:Name>
        <ebTest:Value>2</ebTest:Value>
        <ebTest:Type>integer</ebTest:Type>
    </ebTest:ConfigurationItem>
</ebTest:ConfigurationGroup>

This would allow for passing of <ConfigurationGroup> defined parameters to a
message or payload mutator,
regardless of envelope type.

Comments?
Mike

Thanks. Monica


To unsubscribe from this mailing list (and be removed from the roster of the
OASIS TC), go to
http://www.oasis-open.org/apps/org/workgroup/ebxml-iic/members/leave_workgroup.php.