10/13/2003: Executable Test Case Package

[Monica Martin <monica.martin@sun.com>]

Mike, it appears that looking at your draft executable test case 
document that you have a fairly natural checklist for:

    * Profile
          o Test profile levels - core, core + reliable messaging +
            acknowledgements (core +1),
            (core + 1) + multi-hop
          o References
    * Requirements - Test requirements
          o Pre-Conditions
          o Assertions
          o Mapping
                + Specification requirements
                + Test coverage
          o References
    * Test and test driver configuration
          o Test Driver
                + ConfigurationGroup: Default definition for Test Driver
                      # Default or recommended configurations mapped to
                        test profile levels
          o Test
                + CPA configuration
          o References
    * Message Payloads
          o Normative values for message content
          o References
    * Executable Test Suite
          o References to minimum set of schemas / components (see above)

Secondarily, n 1.x/2.0 enhancements for test framework:
====================================================================================
Issue: “Point to” Test Cases that test a particular 
<FunctionalRequirement> in the Test Requirements document proposed by 
Tim Sakach

Proposed Solution: Optimize Test Driver performance by pointing directly 
at Executable Test Cases that verify a particular <FunctionalRequirement>

Action: Update specification to explain meaning of the “testCaseRef” 
attribute on a <FunctionalRequirement>, and update the Test Requirements 
schema
Specification section 5.2 and Test Requirements schema in Appendix B. 
Modified ebXMLTestRequirements.xsd schema, must still modify spec

Comment: What if there is not an executable test case? What if it is 
specialized so it is tested? How can we verify the executable if we 
don't have a clear mapping between the abstract and executable?

====================================================================================
Envelope Element
Comment: It seems with the inclusion of SOAP, RNIF, other to Envelope we 
may have parameters that dictate different security, reliability and 
packaging requirements on the test framework, and ultimately any 
conformance or interoperability specifications developed thereof. How do 
we maintain the integrity here or do we have a superset test framework 
that has different profiles in and of itself, or is this only allowed at 
the conformance or interoperability level? For example, taking the 
superset approach, how can we expect that the CPA ID will be a part of 
the configuration group? Does it change according to the conformance or 
interoperability being sought?

Thanks. Monica