draft presentation on conformance for F2F, for review

security-conform message

i've attached a draft of the 1/2 hour presentation on conformance for the F2F, Tuesday morning. Let me know of any additions/corrections/suggestions?

-----Original Message-----
From: Robert Griffin [mailto:Robert.Griffin@entrust.com]
Sent: Tuesday, August 21, 2001 10:16 AM
To: security-services@lists.oasis-open.org; 'lynne.rosenthal@nist.org'; 'mark.skall@nist.org'
Subject: notes from conformance subgroup meeting

Attendees:
        Lynne Rosenthal (NIST)
        Mark Skall (NIST)
        Gayathri Krishnamurthy (Sena Consulting)
        Maneesh Bhandan (Sena Consulting)
        Bob Griffin (Entrust)

Actions:
Bob: prepare presentation for F2F with issues discussed at today's meeting and distribute to sub-group for review.

        Bob: continue to define Conformance Program spec test cases and include design-level information.

Summary:
        We quickely reviewed status of conformance work:
                - Conformance Clause is in SAML spec, as presented to June F2F
                - Conformance Program spec is under development; updated since June F2F with test cases
                - Schedule for conformance docs and test suite to be determined after August F2F
        Lynne and Mark reviewed the Oasis Conformance TC meeting/concall yesterday.
                - Documents outlining recommended conformance program for OASIS are being edited and reissued, using examples (including from SAML) and with stronger recommendations ("shall" rather than "should")

                - Would be helpful to develop stronger interaction between SAML and ebXML conformance groups
        We then looked at several issues:
                1) Hal Lockhart's issue (attached) of whether bindings should be specified as part of conformance, or only protocols. Recommendation from sub-group (to be presented to TC at F2F) is that conformance test suite be implemented for all bindings in the standard, that conformance be defined in terms of protocols, but with organization claiming conformance specifying which bindings were tested.

2) Should test cases include all standard bindings. Recommendation was that we do so, so that the test suite can be built with all bindings.

3) How should test cases handle optional elements of assertions, protocols and/or bindings? Recommendation was that test suite verify that any implementation or application using extensions has correctly specified the extension / condition, but that the extension or condition itself will not be tested by the standard SAML conformance test suite. Also need to verify with TC whether registry for extensions is desirable.

4) How should test cases be implemented: for example, by test client inspecting formats of produced assertions, by requiring applications/implementations to interact with black-box consumer/producer? Recommendation is that this be explored in detail with regard to differences between testing profiles and testing protocols, particularly with relation to issue raised by Hal.

-----------------------------------------------------------------------------------------

Should protocol bindings be the subject of conformance? The bindings sub
group is defining both SAML Bindings and SAML Profiles. It has been proposed
that both of these would be the subject of independent conformance tests.

SAML Binding: SAML Request/Response Protocol messages are mapped onto
underlying communication protocols. (SOAP, BEEP)

SAML Profile: formats for combining assertions with other data objects.
These objects may be communicated between various system entities. This
might involve intermediate parties.

This suggests that a Profile is a complete specification of the SAML aspects
of some use case. It provides all the elements needed to implement a real
world scenario, including the semantics of the various SAML Assertions,
Requests and Responses.

A Binding would simply specify how SAML Assertions, Requests and Responses
would be carried by some protocol. A Binding might be used as a building
block in one or more Profiles, or be used by itself to implement some use
case not covered by SAML. In the later case, it would be necessary for the
parties involved to agree on all aspects of the use case not covered by the
Binding.

Thus conformance testing of Bindings might be undesirable for two related
reasons:

· The number of independent test scenarios is already large. It seems
undesirable to test something that does not solve a complete, real-world
problem.

· Parties would be able to claim "SAML Conformance" by conforming to a
Binding, although they would not be able to actually interoperate with
others in a practical situation, except by reference to a private agreement.
This would likely draw a negative response from end users and other
observers.

· Simplifying testing procedures when a Binding is used in several
Profiles that a given party wishes to conform to.