Re: notes from F2F on conformance

["Eve L. Maler" <eve.maler@east.sun.com>]

FWIW, here are the potential "levels"/"layers" of conformance that I heard 
mentioned at the F2F (I hope to compile the minutes and publish them soon, 
which should help matters!):

Idea #1 for layering: A software component might serve as only one of our 
types of authorities (e.g., a PDP), and it might be said to conform as a 
"SAML PDP" because it accepts certain things as input and produces certain 
things as output.  We wouldn't want to force every SAML-using system into 
supplying all the types of system entities.

Idea #2 for layering (as you mention below): As long as the type(s) of 
binding supported are documented/declared, an implementation could be said 
to be compliant with that SAML binding.  There was also talk of designating 
one or more "must-have" bindings, to increase interoperability.

Idea #3 for layering (I think this came from Dave Orchard): There might be 
some sense in which software could conform to our assertions (e.g., use 
just that inner schema without higher XML or binding-related structures), 
without needing to conform to "higher" layers of our spec.  Perhaps he can 
elaborate here (I've copied him because I'm not sure if he's on this list).

I think it's pretty clear that people want as few conformance types as is 
practicable, but of course that's somewhat subjective...

         Eve

At 12:56 PM 4/27/01 -0400, Robert Griffin wrote:

>hi -
>
>i've attached below the presentation i did on conformance at the F2F. 
>There were a couple of issues that came up during the presentation, or at 
>other times during the F2F:
>
>- The most important issue was whether binary compliance/no-compliance was 
>sufficient. The general consensus (in contrast to what Krishna and i 
>suggested in the presentation) was that there needed to be levels or areas 
>of compliance. For example, an implementation that supported the HTTP 
>binding but not the BEEP binding should still be considered compliant with 
>regard to that binding. Similarly, an implementation that consumed 
>assertions but did not produce them could also be considered compliant 
>within its particular area of implementation.
>
>- There was also considerable interest in having at least a strawman for 
>conformance criteria available by next F2F in early June. Several SAML 
>members are already looking at starting reference implementations (or have 
>already begun them, based on earlier submissions to SAML), so we need to 
>start working out the approach to and scope of conformance.
>
>- The question of any OASIS requirements was raised; the only one noted 
>was the expectation of having 3 reference implementations in place before 
>the standard is submitted for a vote (target September).
>
>In the presentation, we had identified 3 major areas of work for the 
>subgroup: 1) reviewing the spec for conformance issues etc; 2) writing the 
>conformance part of the spec; 3) working with implementers, including 
>building/coordinating a test harness for compliance. Given the comments in 
>the F2F, i think it's time to start on the second area? Here's a couple of 
>next steps:
>
>- Get comparable conformance information for other Oasis standards
>- Sub-group brainstorming on compliance levels, etc
>- Draft a strawman conformance section for the spec, including things like 
>compliance levels, criteria for each level, process for demonstrating 
>compliance.
>
>Unless someone else is interested, i'm happy to spend some time next week 
>gathering together conformance stuff from other standards. Krishna, should 
>we try for a sub-group concall sometime like the week after next (that is, 
>week of 7-May)?
>
>regards,
>
>bob

--
Eve Maler                                             +1 781 442 3190
Sun Microsystems XML Technology Development  eve.maler @ east.sun.com