Re: [security-services] conformance testing for SAML 1.1 and SAML 2.0?

[Alistair Young <alistair@smo.uhi.ac.uk>]

> Vendors could then build test suites that implement the conformance 
> test suite
That would be very helpful, certainly in our case, where we've produced 
a SAML1.1 implementation and are working on a SAML2 version. If we 
could be of any help, I think we could contribute.

I don't think it's encumbent upon the TC to produce a test sute but it 
would certainly be a courtesy to vendors and would help ease 
integration issues when building federations.

Would the implementation be "certified"? or would it just be for the 
vendor's peace of mind? Would the TC endorse in any way a SAML 
implementation that fullfilled the conformance requirements?

Alistair

On 14 Mar 2005, at 23:39, prateek mishra wrote:

> Colleagues,
>  
> With the standardization of SAML 2.0 now complete, I would like to 
> bring to your attention the issue of conformance testing of SAML 1.1 
> and SAML 2.0 implementations.
>   
> While there have been a number of implementations of SAML 1.1 and 
> several planned for SAML 2.0, we do not at this time have any means of 
> testing for conformance. This means that we have currently no 
> way to determine whether or not a claimed implementation has fully 
> implemented all of the many MUSTs and MUST NOTs that we have taken 
> great care to include in our specification. 
>  
> The core issue is what role, if any, should the TC play in specifying 
> or managing SAML conformance. My suggestion would be that the TC 
> manage a detailed spreadsheet or conformance test suite derived from 
> the specification as the "official" statement of conformance. Vendors 
> could then build test suites that implement the conformance test 
> suite.
>   
> To progress in this direction, we would need at least one or two 
> participants to become "champions" for this effort, and, also figure 
> out the appropriate logistics. For example, is there a need for a 
> sub-group to work on this issue and report back to the TC or should it 
> take place within the full TC?
>  
> Of course, the major task is for TC participants to first decide what 
> role we should play in this area. The goal of this message is to get 
> that discussion started.
>  
> - prateek