Re: [kmip-comment] Possible solution to referencing issues

[Tim Hudson <tjh@cryptsoft.com>]

On 2/04/2014 10:45 AM, Patrick Durusau wrote:
> PS: I thought an interop test shows that different implementations can
> process the test cases successfully. Not so much conformance being
> tested but that different implementations have the same reading of the
> standard. Not exactly the same as testing conformance, which is
> against the standard and not another implementation.

Actually the profiles are a narrowing of the specification. They all note the list of clauses within the specification which apply and contain a narrowing of those to specific ranges and in addition also specify the flow of messages along with the specifically allowed variations on each of the messages within those flows. All of these are clearly contained in the conformance clauses within each profile.

To make it clear - you can conform to the base specification and not conform to a specific profile. You can also conform to the base specification and not be interoperable given that most implementations will only implement a subset of the very broad range of operations and objects supported by the specification (and may also select different ranges of allowed enumerations and different ranges of length values too).

Your concerns raised to date (assuming I'm correctly following them and nothing so far has indicated that I've misunderstood) have been that you believe that the test cases should be able to be made informative without altering the conformance outcome in the documents. Unfortunately that isn't the case. The test cases within the profiles are explicitly noted as normative (contained within the conformance clauses) and the choice to do that was one made with careful consideration within the technical committee.

The purpose of the KMIP specification is to deliver interoperable key management and we have as a technical committee elected to solve the precision issue by stating the clauses within the base specification, the test cases themselves, and a precisely specified set of variations included within the profiles to achieve that common objective - interoperability. That we are using test cases as part of the conformance clauses provides the precision required and has the added benefit of being explicitly testable. This goes well beyond how many other technical committees within OASIS approach this.

For reference, KMIP 1.0 did indeed take the approach you are suggesting - of having no normative test cases (and in fact there were no test cases in any of the normative documents and a separate document as a committee note containing informative test cases only). The decision within the technical committee (and discussed outside the technical committee itself with a broader group) was to add normative test cases in specific separate narrower profile documents.

A substantial portion of your feedback items have been based on the (incorrect) assumption that the test cases are informative, and we have provided feedback that the documents themselves make it rather clear that they are not. Additionally, if the test cases were made informative rather than normative then the documents would require substantial changes in order to attempt to address what is covered in the test cases.

As a technical committee we will respond to all the public review feedback in due course and that response will represent the view point of the technical committee as a whole rather than the individuals here (including myself) that are simply attempting to provide background as to why we as a technical committee elected to take the approach that we have. I suggest we provide the technical committee with the time in which to perform that task.

Thanks,
Tim.