Test case generation from the core spec

[kuehne@trustable.de]

Hi all,

I like to give you a short status about my effort to derive conformance test cases from the core spec. The 'Test assertion guidelines' spec provides a good tutorial how find the assertions. The first step is to find all 'MUST' and 'MUST NOT', define the product ( in our case 'client' and 'server' ) and mark the whole condition.

In case of the core spec I started enthusiastically and all the MUSTs are marked quikly. But getting into details the problems grow : 

Looking at section 3.1 SignRequest I stuck at the RequestID :
Even though the section is about the the SigningRequest, what's build by the client, the condition says something about the server ! Moreover this condition is a copy of the corresponding paragraph in 3.2 SignResponse.

Next problem in SignRequest is the section about InputDocuments. The heading denotes this element as 'Optional'  but the text says that it is required in a SignRequest.

Generally the term 'REQUIRED' is a bit vague what system is responsible for checking and what should happen in error cases.

Another general point is the definition of a condition for the client. But there is no definition what should happen when the client sends a non-conformant request. In my ideal version there should a set of conditions for a conformant client and a conformant server.

So generally there is much more work to be done than just extracting the conditions semi-automatically.  In some cases this may even affect the core. A workaround to preserve the core could am additional version of the where the original text remains unchanged and the conformance statements are mixed into it marked as conformance requirements.

So many work to, opinions welcome.

Greetings

Andreas