Re: [tag] TA Model

[stephen.green@systml.co.uk]

Quoting Dave Pawson <dave.pawson@gmail.com>:

> On 22/09/2007, stephen.green@systml.co.uk <stephen.green@systml.co.uk> wrote:
>> So how about
>>
>> 1. introducing steps called 'Test' into the flow with IDs,
>> 2. making such 'Test' steps multiple occurrence
> .. of what Stephen?
>
>> 3. adding a unique ID to each part of the step
>> 4. allowing preconditions and postconditions to reference parts of another
>>     step
>
> Why? IMHO a test should be independent. Dependencies on other tests goes
> against this. Perhaps such a test is at too low a level?

An example I'm aware of where a test depends on the pass outcome of a
previous test is a Schematron validation test isn't worth performing
(and therefore can't result in a pass) unless previously there was a
successful pass outcome to a test of the XML syntax such as with XML
Schema or RelaxNG. If the XML Schema validation test was in a previous
TA then there would be a need to make that TA's test result a pre-condition
which I propose could either be in the pre-condition description or be
by a reference to the TA test's result or to the TA as a whole. Hence
the need to have an ID in the TA as a whole but also, I think, in the
Test result to allow greater precision. After all the pre-condition might
be to a post-condition of the previous test - to something that isn't
actually tested formally but is just found to be true (boundary clarity
needed here?).

There may, however, be reason to include the XML Schema validation in the
same TA as the Schematron validation (feasible but I'll take it as an
unproven assumption) but to put the two validation phases into two tests
in the flow in such a way that the dependency can be expressed formally.
Say the second test step of the flow uses an ID from the first step to
express the dependency. It could include a reference in the pre-condition
of step 2 to the first step's result (which requires a unique ID on that
result).

So to summarize so far, things like pre-conditions might need to reference
either things like triggers, results/effects (I too prefer 'result' to
'effect'), pre- and post-conditions in tests so each need unique IDs
*or* they may need to reference the final outcome of a test flow as a whole
so that needs an overall unique ID which could just be the ID of the TA.

I could imagine there might be some situations where the pre-condition
might need to reference something other than the entire TA or the result
of a particular step/test in a multi-step/test flow. If there is an
aspect of the referenced TA or step which isn't actually formally tested
it could be something which is a post-condition. So the post-conditions
need unique reference too to facilitate that.

Either way, I see these as all aspects of a spec (the example is part of
the in-progress OASIS Codelist Methodology and plays a part in UBL's spec)
and therefore to need TAs and not therefore just for the test designers.
They live in the space between spec and tests in such cases. (Two-phase
codelist validation is the example I worked from.)

Regards
-Steve

>
>
>> 5. adding 'Previous Test' and 'Following Test' pointers to each step
>
> I find this confusing. Test flow / routing shouldn't be set by the test.
> It may be that a test could be re-used in different places?
>
>
>> 6. adding 'Comment' outside the flow
>> 7. adding something to state what constitutes overall pass or fail of TA
>> 8. adding specification quote
> or reference. By whatever means are appropriate.
>> 9. being explicit in the name of the part about it being an ID or reference
>
> I don't understand this one.
>
>

>>
>> Is that too complex or just complex enough? Maybe most things are optional
>> I suggest where there is a reference such as 'Test Trigger Cross Reference'
>> then it could be implemented as a hyperlink if the implementation is HTML
>> (so it might not add complexity all that much).
>>
>> Extended example:
>>
>> Test Assertion ID: TA-101c
>> Specification requirement ID: < req 101>
>> Specification requirement quote: "Purchase Order Receipt documents
>> must be XML documents that are valid against the XML schema
>> po-receipt.xsd, and that must use the approved format for product
>> identifiers."
>> Item under test: A business endpoint.
>> Assertion prose: "Purchase Order Receipt documents must be XML
>> documents that are valid against the XML schema po-receipt.xsd, and
>> that must use the approved format for product identifiers."
>> Comment: Test TST#112 is only required when test TST#111 results in  
>>  a warning
>> Test Assertion Pass Criteria: Test Assertion results in 'pass' if no
>> test result is 'fail' and all required tests are run.
>
> Except for the test criteria, a good example.
> Pass conditions must be explicit and UUT oriented. This would might be
> 'Parser must not report any errors when validating the instance.
>
> Other than that this seems to be about the right 'level' of abstraction.
> Says 'what' to test. Not how.
> Is independent of other tests.
> (Might add a precondition: An xml instance - PO receipt, has been received
> by the end point).
> Post condition might be, PO receipt on disk as file identified by
> ...PO ident number
> with XML extension.
>
> I'd generalise that the 'right' level of abstraction is at, or
> slightly lower than,
> the sentence level in the specification. The usual caveats about no   
> conjunctions
> etc. Unsure if its reasonable to go much beyond such a generality.
>
>
>
> regards
>
>
> --
> Dave Pawson
> XSLT XSL-FO FAQ.
> http://www.dpawson.co.uk
>