RE: [tag] Test Assertion Modeling - comments, etc

["Serm Kulvatunyou" <serm@nist.gov>]

-----Original Message-----
From: stephen.green@systml.co.uk [mailto:stephen.green@systml.co.uk] 
Sent: Tuesday, August 14, 2007 4:55 PM
To: tag@lists.oasis-open.org
Subject: Re: [tag] Test Assertion Modeling - comments, etc

Having to rethink the idea of test assertions for this SBS rule -

The document which conforms to the spec, a UBL SBS document, is a kind of
spec itself - it specifies some business, whether a payment that needs to be
made in conformance to an invoice or a supply of goods or services in
conformance to an order.
The spec is kind of saying with this SBS rule that certain specified parts
of that document should be regarded as informative only. Now how on earth
can you test for that? :-)

>>> How about this? The response based on a document with or without the
informative information must be the same.


Maybe you can't in terms of software at all. Maybe it's just a legal thing
and not for testing at all. How do you test that something is being treated
as normative or informative in a particular implementation? In this case it
is a matter of testing that say an order is part informative and part
normative and that the informative part, if present at all, does not require
normative treatment. That can only be a matter or business and legal
vigilance can't it - not a software testing thing at all perhaps. A lesson
seems to be here though that not everything need be tested for it to be be
worth writing in test assertion formal language - at least not tested with
typical software tests. The test assertion technique might be used to
facilitate business and legal implementation of a business document and
software which produced it in a way that software testing itself cannot do.

--
Stephen Green

Partner
SystML, http://www.systml.co.uk
Tel: +44 (0) 117 9541606

http://www.biblegateway.com/passage/?search=matthew+22:37 .. and voice



Quoting stephen.green@systml.co.uk:

> Thanks. Excellent.
>
> Of course, the tests can be limited to a business process of which the 
> document under test is a part. In a way this binds the testing not 
> only to the business document but also to it's business process - an 
> interesting aspect of the SBS and other business language profiles 
> perhaps.
>
> So here the TA has to take into account not just the subset but also 
> the business process of which it participates. So the spec can't 
> really include the TAs as such unless it can generalise them for all 
> anticipated business processes or somehow make the assertions 
> independant of the processes.
>
> I did imagine there will be quite a few layers to test and also that 
> these might inter-relate in some way - is that a problem though? 
> inter-relating layers of validation? A test in one layer might depend 
> on results from another layer.
> This means there is a sequence in testing with dependencies even 
> crossing between layers of testing. What happens if any tests are 
> mutually dependant or if two layers of tests are mutually dependant? 
> Is it necessary to declare all depend- encies somehow in the TAs? What 
> if dependencies existed between different sets of assertions (if the 
> assertions were split for different audiences as I think you suggest) 
> but not all test sets were used as supplied. In that case making 
> dependencies explicit would show up the mismatches. But if the test 
> assertions were put into the same file then ID and IDREF could maybe 
> be used to assure of the integrity of dependencies between id and ref 
> to some extent when validating the TA document. Else XInclude or the 
> like might be used if dependencies crossed file boundaries.
>
> --
> Stephen Green
>
> Partner
> SystML, http://www.systml.co.uk
> Tel: +44 (0) 117 9541606
>
> http://www.biblegateway.com/passage/?search=matthew+22:37 .. and voice
>
>
>
> Quoting Dave Pawson <dave.pawson@gmail.com>:
>
>> Wow. Quite a test.... or more correctly a test sequence?
>>
>> On 14/08/07, stephen.green@systml.co.uk <stephen.green@systml.co.uk>
wrote:
>>> Hi. Thanks Jacques and David and for comments.
>>>
>>> Yes, the testing of this item for the SBS profile for UBL which 
>>> relates to the Sender (there is another rule which relates to the 
>>> Receiver) could be like the following:
>>>
>>
>>
>>> Essentially it is a business rule, albeit of an unusually technical 
>>> nature but there are still tests which can be applied. Some such 
>>> tests might be best applied by technical auditors though, even by 
>>> legal experts in some cases.
>>>
>>> Not sure how these test requirements would be expressed as test 
>>> assertions though since maybe the target audience of the test 
>>> assertion would be a technical auditor or legal expert, even a legal 
>>> court (if say there was a large sum unpaid because the receiver had 
>>> ignored some payment terms or tax amounts which were external to the 
>>> subset) eventually.
>>
>> Layering needed?
>> A single test (pass or fail).
>> A test group (again pass or fail with test results) If test group 
>> passes, output the message in an appropriate format?
>> Appropriate to the audience that is.
>> Groups layered into supergroups as needed, culminating in a complete 
>> application.
>> The end resut is the summation (done automatically, not collated by 
>> the application, which cheats) of the test groups. Again either pass or
fail.
>>
>>
>>
>> Nasty:
>>  Writing tests with built in debug.
>>  I.e. run it with the debug flag set and all the techie garbage flows.
>>  Switch it off and the legal eagle sees pass, or fail, test number 1,290.
>>
>> If you've multiple audiences, the test application layer needs
parameters.
>>
>>
>> regards
>>
>>
>>
>>
>> --
>> Dave Pawson
>> XSLT XSL-FO FAQ.
>> http://www.dpawson.co.uk
>>