RE: [oic] Test Assertions and Test Cases

["Dennis E. Hamilton" <dennis.hamilton@acm.org>]

Thanks, Rob and Bart.

I think this discussion is important.

This reminds me that one of the David Marsten questions for getting started
was whether or not we would be creating test assertions.  Those questions
fell off our agenda.  This is a key one for us to revisit.

It seems to me that we need to be specific about test assertions and
prescription levels.   I also think we need to consider prescription levels
and/or test assertions that are not based on normative requirements in the
specification.  We need to differentiate those where interoperability
conditions are to be assessed where the ODF specification does not establish
a measurable determination of conformance.

The OASIS Test Assertion Guidelines (TAG) TC has been developing committee
drafts on this subject, and an important case that has been raised on the
tag-comment list is the possibility of prescription levels that allow test
assertions for conformance to be integrated with test assertions for other
purposes (recommended practice, interoperability cases, ... ?).  See
<http://lists.oasis-open.org/archives/tag-comment/200812/msg00000.html>.

We should probably examine their Guidelines cd3 1.0 to see if there is some
useful perspective for our work:
<http://www.oasis-open.org/committees/documents.php?wg_abbrev=tag>
This draft doesn't seem to provide for prescription levels that are not tied
to explicit normative statements and conformance targets.  For the OIC TC to
accomplish much, I think we do have to address that.

The OASIS TAG TC is also developing markup for Test Assertions and their is
a download on that also.  I don't know if that will inform our reliance on
metadata for test cases.  It is probably worth reviewing.

 - Dennis

-----Original Message-----
From: robert_weir@us.ibm.com [mailto:robert_weir@us.ibm.com] 
Sent: Tuesday, April 07, 2009 08:15
To: oic@lists.oasis-open.org
Subject: RE: [oic] Groups - svn-metadata-barth.zip uploaded

[ ... ]

I think you implicitly, or at least mentally, create a test assertion 
before you write a test case.  The ODF standard doesn't test itself. 
Somehow you need to read the text, decide what requirement is stated, and 
then design test cases that test that requirement.  These are two 
different skills. 

[ ... ]

Producing this test assertion is straightforward and amounts to 
interpreting the text of the standard and determining the prescription 
level.  In some cases, like spreadsheet functions, this will be simple. In 
other cases it will require more effort to figure out how to define the 
predicate.

Developing test cases from a test assertion requires a different set of 
skills.  It is more a QA specialization -- how do you design a test?  The 
craft of test design is to enumerate a small set of test cases that will 
find the maximum number of errors.  So you typically test the main case, 
an error case, limits and edge cases, etc.  Of the infinite number of 
possible test cases, which ones will find the most bugs?

[ ... ]

So I see these two steps as being different. To create the test 
assertions, someone reads the ODF standard very closely noting all 
testable provisions in the form of test assertions.  In the 2nd step, 
someone crafts one or more test cases for each test assertion. 

[ ... ]

2) The ODF standard defines conformance for documents as well as for 
applications.  How do we test the conformance of an ODF document? 
Obviously, creating document test cases does not help here.  You can't 
test a document with another ODF document.  But having a set of test 
assertions related to ODF document conformance would be very useful, since 
that could be automated via other means, such as an ODF Validator tool. 

3) Test assertions fulfill the part of our charter that calls for us to "
produce a comprehensive conformity assessment methodology specification 
which enumerates all collected provisions, as well as specific actions 
recommended to test each provision, including definition of preconditions, 
expected results, scoring and reporting"

-Rob

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that
generates this mail.  Follow this link to all your TCs in OASIS at:
https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php