FW: [tag-discuss] Re: Test case metadata...

["Serm Kulvatunyou" <serm@nist.gov>]

-----Original Message-----
From: david_marston@us.ibm.com [mailto:david_marston@us.ibm.com] 
Sent: Sunday, December 03, 2006 12:17 PM
To: tag-discuss@lists.oasis-open.org
Subject: RE: [tag-discuss] Re: Test case metadata...

I am responding to those parts of Serm Kulvatunyou's message (of 12/2)
that I can address without a detailed reading of the ebXML documents.

SK>There are a lot of terminology mismatches around here.

Agreed.

SK>Having read some software testing literatures, the terminology in
>the IIC specification seems more common.

Please provide some citations. I think this is one of the reasons I
wanted to suggest a workshop, to try harder to find out what has been
done elsewhere.

<serm> I think I can provide some.

My reference of the IIC work is still v. 1.0 (I couldn't find the link on
OASIS, I hope Jacques can provide the link as the list has rejected the
attachment two times already).  The current version is 1.1 (right Jacques?).
For version 1, I think you can jump right to Part II around page 27. Test
Requirements data model is in section 5.

Another reference that might be useful is from STEP testing, standard for
exchange of product data, www.nist.gov/msidlibrary/doc/stepbook.pdf.

I also have read a few papers related to testing using TTCN-3 (one I found
in my archive is attached). I think I didn't experience too much terminology
mismatches.

And 'yes', I did read the OASIS Conformance TC document. I think the IIC
terminologies are also derived from that spec.

I am eager for the workshop!

</serm>


SK>...Note that in IIC, a 'test case' contains everything from
>relationship to the TR, machine interpretable test execution steps,
>inputs, and verification conditions/expected result, and more.

The only terminology issue here is that I and my metadata colleagues
treat the "test case" as an abstraction and the "test case metadata"
as a real entity (an XML file) that has the data. From the above, it
seems that we agree about the set of data that describes a test case.

<serm> Definitely a problem here, I only experience with literatures
treating the term 'test case' as concrete - a machine interpretable
execution instruction, as opposed to the term 'abstract test case'. </serm>

SK>...some elements in the [W3C Note on] test case metadata are
>defined to be machine processable.... On this basis, perhaps it is why
>some of the list participants are thinking of some automations based
>on the TA, if it includes such elements.

This gives me a chance to clarify something about automation in
testing. There are two thoughts commingled above.
1. Automation to run a set of test cases against a test subject has
been and will continue to be facilitated by having test case metadata
that is platform-independent and describes the phases (setup, test
execution, comparison of results, cleanup) in platform-independent
terms. TAs are beneficial for labeling and coverage analysis, but are
not essential.
2. Automation could be used (and maybe is already) to create parts
of a test suite. TAs are much more central to this kind of work, and
being machine processable is where true advances in the field could
arise.

Perhaps I make the notion of machine processable TAs sound too
ambitious. If we assume an XML notation for a moment, I see the
difference as something like this.
XML, but not machine processable in any meaningful sense:
<assertion>A sentence in English prose.</assertion>
XML that supports to machine processing:
<assertion>
  <contingency>fixed wording for a contingency</contingency>
  <contingency>fixed wording for a contingency</contingency>
  <contingency>fixed wording for a contingency</contingency>...
  <stimulus>fixed wording describing how triggered</stimulus>
  <result>fixed wording describing the result</result>
</assertion>
To a great extent, the "fixed wording" parts use wording specific to
the domain of the spec, and may use additional XML sub-elements or
attributes to isolate certain details. A set of TA guidelines would
try to describe the desirable qualities of the fixed wording. One
such desirable quality is that each sub-element could be processed by
automation to select and configure building blocks of testing. A
separate ambition is to have the contingencies of more complex test
assertions build on the results of simpler ones, which would allow
TAs to be compounded, in which case the test cases they inspire
could also be compounded.

<serm>
I agree with (1) and that it is out of scope for TA/TR spec. And yes TAs
will be beneficial....essential or not perhaps may be depending on what your
job is :).

It would be great if you can provide some simple examples of those "fixed
wording". Are you thinking of some sort of patterned natural language?
Otherwise, my feeling is that it is stepping into the area of test case
information. I think the most critical point is what standard
developers/domain experts are willing and comfortable to write. Just prose
or a more machine processable version of it. Of course, the more it is
machine processable the easier the life of test engineers and developers.
Maybe we need to survery.
</serm>


SK>...the TA or TR spec should include only descriptive info, i.e.,
>it should not be much different from those already included in the
>IIC spec for TR.

Serm, you seem to think that a TC should be formed, despite the IIC
spec and what you've read in the literature. Can you provide more
justification?

<serm>
Well, the IIC spec v. 1.0 only considers ebMS spec (and messaging part of
the ebRR). The new version will include aspects of ebBP. I think there are
rooms for imporvements to the requirement spec, e.g.,
prerequisite/relationship b/w TRs was not there, maybe the assertion element
could be divided into input, expected result and/or expected behavior. We
can also take a better look at others to see what they have. I think Jacques
should have a better answer, since he eats and sleeps with the IIC :).
</serm>


I think that if a TC is formed, it must at least be mindful of how
much could be done to machine-process TAs, if the format supports it.
.................David Marston

<serm> Why not....as mentioned above...as much as it is chewable by the
standard developers. 

Cheers!
</serm>

---------------------------------------------------------------------
To unsubscribe, e-mail: tag-discuss-unsubscribe@lists.oasis-open.org
For additional commands, e-mail: tag-discuss-help@lists.oasis-open.org

iso9646.pdf