Kevin: thanks for comments.
See Inline <JD>
-jacques
From: Kevin Looney [mailto:kevin.looney@oracle.com]
Sent: Monday, March 07, 2011 2:33 PM
To: Jacques Durand
Cc: tag@lists.oasis-open.org
Subject: Comments: [tag] Groups - TestAssertionsModel-csd03-diffs (testassertionsmodel-1.0-cd-03-diffs.odt) uploaded
Hello Jacques,
I've had a chance to review CD-03. I've been reviewing the entire document (rather than just the current changes), so pardon if some comments question older material.
Thanks,
Kevin
---------------------------------------------------------------------------
1.) Page 6, Section: Domain terminology
Conformance clause - not entirely clear what an 'artifact' is. Is it an implementation of some specification that is being analyzed? This also comes up in the Conformance section (section 5).
2.) Page 8, Definitions and Rationales
The diagram presents an item "Conformance Profile". There is no previous definition for this.
<JD> yes – to be renamed just “Conformance”.
3.) Page 9, Test Assertion Set and Test Assertion Document
Conformance module/profile/level -- none of these terms are previously defined. They may even be superfluous/confusing to this paragraph.
<JD> replaced with “…which define various ways to conform to a specification.”
Same thing for Test Assertion Set / Test Assertion Document - Even though they are in the title, there is no specific definition of these in the previous text.
<JD> right. Modified the paragraph and its title to address this.
4.) Page 9, Benefits of Test Assertions.
As a side note - (perhaps you can turn this into an additional benefit in this section)
In the Java world, other organizations were motivated/empowered to create their own white-room implementations of Java APIs and VMs. These implementations were originally incompatible (to various degrees) with Sun's own implementation.
Early on, Sun realized these incompatibilities could fracture the market, and make the lives of App Developers and Users impossible (since they could not rely on the stability of the operational environment they may have chosen). Because of this, Sun created Public Specifications, and a commercial conformance test suite (TCK) that would test the conformance of a given Java API/VM to the Public Specifications (Sun also provided a reference implementation of the API/VM that passed the TCK suite). The basis of the TCKs were Test Assertions, identified from the Public Specifications (JSRs), and realized by the Test Cases within the TCK.
So, the benefit for Sun/Oracle was the coherence of the operating environment. App Developers and Users could rest assured that their apps would behave consistently across different (conforming) API/VM implementations, and on different OS and HW environments. This portability was the key to the Java Language's ubiquity.
<JD> I have tried to abstract this benefit in a new short paragraph: “Aligning Implementations”
5.) page 10-11, Optional Test Assertion (grammar, consistency issue)
Tag and Variable descriptions begin with "Test Assertions May ..." (as in 'may include a Tag description for ...'). For consistency, all of the optional constructs should probably begin this same way, indicating their optionality.
<JD> I have changed these defs.
6.) page 11, Informational Notation.
For the predicate, "here an assertion stating the Predicate and referring to the Target. Notational convention: the reference to the Target is within square brackets."
How much of this is recommendation (versus requirement)? For Oracle, we view the 'Target' as non-essential for our own assertion evaluation, but we are willing to compromise by implicitly identifying a Spec as the default Target. However, requiring that the 'predicate refers to the target' seems a rather strong requirement. Most likely, we would use the predicate to identify the behavioral aspects of an assertion being tested - without respect to the Target (nominal specification), and without trying to re-identify a target for hundreds of thousands of existing assertions/tests.
I think we should be careful here in stating that a predicate may/should refer to a Target (in non-absolute terms).
<JD> Not sure I follow this… the Target is not a specification, but an implementation of it.. (although the Target may be implicit, the predicate is supposed to be about it.)
7.) page 12, section 3.2, Conventions used for formally defining the model
I'm wondering how redundant it is here, re-stating that the term 'class' does not carry the object-oriented meaning (as already stated in section 1.1)
<JD> Right. To be reduced.
8.) Page 27, 3.3 Outcome of a Test Assertion
This section seems a bit weird. I agree that these are possible outcomes "for some system, running some tests (where the tests may be based on some test assertions). Are the Assertions themselves part of an executable system (No). Are the Assertions executed (No, test cases are executed).
I believe a test execution system (implementation) determines its specified outcome. Those outcomes may be what is specified here, or they might be just 'Test Pass' and 'Test Fail', not identifying tests that were filtered from execution.
I don't have a specific recommendation here, but I think this section might be a candidate for "Overspecification".
<JD> To review…. I have proposed a rewording focusing on the overall “semantics” (not the outcome) , also w/r to a Test Case deriving from the TA (what it means to “derive” from a TA).
9.) pages 28 - 32, Sets and "shared"
I don't have a specific recommendation here. I feel sort of dubious about the need to define Sets/shared properties of assertions. Clearly, this mechanism is a short-hand (ie. one could easily -possibly automatically - enumerate all assertions in a set, copying the same shared sets of properties). In some way, I feel like the purpose of the "Test Assertion Model' should be solely the specification of an Assertion, and not how assertions may be aggregated, and commonalities factored out.
<JD> That is a very legitimate question. I also feel that the “TA Set” model may be a turn-off for many, given the considerable effort needed to understand all its options… NOTE: even if removed from the Model specification, the notion of “set” could be developed in the Markup as an extension. To be discussed at next meeting.
--
kevin.looney@oracle.com