Analysis: What Is the ODF Specification Conformance Proflie?

["Dennis E. Hamilton" <dennis.hamilton@acm.org>]

I have asked forms of this question before.  I don't recall receiving any
response so I will ask again.  Without understanding the answer to this
question, it is difficult to review the current specification drafts with an
eye to how conformance is handled.  I have assumed the question have already
been addressed and I simply don't know what the agreed approach is.

If there has been no discussion and agreement on the approach, perhaps now
is a good time to come up with a deliberate resolution (separate from the
conformance-section proposal before us, but influencing its further
refinement and alignment of the entire specification with that).

THE QUESTION:  What are the requirements for how conformance is treated in
the ODF 1.2 Specification?  I don't mean, what is ODF conformance?  I mean
what is the commitment to how conformance is dealt with in the specification
itself.  This is a specification quality question, not an ODF conformance
quality question (although the quality of one impacts the achievable quality
of the other).  This is a meta-question, because guidelines on conformance
clauses are a meta-topic related to the quality of an OASIS specification.

WHY ASK: I see that we are adding material to the former Document Processing
and Conformance section (1.5 in ODF 1.1).  This is apparently in response to
the OASIS Guidelines to Writing Conformance Clauses [1] put in place in
September 2007.  The creation of a distinct conformance section seems to be
just one small part of what those Guidelines are about.  The Guidelines
address how conformance is treated as a whole.  The Guidelines are intended
to support the achievement of a particular purpose regarding the quality of
the specification and its reliability in support of the successful creation
of conforming implementations.  My question is not about the technicality of
having a conformance section with a particular format, but with regard to
what it is we hope to achieve and how we will determine that we have
accomplished that.  What is our purpose and how does it align with the
apparent purpose of the OASIS Guidelines?  

A PERSONAL CONCERN: Because we are starting from an existing series of
specifications, it also seems useful to understand how much we are taking a
step toward the aspirations that the Guidelines support and when we can say
that we've done enough for now in 1.2. That's a personal concern for me
because it determines how much work is appropriate in reviewing the current
drafts once they are stable enough for that kind of scrutiny.  If there's
some profile of the Guidelines beyond which we do not have to reach, that
helps in calibration of that effort.  It also establishes "good enough" for
this iteration of ODF specification development.

 - Dennis

PS: I notice in the OASIS Document Templates that the Conformance section is
supposed to be the last-numbered section of the specification.

REFERENCES

[1] OASIS.  Guidelines to Writing Conformance Clauses.  4 September 2007.
Available at
<http://docs.oasis-open.org/templates/TCHandbook/ConformanceGuidelines.html>
.  This document referencs [2] but not in any normative way.

[2] Lynne Rosenthal and Mark Skall (eds.).  Conformance Requirements for
Specifications v1.0, OASIS Committee Specification 15 March 2002.  Available
at <http://lists.oasis-open.org/archives/office/200902/msg00069.html>.  This
document provides sharper definitions and a statement of purpose and scope.
Although written as a statement of requirements, it is apparently an
informative document and not a requirement to be satisfied in the creation
of OASIS specifications.

ANALYSIS
========

Here are the parts of the Guidelines that I think we need to calibrate for
ourselves and to declare how we intend to address them.

PURPOSE AND CONCERNS

The guideline front matter says that 

    This document describes the purpose and scope of 
    Conformance Clauses, and associated issues that 
    Conformance Clauses shall address.

I would argue that it fails to do that, especially with regard to purpose.
The closest to a statement of scope is in [2: section 7 Conformance Clause]:

    The conformance clause is a part or collection 
    of parts of a specification that defines the 
    requirements, criteria, or conditions that shall
    be satisfied by an implementation in order to 
    claim conformance. The conformance clause identifies
    what must conform and how conformance can be met. 
    ...

followed by [2: section 7.1. Rationale for a conformance clause]:

    A conformance clause:
     * promotes a common understanding of conformance
       and what is required to claim conformance to a 
       specification,
     * facilitates consistent application of conformance
       within a specification,
     * facilitates consistent application of conformance
       across related specifications,
     * promotes interoperability and open interchange,
     * encourages the use of applicable conformance test 
       suites,
     * promotes uniformity in the development of conformance
       test suites.

In the Guidelines document, the best place to intuit the purpose is by
noticing what is achieved when the checklist has been satisfied  [1: section
6 Checklist]:

      ...
    * If you are using ISO keywords, have you explicitly 
      stated this in the specification ? 
    * Have you defined your Conformance Target(s)?
    * Are all Normative Statements clearly identifiable? 
    * Are all Normative Statements understandable, clear,
      and concise? 
    * Are all Normative Statements referenced directly 
      or indirectly from a Conformance Clause?
      Note: A Normative Statement that is not related
      to any Conformance Clause has no meaning 
    * Is each Normative Statement related to a 
      Conformance Target(s)? 

[I take this item as meaning we are able to directly determine the target(s)
for which the statement applies.  This seems particularly critical for
optional normative statements involving SHOULD and MAY in the ISO parlance,
but for the absolutely-normative statements too.]

      ...
    * Are all Conformance Clauses understandable, clear, 
      and concise? 
    * Are the top-level Conformance Clauses clearly 
      identified and related to a Conformance Target?

[This connection of targets to clauses suggests to me that we should have
separate definitions of conformance targets, each accompanied by
identification of the conformance clauses that are applicable.]

    * Are all Conformance Clauses either top-level 
      or referenced directly or indirectly from a 
      top-level Conformance Clause?
      Note: A Conformance Clause that is not related
      to any top-level Conformance Clause has no 
      meaning.

    * Are there any contradictions between Normative 
      Statements on the one hand and a Conformance 
      Clause and any referenced Conformance Clauses 
      on the other hand? If there are, have these been
      explicitly noted and have any rules to over-ride
      the contradictions been made? 

WHAT MUST CONFORM

The 2002 Conformance Requirements document does not use the "Conformance
Target" nomenclature.  However, it does specify [2: section 8.1 What needs
to conform]:

    The conformance clause identifies the "class of 
    products" (i.e., object of the claim) that will 
    be developed, where "class of product" may be an
    implementation, application, service, and/or 
    protocol (e.g., content, user agent, authoring tool).
    Additionally, the clause specifies the conditions 
    that SHALL be satisfied in order to claim conformance 
    for that class of product (i.e., make a valid claim).
    It MAY also specify that which is not a requirement. 
    There may be several classes of products that are 
    identified, each with its own conformance statement 
    or set of conformance criteria.

I take it that it is the conformance clause that defines what the
conformance targets are, not the body of the specification.  The following
clause seems particularly applicable to our efforts [2: section 8.1.1
Modularity]:

    A class of product may consist of several integrated
    components rather than a single piece of software 
    (e.g., browser). Conformance may be defined in terms 
    of the integrated components (system) and/or for each
    component. Any restrictions or constraints on the number
    or types of components that make up the "subject of a 
    conformance claim" SHALL be specified.

    For systems that are comprised of several components, 
    it may be sufficient to state that conformance to the 
    system is equivalent to conformance to all the required 
    components considered individually, and the system 
    satisfies at least the minimum conformance requirements 
    for each of those components.

THE TERMINOLOGY [1: section 2]

    Conformance Clause - A statement in the Conformance section
    of a specification that provides a high-level description 
    of what is required for an artifact to conform. It, in turn,
    refers to other parts of the specification for details. 
    A Conformance Clause must reference one or more Normative 
    Statements, directly or indirectly, and may refer to another
    Conformance Clause.

Note the requirement to reference normative statements (and we need to know
how indirectly is achieved).

    Conformance Target - an artifact such as a protocol, document,
    platform, process or service, which is the subject of 
    Conformance Clauses and Normative Statements. There may be 
    several Conformance Targets defined within a specification, 
    and these targets may be diverse so as to reflect different 
    aspects of a specification. For example, a protocol message 
    and a protocol engine may be different targets.

    Normative Statement - a statement made in the body of a 
    specification that defines prescriptive requirements on a 
    Conformance Target. 

CONFORMANCE KEYWORDS [1: section 3]

We are using the ISO keywords in normative statements.  It is important to
recognize that the very same terms used in IETF RFC 2119 have a different
sense in the ISO usage.  SHOULD and MAY are each quite different in the two
normative styles.  The ISO CAN and CANNOT have no counterpart in RFC 2119.

There are two other keywords that we need to understand for use as well [2:
7.2 Conformance key words]:

    NORMATIVE - statements provided for the prescriptive 
    parts of the specification, providing that which is 
    necessary in order to be able to claim conformance 
    to the specification. Note: the conformance scheme 
    of a specification can allow claimants to exempt 
    certain normative provisions as long as the claim 
    discloses the exemption.

[This argues against making normative references to non-normative content
and it also suggests that the ability to exempt from a normative provision
will depend on that permission being granted in the conformance clause.]


    INFORMATIVE (NON-NORMATIVE) - statements provided 
    for informational purposes, intended to assist the 
    understanding or use of the specification and shall 
    not contain provisions that are required for 
    conformance.


NORMATIVE STATEMENTS [1: section 4]

    A specification broadly consist of descriptive text and 
    Normative Statements. The Normative Statements define 
    what a Conformance Target MUST do to adhere to that 
    part of the specification, and the descriptive text 
    provides background information, descriptions and 
    examples. Descriptive text is not normative and is used
    to provide contextual information. Normative statements 
    are those that use the [conformance] keywords ..., 
    descriptive text does not use these reserved words as 
    keywords.

The wisest approach is to not use those words at all in the descriptive text
and to have the normative statements stand separately.   

CONFORMANCE SECTION [1: section 5]

This is what we have been fussing over lately.  I think it is important to
review some of the important considerations:

    ... A specification may define a number of different 
    clauses in the Conformance section, where each clause
    identifies different Conformance Targets that SHOULD 
    conform, such as an implementation, a document, an 
    authoring tool, a protocol etc. Defining more that 
    one Conformance Clause segments the specification 
    into different targets for possible Conformance. 

    A Conformance Clause identifies that to which a 
    Conformance Target MUST conform and this is done by
    reference to Normative Statements in the specification.
    A Conformance Clause therefore identifies a sub-set 
    of the Normative Statements defined in the body of a 
    specification. Thought should be put into the 
    granularity of references to Normative Statements. 
    If there are many Normative Statements referenced 
    by a Conformance Clause then simply referencing each
    statement might not be readable or easy to follow. 
    In such cases it may be better to revisit the 
    Normative Statements and group them into larger 
    referenceable units.

Notice the presumption of specifically referenceable normative statements.  

    There MAY be more than one Conformance Clause in 
    a specification, and like Normative Statements 
    they MUST be clear, concise, and unambiguous.

    Each Conformance Clause MUST be uniquely labeled.

    ... 

    If any Conformance Clause references another one,
    it is essential that there be no contradictory 
    Normative Statements within the clauses. If there 
    is a contradiction, then the writers should either 
    examine and try to remove the contradiction in the 
    specification text itself or state in the Conformance
    Clause what must be done to avoid the contradiction, 
    for example by stating that one overrides the other. 

    When multiple Conformance Clauses exist, it must be 
    clear which are the top-level. It is these top-level 
    clauses that relate to the Conformance Targets that 
    users and vendors MUST conform to, and are the clauses
    that should be referenced when claiming Conformance 
    to a specification and in making statements of use. 

    Within the Conformance section, a clear statement 
    MUST be made as to how optional Normative Statements 
    (i.e. those using the MAY keywords) must be handled. 
    This decision relates to the type of Conformance 
    Target and the use of the specifications. For example
    a document that claims Conformance to a schema does 
    not have to use any optional features. However, in 
    another scenario, a protocol target should implement 
    optional features in case another party using the 
    protocol makes use of the optional features. In 
    deciding how to dispose of optional features, issues
    that effect interoperability and portability need 
    to be considered.


[that's all]