OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

oasis-member-discuss message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [members] Request for feedback: Proposed procedure for handling public review comments

Hmmm - Several thoughts occur.

a) Why?

By which I mean, what problems does the proposed process intend to solve? Working towards ANSI accreditation by means of a defining a process doesn't help clarify what has been problematic, and how the proposed process actually fixes those problems. In other words, what are the requirements for this particular form of process, and what are the use-cases they're derived from? For example, have TCs overlooked comments? The TCs I've been involved with have been extremely thorough in dealing with these issues.

It may be that you have such materials, I just am unaware of them.

b) If the point is to make sure that *every* comment received is tracked and responded to, why isn't the solution a database & UI to track/record/update? Perhaps automate it, and send every single email is directly into the database? Then the "process", if you will, boils down to an additional item on the checklist to produce a next draft - "make sure your comments have been addressed." Everything else can be automated and reformed based on the needs of the TC Admin, without each individual TC having to jump through hoops to conform to new criteria generated by TC Admin. For example, if TC Admin needs a PDF report, add a button to the UI, and enhance the implementation to provide the PDF in exactly the form required.

As it reads, this seems like pushing more work at the TCs, much of which could be avoided. In the short-term, this might be justified by scarce resources at the TC Admin level, but long term doesn't make sense to me.

Open source projects solve what seems to be this problem by expecting that people submit bug reports. Why is this context substantively different? A form-based implementation has the benefit that it is much easier to change the information captured up front in the request.

c) The exclusion of comments that fall outside of official review periods seems spurious to me. All comments ought to be addressed. Perhaps the time frame in which they're addressed can change based on the urgency of the content, and the relevance to an upcoming review. But as a matter of "customer service" to clients of the spec, it seems like the default commitment should be to respond to all comments, regardless of when they are received. Granted, a "default" commitment doesn't sound like "MUST", or "SHALL", which might have intrinsic appeal to people on this mailing list.

d) Having read Atul Gwande's "Checklist Manifesto", I start to worry whenever I see a "process" where a checklist might do.

e) Formalism is a great way to slow down a process, and stifle honest discussion. I have observed that companies tend to put significant effort into crafting a single official comment that covers an entire specification. Which is useful. But such formality keeps out useful comments like "gee, I don't understand _____. Is it just me, or could it be clarified in some way?" - that might trigger an email thread clarifying the points of confusion. Or it could trigger an official response from the TC which boils down to "TC feels like the current text is sufficiently clear." Talk about a conversation killer. This proposed process seems to push strongly towards the latter outcome.

That's all I've got for now.


On 1/9/13 7:43 AM, Chet Ensign wrote:
OASIS members,

In December, on the chairs@ mailing list, I discussed what procedures to put in place to implement the OASIS requirements for handling comments received during public reviews. I also presented the proposed procedure with the OASIS Board's Process Committee for their information.

Below is the draft proposal, amended with the feedback I received. I am sharing it with all of you now to gain broader member feedback before implementing a final set of rules. My goal is to publish the procedure in the last week of January with the goal of making it effective for all public reviews and charter proposals beginning after February 1st. (I do not intend to make it retroactive.)

Please set aside some time over the next couple of weeks to review the proposal and share any thoughts or suggestions that you have. The final document will become the 'law of the land' so it is in all our best interests to ensure that the right balance is set between satisfying the requirements of the OASIS TC Process and minimizing the additional workload for TCs.

To make sure that the discussion is public, please use the oasis-member-discuss@lists.oasis-open.org mailing list for your feedback (cc'ed on this message). You should not need to subscribe to the list; you should be able to use it automatically. If you have a problem posting to it, let me know.

Also, attached is the comment resolution log for the feedback I received so far.

Executive summary:

We need to get better consistency in how TCs track and report comments received during public reviews. Today, TCs have no clear direction or expectations from OASIS on how this is to be done. As a result, each TC arrives at their their own solution and candidly, some do not bother to track or report comments at all.

With our strong liaison relationships with other Standards Development Organizatoins and, most recently, with the review of the OASIS Process as part of our application for ANSI accreditation, it is clear that we need a documented and consistent protocol that ensures:

(a) a TC actively acknowledges receipt of comments,
(b) at least the minimum facts needed are captured, tracked and reported,
(c) the timeframe for a TC to deliver a final comment resolution log is clear, and
(d) the reports of comments and their resolutions are persistent, immutable, publicly visible and readily found in the TC archives.

The steps outlined below seek to meet these requirements with minimal impact on TCs.


- For the purposes of this procedure, a "comment" is a statement received (a) in an email to a TC's -comment@ mailing list or (b) in the case of a TC member, in an email to the TC's primary mailing list during an announced public review period that refers to the work product under review.

In the case of a proposed charter call for comments, a "comment" is a statement received in an email to the oasis-charter-discuss@ mailing list that refers to the proposed charter. 

While a TC is free to handle any comments received at any time (and indeed is encouraged to do so), comments referring to earlier drafts or general comments unconnected with a public review underway do not fall under the requirements laid out below.

- For the purposes of this procedure, the "the end of the review period" mentioned in several places in the TC Process as the deadline for posting a link to a comment resolution log shall be deemed to be the point in time at which the TC next requests an action on a document (e.g. the next public review, a Special Majority Vote to approve a Committee Specification).

- The TC Process refers to the record of comments and their resolutions in several different ways. For the purposes of this procedure, the document that records comments and their dispositions will be called a "comment resolution log."

Draft Procedure:

1) TC Obligations With Respect to Comments

The requirements for handling comments are laid out in sections 2.2, 2.8, 3.2 and 3.4 of the OASIS TC Process. Those requirements are:

- That comments be received only through the TC's <short name>-comment@ mailing list for non-members, the TC's primary mailing list for members, and the oasis-charter-discuss@ mailing list for comments on proposed charters.

- That comments be received by the Chair and optionally by one or more other members of the TC.

- That comments be acknowledged.

- That comments be tracked and their resolutions documented:

a) During a proposed TC call for comment period;
b) During a public review of a TC work product;
c) During a public review of a Candidate OASIS Standard.

- And that at the end of the comment / review period, the TC post to its email list a pointer to an account of each of the comments raised during the comment period along with its resolution.

2) Procedure for Meeting These Obligations

To meet the requirements, TCs must take the following steps:

a) The TC Chair is automatically subscribed to the TC's <tcname>-comment@lists.oasis-open.org mailing list and will always receive emails sent to that list. TCs may want to assign another member (e.g. the Secretary or editor)to subscribe to the list and track and report comments and issues back to the TC.

b) Someone from the TC must send an email back to the commenter acknowledging the comment. A simple "thank you for your comment; the TC will be consider it" message is sufficient.

c) The TC must track each comment received and how the group decides to resolve it. Per the TC Process, a TC is not required to take any action on the substance of the comment, but the decision not to must be recorded.

The TC can choose to track comments and resolutions however they wish: in a word processing document, in a spread sheet, in JIRA, in the TC wiki. The TC must at a minimum record:

- The date the comment was received;
- A link to the email in which a comment was received;
- The name of the person or entity providing the comment;
- A brief summary of the comment;
- A statement of the TC’s decision on how to handle the comment, in as much detail as the TC wishes to provide.

In the event that multiple emails raise the same issue, the TC can track and resolve them with one entry in the document, however a link to each of the emails and the names of each of the commenters must be included in the record.

Keeping a cumulative list of comments will likely to be the most convenient approach. The TC Process states that, before an OASIS membership ballot can be held on a Candidate OS, the TC must provide "a pointer to an account of each of the comments/issues raised during the public review period(s), along with its resolution." Having all the comments and their resolutions stored in one list will make satisfying this requirement easy.

d) By the end of the review period at the latest, the TC must post to its email list a document containing, at a minimum, the pieces of information listed above. The document can be either a word processing document, a spread sheet or a PDF document. A stand-alone document is required to ensure that the comment resolution log is persistent and immutable and publicly available on the OASIS system.

If the TC is using JIRA or a wiki to track comments, a document must be created by exporting the information from that source.

If the document is a cumulative list of comments and resolutions, it should also contain, for each comment, identification of the draft to which it applied.

The posting email can be created by loading the document into the TC's document repository and using the automatically generated email to produce the pointer or by attaching the document directly to the email. The TC may also forward a pointer to the document to the TC's -comment@ mailing list if it wishes to do so.

With that email, the TC obligations will be considered fully met.

3) What TC Administration Will Do With the Comment Resolution Document

The key objective is to ensure that comments and their resolutions can readily be found by others who are interested in the TCs work. Here is what TC Admin will do with the documents.

a) When the TC is ready to take another action on its work product (e.g. request another public review, request a ballot to approve a Committee Specification), a pointer back to this document will have to be provided as part of the support request. TC Administration will not proceed with a requested action until the comment resolution document is provided.

b) A link to the comment resolution log will be included in the announcements of subsequent public reviews, document approvals, Special Majority ballots and other communications where it will provide additional, useful information.

c) A copy of the comment resolution log will be stored in the OASIS Library along with the public review draft to which it applies. This will help to ensure transparency and traceability between comments received during a public review and their resolution.

--- / ---

Chet Ensign
Director of Standards Development and TC Administration
OASIS: Advancing open standards for the information society

Primary: +1 973-996-2298
Mobile: +1 201-341-1393

This email list is used solely by OASIS for official consortium communications.
Opt-out requests may be sent to member-services@oasis-open.org, however, all members are strongly encouraged to maintain a subscription to this list.

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]