Code list extensibility and substitution groups

[jon.bosak@sun.com]

Hello ubl-dev,

The OASIS UBL Technical Committee is considering a possible
addition to the UBL 1.0 Code List specification and would like the
opinion of the XML schema experts and business experts on the
ubl-dev list before coming to a decision.  The TC has authorized
me to moderate a discussion on this list beginning today and
ending Monday 28 February in order to gain your input.

At issue is one of the possible future requirements listed by
Marty Burns in his message of 8 February at

   http://lists.oasis-open.org/archives/ubl-dev/200502/msg00003.html
   http://lists.oasis-open.org/archives/ubl-dev/200502/pdf00000.pdf

In the current discussion draft, that requirement reads as follows:

   2.5.8 [R26 (Future)] The ability to import, extend, and/or
   restrict values and elements of other code lists

      The model for code lists must provide the ability to extend,
      restrict or import additional values and/or elements of
      other code lists.

      Each code list and the format used to represent it must
      support derivation of descendant code lists.

      Derivation in this context shall include adding and/or
      removing notional codes and/or sets of values associated
      with the list as well as adding and/or removing keys,
      descriptive information, etc.

      Any such derivation shall be done in a deterministic fashion
      that is repeatable and auditable (see [R17], [R21]).

(The present author is not sure about the meaning of the word
"other" in the language above, but this can probably be considered
an editorial issue.)

The UBL 1.0 Code List specification, released as part of UBL 1.0
in November 2004, contains a section (4.8) that describes a
mechanism for meeting this requirement through the use of XSD
substitution groups.  You can find the UBL 1.0 Code List
specification at

   http://docs.oasis-open.org/ubl/cd-UBL-1.0/doc/cl/wd-ublclsc-codelist-20040420.pdf

However, this section was labeled "future" to indicate that the
UBL TC had not yet decided whether to adopt it.  As noted in
Section G.1.2 of the UBL 1.0 Standard, "Substitution Groups for
Code List Customization,"

   The UBL Code List Subcommittee has produced a comprehensive
   solution for code lists (see Appendix E) that relies upon XSD
   substitution groups for code list customization. Lacking a
   clear industry consensus on the use of XSD substitution groups
   in business document schemas, the UBL TC has deferred the
   adoption of this extension mechanism for code lists pending
   further discussion. Care has been taken to construct UBL 1.0 in
   a way that will allow the adoption of substitution groups (if
   deemed appropriate) in later releases without invalidating UBL
   1.0 instances.

The TC is now attempting to dispose of this issue so that we can
freeze changes to our schema generator and continue the
development of UBL 1.1.

It should be understood that the purpose of the UBL Code List
specification is to provide a single form of representation for
the code lists published by various code list maintainers (ISO,
BSI, DIN, etc.) so that efforts such as UBL can include standard
code lists by reference rather than including them directly as was
done in UBL 1.0.  It is generally accepted that such a format is
needed, and the UBL 1.0 Code List specification is widely regarded
as a good start.  The issue of code list extensibility is one of
the few large problems remaining to be solved before this format
can be recommended for general adoption.

There are three key questions to be decided here:

 - Is it necessary to provide for the kind of extensibility
   described in Requirement 26?  In other words, is ad hoc
   extensibility of standard code lists really a requirement?
   This is basically a business question.

 - If it is a requirement, is the substitution group mechanism of
   XSD the right way to meet it?  (The UBL 1.0 Naming and Design
   Rules explicitly forbid the use of substitution groups in other
   UBL schemas, so adoption of subsitution groups to provide for
   code list extension would mean that an exception to this rule
   would have to be made for code list schemas.)  This is
   basically a W3C Schema philosophy question.

 - If substitution groups are indeed an acceptable mechanism for
   this purpose, do currently available XML tools actually provide
   sufficient support for it?  This is a question of fact upon
   which we lack sufficient data.

In a preliminary discussion of this issue at the recent UBL TC
meeting in McLean, the following use cases were identified:

   1. It is necessary to add a value to a standard code list, or
      remove one, or both.  Sometimes this change should
      eventually be reflected in an updated version of the
      standard code list itself (for example, the addition of a
      new country code), and sometimes it should not (for example,
      the inclusion of a nonstandard currency code to accommodate
      a particular trading scenario, or the removal of certain
      country codes to implement national trade restrictions).

   2. A standard code list is revised, but users must support an
      old code value alongside the new one.  For example, the
      value "piece" in UN Recommendation 20, Units of Measure, is
      represented by the code "PCE" in older versions and the code
      "C62" in newer versions.

   3. Different trading partner agreements may require different
      restrictions or extensions of the same standard code list.

The discussion revolved around the following points:

 - Supporters of the substitution group mechanism maintain that
   users need the ability to modify standard code lists in a way
   that minimizes changes to the UBL document schemas in which
   they are referenced.

 - Opponents of the mechanism maintain that modifications to
   agreed-upon standard lists should not be made invisibly, and
   they observe that this problem has formerly been solved in EDI
   systems simply by explicitly substituting a revised code list
   for the old one.

 - Supporters reply that this simple swap was possible only
   because the code list schema was not itself being used to
   implement validation.  Further, they claim that it is
   impossible within the limits of XSD to make such a swap in UBL
   without making changes to all of the schemas that reference the
   revised code list.

The TC concluded that this last point is impossible to prove
one way or the other without explicit examples and that an email
discussion such as this one is the appropriate medium in which to
display such examples.  Three test cases were suggested:

   1. Add a currency code to the standard list used in UBL
      Invoice.

   2. Restrict the list of currency codes accepted in a UBL
      Invoice to just two values.

   3. Do both (restrict the list of codes and also add some new
      ones).

We now turn to the business and schema experts in ubl-dev for help
in exposing all sides of this problem.  Please note the following
points:

 - In order to provide schema validation of code list values found
   in UBL instance documents, UBL provides its standard code lists
   as XSD schemas in which the code values are enumerations.  For
   this reason, the UBL Code List specification does not provide a
   fixed schema for code lists, but rather a template for such
   schemas.

 - However, it is assumed that no actual UBL software will rely
   upon schema validation alone to implement data checking; there
   will always be at least one layer of logic, and probably more,
   to implement business rules relating to codes (for example, a
   validator can check that a country code is valid and a currency
   code is valid, but it cannot check that a specific currency
   code is the one that is allowed for a specific country).  So
   the relative importance of code list validation itself remains
   a possible question in this discussion.

 - The discussion so far has been hindered by the lack of concrete
   examples prepared by genuine W3C Schema experts.  Our greatest
   need at this point is for valid examples based on the UBL 1.0
   schemas that will show the actual implementation of the
   strategies we've been discussing (see the test cases suggested
   above).

We are asking for input on the ubl-dev list in order to get the
widest possible range of views on this question into the public
record.  If you know of a person or group that has an interest in
code lists, please point them to this discussion and ask them to
subscribe.  Note again that we must stop taking input at the end
of February in order to maintain the UBL 1.1 release schedule.

Jon Bosak
Chair, OASIS UBL Technical Committee