Re: [ubl-dev] Code list extensibility and substitution groups

["David Webber \(XML\)" <david@drrw.info>]

William,

Well put clarifications.  I'm looking to paraphrase this here - 
if the schema contains the "all possible results set" - in terms
of both the structure and the allowed values - then you
can use tools like CAM to augment the schema and serve
as the business agreement rules - that enforce the deltas.

Ideally in an ebXML world this goes from the BPSS step
that is identifying a business transaction document (in our case
the UBL transaction) - with a context conditional of 
- majorCurrencyOnly - to the CPA agreement on 
service/actions between partners, to the ebXML envelope, 
that the ebMS then recognizes, and references the CPA 
isntance in the Registry - that then tells it that the validation 
script for that document type is the CAM template.

As you point out - this avoids getting tangled in schema
mechanics that are almost certain to be hard to diagnose
or implement - and puts the logic where it needs to be
as part of the business agreement layer - using the
CAM template to capture that context and detail.

Thanks, DW


----- Original Message ----- 
From: "William J. Kammerer" <wkammerer@novannet.com>
To: <ubl-dev@lists.oasis-open.org>
Sent: Friday, February 18, 2005 8:15 AM
Subject: Re: [ubl-dev] Code list extensibility and substitution groups


> I'm not going to outright disagree with my good friend David RRR Webber
> (I've promoted him to have 3 middle initials, 2 short of the future
> King). But I'm not sure "ad hoc extensibility of standard code lists
> [is] really a requirement."
> 
> Putting aside versioning, if a value is constrained to contain an ISO
> 4217 currency code, then I rightfully assume any of its values can be
> used to form a valid document, for example. Now, it may be a business
> requirement that the supplier only accepts prices in USD and EUR, but it
> would be truly frustrating to have my XML tools or translator bark at me
> because I used GBP - what to me appears to be a perfectly sound, hard
> currency. It would be hard to trace back through the rat's nest of
> schema and overrides to see just why this is happening - why a valid ISO
> 4217 code is causing me conniptions. IF IT'S A PAROCHIAL BUSINESS
> REQUIREMENT, THEN SCHEMA DO NOT SEEM TO BE THE APPROPRIATE PLACE TO HIDE
> THE REQUIREMENT. I guess I'd rather see the requirement that only
> Dollars and Euros are used by my trading partner within an ancillary
> piece of documentation - or even in a response message saying "I
> expected only EUR and USD."
> 
> On the other hand, code value restriction might have found a place in
> X12 and EDIFACT EDI implementation guidelines because codes were used
> all over the place to do things names are used for in XML - e.g., to
> differentiate delivery dates. The only way you could make the DTM
> segment make sense is if you "restricted" (in the printed guideline) the
> code value list to the 2 or 3 sensible values out of an EDIFACT codelist
> containing hundreds. But a Delivery core component gives you only those
> "Delivery" dates that make sense in that context, e.g.,
> ActualDeliveryDateTime and RequestedDeliveryDateTime. There's certainly
> no need for code restriction in UBL to differentiate Delivery dates,
> because there are no codes - only element names. But that's the primary
> reason folks restricted code lists in EDI; why does it have to be
> carried over to core components and XML?
> 
> William J. Kammerer
> Novannet
> Columbus, OH 43221-3859 . USA
> +1 (614) 487-0320
> 
> ----- Original Message ----- 
> From: "David Webber (XML)" <david@drrw.info>
> To: <jon.bosak@sun.com>; <ubl-dev@lists.oasis-open.org>
> Sent: Tuesday, 15 February, 2005 12:36 PM
> Subject: Re: [ubl-dev] Code list extensibility and substitution groups
> 
> 
> Jon,
> 
> This of course has been EDI 101 since time began.
> 
> Good that we are moving to tackle these issues.
> I've provided my feedback <resp/> below.
> 
> I believe the new ebXML solution stack including
> BPSS V2 and ebMS V3 and registry V3 is well
> equipped to solve the needs here - and my
> thought would be to lean there first, and let
> W3C XSD catchup later - if and when it indeed
> does.
> 
> Thanks, DW
> 
> >
> > There are three key questions to be decided here:
> >
> >  - Is it necessary to provide for the kind of extensibility
> >    described in Requirement 26?  In other words, is ad hoc
> >    extensibility of standard code lists really a requirement?
> >    This is basically a business question.
> 
> <resp> Absolutely - and preferrably context / role driven. </resp>
> 
> >
> >  - If it is a requirement, is the substitution group mechanism of
> >    XSD the right way to meet it?  (The UBL 1.0 Naming and Design
> >    Rules explicitly forbid the use of substitution groups in other
> >    UBL schemas, so adoption of subsitution groups to provide for
> >    code list extension would mean that an exception to this rule
> >    would have to be made for code list schemas.)  This is
> >    basically a W3C Schema philosophy question.
> 
> <resp> Schema is just not equipped for this, and I'm not
>     sure the W3C want to support B2B needs.  Maybe that
>     has changed - but historically their position has been
>     "not our problem". The biggest *gotcha* is versioning.
>     Even if their mechanism does partly do what you want;
>     EDI 101 teaches us that versioning is essential as
>     codelists change - and versioning at the element /
>     attribute level again is a blindspot for XSD.</resp>
> 
> >
> >  - If substitution groups are indeed an acceptable mechanism for
> >    this purpose, do currently available XML tools actually provide
> >    sufficient support for it?  This is a question of fact upon
> >    which we lack sufficient data.
> >
> > In a preliminary discussion of this issue at the recent UBL TC
> > meeting in McLean, the following use cases were identified:
> >
> >    1. It is necessary to add a value to a standard code list, or
> >       remove one, or both.  Sometimes this change should
> >       eventually be reflected in an updated version of the
> >       standard code list itself (for example, the addition of a
> >       new country code), and sometimes it should not (for example,
> >       the inclusion of a nonstandard currency code to accommodate
> >       a particular trading scenario, or the removal of certain
> >       country codes to implement national trade restrictions).
> <resp> Versioning is crucial as noted above.  ebXML registry can
>            fullfil this much better - using business-noun definitions
> and
>            referencing them via LID(UBL-id) lookups to retrieve the
>            actual codelist at runtime.  V3 of Registry supports this.
> </resp>
> >
> >    2. A standard code list is revised, but users must support an
> >       old code value alongside the new one.  For example, the
> >       value "piece" in UN Recommendation 20, Units of Measure, is
> >       represented by the code "PCE" in older versions and the code
> >       "C62" in newer versions.
> <resp> Switching between code values requires transform technology.
>             This is why OASIS CAM templates are designed to augment
>             schema by providing exactly this functionality.
> </resp>
> >
> >    3. Different trading partner agreements may require different
> >       restrictions or extensions of the same standard code list.
> <resp>Context and role is crucial. And the ability to reference
>            between two codelists - eg <VehicleType> and <EmissionCode>
>            The value selected in list #1 - effects the allowed values in
>            list #2.  I believe that schema cannot support that.  Again
>            OASIS CAM templates do.  And BPSS V2 allows you to
>            provide context and role to the ebMS so that tools like
>            CAM can inspect that and apply rules accordingly.  This
>            is clearly an area where CAM augments schema and a
>            simple XML parser simply is not equipped to support
>            this extended behaviour.
> </resp>
> >
> > The discussion revolved around the following points:
> >
> >  - Supporters of the substitution group mechanism maintain that
> >    users need the ability to modify standard code lists in a way
> >    that minimizes changes to the UBL document schemas in which
> >    they are referenced.
> <resp>Issues with versioning and cross-referencing between
>            dependent codelists</resp>
> >
> >  - Opponents of the mechanism maintain that modifications to
> >    agreed-upon standard lists should not be made invisibly, and
> >    they observe that this problem has formerly been solved in EDI
> >    systems simply by explicitly substituting a revised code list
> >    for the old one.
> <resp>Concur.  This is why we have CAM - so that "hidden"
>          rules can be freely shared between partners in a systematic
>          and easy to read format (simply XML scripted syntax)
>          that can be agreed to and referenced via the CPA and
>          stored in registry.
> </resp>
> >
> >  - Supporters reply that this simple swap was possible only
> >    because the code list schema was not itself being used to
> >    implement validation.  Further, they claim that it is
> >    impossible within the limits of XSD to make such a swap in UBL
> >    without making changes to all of the schemas that reference the
> >    revised code list.
> <resp>Correct!  You have to *de-couple* this and prevent tight
>  coupling - otherwise you have a maintenance and implementation
>  nightmare.  Moving the codelists to a separate representation within
>  registry would solve this - and then letting people use their own
>  validation tools - such as CAM or XSLT - to resolve those.
> </resp>
> 
> >
> > The TC concluded that this last point is impossible to prove
> > one way or the other without explicit examples and that an email
> > discussion such as this one is the appropriate medium in which to
> > display such examples.  Three test cases were suggested:
> >
> >    1. Add a currency code to the standard list used in UBL
> >       Invoice.
> >
> >    2. Restrict the list of currency codes accepted in a UBL
> >       Invoice to just two values.
> >
> >    3. Do both (restrict the list of codes and also add some new
> >       ones).
> >
> <resp> I would add the fourth case - inter-related codelists - where
>             values selected in second codelist depend on the value(s)
>             selected in the first codelist.
> </resp>
> 
> > We now turn to the business and schema experts in ubl-dev for help
> > in exposing all sides of this problem.  Please note the following
> > points:
> >
> >  - In order to provide schema validation of code list values found
> >    in UBL instance documents, UBL provides its standard code lists
> >    as XSD schemas in which the code values are enumerations.  For
> >    this reason, the UBL Code List specification does not provide a
> >    fixed schema for code lists, but rather a template for such
> >    schemas.
> <resp>Preferably accessible from Registry</resp>
> >
> >  - However, it is assumed that no actual UBL software will rely
> >    upon schema validation alone to implement data checking; there
> >    will always be at least one layer of logic, and probably more,
> >    to implement business rules relating to codes (for example, a
> >    validator can check that a country code is valid and a currency
> >    code is valid, but it cannot check that a specific currency
> >    code is the one that is allowed for a specific country).  So
> >    the relative importance of code list validation itself remains
> >    a possible question in this discussion.
> <resp> Precisely!  The schema is there to denote the structural
>  premutations.  More vigourous business rule driven validation
>  engines - such as CAM - can then be used to augment schema.
>  Providing a cheat-sheet of validation engines can then give people
>  their implementation options - in BPSS we have
>  XPATH, XSLT, CAM as a base set.
> </resp>
> >
> >  - The discussion so far has been hindered by the lack of concrete
> >    examples prepared by genuine W3C Schema experts.  Our greatest
> >    need at this point is for valid examples based on the UBL 1.0
> >    schemas that will show the actual implementation of the
> >    strategies we've been discussing (see the test cases suggested
> >    above).
> >
> > We are asking for input on the ubl-dev list in order to get the
> > widest possible range of views on this question into the public
> > record.  If you know of a person or group that has an interest in
> > code lists, please point them to this discussion and ask them to
> > subscribe.  Note again that we must stop taking input at the end
> > of February in order to maintain the UBL 1.1 release schedule.
> >
> <resp> I believe we have the means to solve this without depending
>            on W3C to cover this off - either in the short-term, or the
>            long term - and certainly not by Feb 28th!
> </resp>
> 
> Thanks, DW
> 
> 
> 
>