OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.


Help: OASIS Mailing Lists Help | MarkMail Help

office message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]

Subject: Re: [office] Fwd: New Last Call: 'Tags for Identifying Languages' toBCP

Hi David,

David Faure wrote:
> This might be relevant for us since we use fo:language to specify the language
> of a run of text. Not for switching to it yet, of course, better keep following XSL
> for now, but just in case any of you has input on the IETF draft.
> On this topic, I just noted that our fo:language is validated with [A-Za-z]{1,8} 
> (languageCode definition)
> This basically means it's "an RFC3066 language code" but without country code.


> Shouldn't we allow things like fr_CA? (or is that fr-CA ? I'm confused by the RFC
> talking about a hyphen, I thought it was an underscore).

My understanding of RFC3066 is that it uses a hyphen. The type
specifications for "language", "languageCode" and "countryCode" have
been derived directly from RFC3006 and XSL-FO.

RFC3066 specifies a language as

>The syntax of this tag in ABNF [RFC 2234] is:
>    Language-Tag = Primary-subtag *( "-" Subtag )
>    Primary-subtag = 1*8ALPHA
>    Subtag = 1*8(ALPHA / DIGIT)
>   The productions ALPHA and DIGIT are imported from RFC 2234; they
>   denote respectively the characters A to Z in upper or lower case and
>   the digits from 0 to 9.  The character "-" is HYPHEN-MINUS (ABNF:
>   %x2D).

This definition is what we use for the type "language".

In XSL, the datatype used for the language attribute is summarized as

> A language-specifier in conformance with [RFC3066].


> The language may be the language component of any RFC 3066 code (these
> are derived from the ISO 639 language codes).

That's from my understanding the "primary-subtag" of RFC 3006, that must
not contain a hyphen (or underscore).

However, my interpretation of XSL-FO may be wrong.


> ----------  Forwarded Message  ----------
> Subject: New Last Call: 'Tags for Identifying Languages' to BCP
> Date: Thu, 9 Dec 2004 09:56 am
> From: The IESG <iesg-secretary@ietf.org>
> To: IETF-Announce <ietf-announce@ietf.org>
> The IESG has been considering
> - 'Tags for Identifying Languages '
>    <draft-phillips-langtags-08.txt> as a BCP
> There have been considerable changes to the document since the
> initial last call, and the IESG would like the community to consider
> the changes.  In addition, the authors have prepared text describing
> why this mechanism is needed as a replacement for the existing
> procedure; it is included below.
> The IESG plans to make a decision in the next few weeks, and solicits
> final comments on this action.  Please send any comments to the
> iesg@ietf.org or ietf@ietf.org mailing lists by 2005-01-05.
> The file can be obtained via
> http://www.ietf.org/internet-drafts/draft-phillips-langtags-08.txt
> Author's discussion of drivers for this work:
> Reasons for Enhancing RFC 3066
> RFC 3066 and its predecessor, RFC 1766, define language tags for use on the
> Internet. Language tags are necessary for many applications, ranging from
> cataloging content to computer processing of text. The RFC 3066 standard for
> language tags has been widely adopted in various protocols and text formats,
> including HTML, XML, and CLDR, as the best means of identifying languages and
> language preferences.
> This specification proposes enhancements to RFC 3066. Because revisions to
>  RFC 3066 therefore have such broad implications, it is important to
>  understand the reasons for modifying the structure of language tags and the
>  design implications of the proposed replacement.
> Problems
> This specification, the proposed successor to RFC 3066, addresses a number of
> issues that implementers of language tags have faced in recent years:
>     * Stability of the underlying ISO standards
>     * Accessibility of the underlying ISO standards for implementers
>     * Ambiguity of the tags defined by these ISO standards
>     * Difficulty with registrations and their acceptance
>     * Identification of script where necessary
>     * Extensibility
> The stability, accessibility, and ambiguity issues are crucial. Currently,
> because of changes in underlying ISO standards, a valid RFC 3066 language tag
> may become invalid (or have its meaning change) at a later date. With much of
> the world's computing infrastructure dependent on language tags, this is
>  simply unacceptable: it invalidates content that may have an extensive
>  shelf-life. In this specification, once a language tag is valid, it remains
>  valid forever. RFC 3066 Language Tags: A brief survey
> Tags defined by RFC 3066 take two forms. Most tags are formed using an ISO
> 639-1 (two-letter) or ISO 639-2 (three letter) language tag, optionally
>  followed by an ISO 3166 country code. Tags formed in this manner are not
>  individually registered and anyone can use such a combination of codes to
>  identify their language preferences or the language of some piece of
>  content. Because this system allows a broad range of tags to be formed by
>  reference to the underlying standards, these tags are referred to as
>  generative in nature. The generative system is very powerful and allows
>  content authors and others to form and use very expressive tags without the
>  need to engage in a long and arduous registration process. Examples of such
>  tags are:
>     * en-US (English as used in the United States)
>     * fr-CA (French as used in Canada)
>     * de-CH (German as used in Switzerland)
>     * ja (Japanese)
>     * ale-CA (Aleut as used in Canada)
>     * ale-BE (Aleut as used in Belgium)
> While it is possible to generate tags that do not identify any likely
> real-world content, such as Aleut as used in Belgium, tags of this nature do
>  not represent a serious problem. Consider the case of a database that can
>  identify people by national origin and by hair color. It is not a problem
>  that one could compose a query for blond Mongolians, even though no results
>  would ever be returned.
> There are problems with the the RFC 3066 definition of generative tags,
> however. The ISO 639 and ISO 3166 standards are not freely available and
>  evolve over time. For example, ISO 3166 has withdrawn tags in the past and,
>  worse, then reassigned them to a different country altogether. As a result,
>  it is difficult for implementers to obtain a correct list of codes and then
>  ensure
> interoperability with other implementations of language tags.
> The other way to form an RFC 3066 tag is via registration with IANA. Tags
> registered with IANA identify a specific language, dialect or variation.
>  Unlike the generative tags, the registered values cannot be combined with
>  other standard subtags to form additional tags that are more descriptive.
>  Examples of such tags are:
>     * no-nyn (Nynorsk variation of Norwegian,
>               deprecated: use 'nn' instead)
>     * cel-gaulish
>     * i-klingon (deprecated: use 'tlh' instead)
>     * etc.
> Registration, besides being a long and arduous process, also presents a
>  variety of problems for implementers. Although the tags are freely
>  available, most implementations do not support these tags because they do
>  not fit neatly into the generative system. Special logic is required to
>  handle them, especially when performing language negotiation or fallback. In
>  addition, many of the tags are deprecated because the registration process
>  is less opaque and time-consuming than registering a language with ISO 639
>  MA/RA has historically been. Eventually ISO 639 does catch up and assign the
>  language a code, resulting in overlapping tag choices. Implementations must
>  also deal with the implications of multiple valid tags identifying what is
>  essentially the same content.
> But most problematic is the lack of a relationship to the generative
>  mechanism. Since each variation of a tag must be separately registered,
>  language variations with a broad range of valid uses require an enormous
>  number of registrations. For example, there are 8 registrations to deal with
>  minor spelling reforms in the German language and these registrations cover
>  just three countries where German is commonly spoken--and no countries where
>  it is not the major language. Variations in languages with a broader
>  diffusion (such as Chinese) may require 20 or more registrations to gain
>  full coverage, sometimes of important distinctions.
> Solving the Problems
> This specification addresses each of these issues with a simple, elegant
>  design that is compatible with existing language tags and implementations.
> This compatibility exists on several levels. All language tags, both
>  generative and registered, that were valid under RFC 3066 are still valid
>  under this specification. In addition, and very importantly, language tags
>  that are newly defined by this specification are compatible with the ABNF
>  syntax, matching, parsing, and other mechanisms defined by RFC 3066.
> Thus for an implementation of RFC 3066, all of the new tags defined by this
> specification are still in the form of valid registered tags, and will simply
>  be dealt with in whatever fashion the implementation used to handle future
>  registrations, those that were added to the registry after the
>  implementation was created. In other words, tags formed under this
>  specification that are unfamiliar to RFC 3066 implementations will be
>  treated by those implementations as if they were registered tags from a
>  future version of the 3066 registry.
> Subtags and the Registry
> The largest change in the specification is that it modifies the structure of
> the language tag registry. Instead of having to obtain lists of codes from
>  five separate external standards (not all of which are easily available),
>  the IANA registry will maintain a comprehensive list of valid subtags that
>  can be used in the generative mechanism in a machine-parseable text format.
>  This registry will continue to track the existing core standards and will
>  start with the current list of valid codes. As future codes are assigned,
>  the IANA registry will be updated to reflect the changes.
> Having a separate registry allows IANA language tags to resolve ambiguity and
> stability problems with the underlying standards. Language tags formed today
> will be guaranteed to maintain their validity and meaning essentially
>  forever, something that is not true today.
> In addition, switching to a subtag registry changes the nature of
>  registrations themselves. Instead of registering complete tags and therefore
>  potentially having to register a very large number of them (complicating
>  life for implementers and discouraging support for the registry), a single
>  subtag can be generatively combined to form many useful tags.
> For example, one registered tag today is 'zh-Hans', which represents "Chinese
> written in the Simplified Chinese script". Only this tag is valid under RFC
> 3066. Useful tags such as 'zh-Hans-SG' (SG=Signapore) or 'zh-Hans-CN' are not
> valid. By switching to a registry in which 'Hans' is a registered subtag, any
>  of these valid and useful tags can be formed generatively.
> In addition, the subtag registry will encourage implementers to support
> registered items, since the subtags will fit the generative mechanism and
> exception handling code will no longer be necessary.
> To prevent the IANA language registry filling up with deprecated entries,
>  rules have also been introduced to curb harmful registrations that should be
>  handled by the various ISO maintenance and registration authorities (such as
>  ISO 639).
> The new structure and registry allows implementations to determine much more
> about tags, even in the absence of registry information. This is important
> because at any given point in time there will be a mixture of implementations
> that have different snapshots of the registry. The new structure allows these
> implementations to to interoperate effectively. In particular, the category
>  of all subtags (as language, region, script, etc.) can be determined without
>  reference to the particular version of the registry snapshot by the
> implementation. This allows for much more robust implementations, and greater
> compatibility over time.
> In addition, this specification also makes it possible, for the first time,
>  to effectively test whether an implementation conforms to the specification.
>  The problem with RFC 3066 is that to determine the status of an
>  implementation produced at a given point, one has to reconstruct the
>  historical contents of each of the ISO standards and the historical contents
>  of the registry. This is a time-consuming and error-prone process. The new
>  registry provides a complete, easily parseable file which provides the
>  precise the contents of valid tags for any point in time.
> Additional Subtag Sources
> This specification introduces two additional international standards as
>  sources for language tags.
> ISO 15924 represents script codes. (The example above of 'Hans' is from ISO
> 15924.) Writing system variations are often crucial to communicate,
>  especially when selecting content using language negotiation. Addition of
>  this standard will allow these distinctions to be formed generatively,
>  rather than via individual registration.
> UN M.49 represents region and country codes. The UN M.49 standard is used by
> ISO 3166 to determine what a country is. The UN M.49 codes are used by this
> specification in two ways. First, if ISO 3166 reassigns a country code
>  formerly associated with one country to another country (as it did in 2001
>  with the 'CS' code, formerly Czechoslovakia and now assigned to Serbia and
>  Montenegro), then the UN M.49 code can be placed in the registry to preserve
>  stability. Secondly, the UN M.49 standard defines regional codes for areas
>  such as "Central and South America" which can be useful in forming language
>  tags for larger regions.
> Future-Proofing: Private Use and Extensions
> Because of the widespread use of language tags, it is potentially disruptive
>  to have periodic revisions of the core specification, despite demonstrated
>  need. This specification addresses this problem by fully specifying the
>  valid syntax of language tags, while providing for future, unforeseen,
>  requirements. One of these mechanisms is the extlang subtags, which allows
>  for future extensions of ISO 639, in particular, ISO 639-3.
> Private use subtags is another one of these mechanisms. In RFC 3066, any tag
> that was not registered or wholly made up of generative subtags must be
> completely tagged as private use. Recipients of such a tag are not allowed to
> infer any information from such a tag, except by private agreement. Thus if
>  any private-use information needed to be included in the tag, the entire tag
>  had to be private use; making the entire tag uninterpretable to other
>  implementations.
> This specification allows for private use subtags in a particular, prescribed
> manner. Consider the IANA registered tag 'sl-nedis', which represents the
> Natisone dialect of Slovenian. The subtag 'sl' is a valid ISO 639-1 code for
> Slovenian. Prior to its registration with IANA, if users wished to tag
>  content as being in the Natisone dialect, they had two choices for language
>  tags: 'sl' and 'x-sl-nedis' (or similar). The first tag does not meet the
>  need of distinguishing the text from other varieties of Slovenian, while the
>  second one does not convey the relationship to Slovenian to outside
>  processors (a human might look at the tag and infer Slovenian, but the 'sl'
>  subtag doesn't necessarily represent that language).
> Under this specification, if a new dialect of Slovenian were needed (let's
>  call it the 'xyzzy' dialect), a tag such as 'sl-x-xyzzy' can be used. In
>  fact, a quite comprehensive amount of information can be communicated:
> 'sl-Latn-IT-x-xyzzy' would represent Slovenian written using the Latin script
>  as used in Italy with some additional private distinguishing information
>  (which implementations of this specification can match algorithmically).
> Note that RFC 3066 private use tags are still permitted and have the same
> information content and treatment as they did previously.
> The extension mechanism also provides a way for independent RFCs to define
> extensions to language tags. These extensions have a very constrained,
> well-defined structure to prevent extensions from interfering with
> implementations of this specification (or RFC 3066).
> Matching and Language Negotiation
> Content tagging is only one of the applications for language tags. The other
> major applications are querying for for matches and in content negotiation.
>  RFC 3066 defines "language ranges" for use in content negotiation and
>  querying and describes a very simple matching algorithm. This specification
>  maintains compatibility with this language negotiation scheme, while
>  providing additional information on the implementation of language matching.
> Well-Formed vs. Validating
> Existing language tag processors already fall into two categories. There are
> language tag processors that check if language tags have the proper,
> well-formed, syntax, but which do not validate their content, and there are
> language tag processors that in addition validate and reject unrecognized
>  tags. Each of these categories is appropriate to different implementations.
>  For example, to process incoming tags that may have been formed under a
>  future registry, an implementation may restrict itself to only checking
> well-formedness. Another implementation that allows users to generate tags
>  may fully validate.
> This specification clearly distinguishes these two possible classes of
> conformance, and provides an explicit, testable definition of each one.
> Impact of the New Design on Existing Implementations
> One concern that is crucial to acceptance of the new language tag design is
>  how it works with existing implementations of RFC 3066 and how existing
> implementations will interact with implementations of the newer language
>  tags.
> It is important to recognize that all language tags that were valid under the
> existing RFC 3066 will remain valid, with their meanings intact, under this
> specification. In fact, this specification stabilizes these meanings so that
> existing implementations can be continued forward for as long as it
>  necessary. Content, regardless of its format, will remain valid, essentially
>  forever.
> As content and systems begin to make use of the new language tags by adopting
> the additional fields defined by this specification, there will be an impact
>  on software and systems that expect only the older tags. The design of this
>  specification was carefully created so that all of the new values that can
>  be assigned fit the pattern for registered language tags under RFC 3066.
>  Thus while existing implementations will not recognize the meaning in the
>  tags, they will be able to process them as if they were
>  unrecognized-but-well-formed registered tags.
> In addition, although this specification acknowledges the possibility of
> alternate or advanced matching and negotiation strategies, it maintains the
> existing matching algorithm (by removing subtags from the right side of a
> language tag until a match is obtained), simply providing more detail on
>  usage.
> Summary
> The authors of this specification have worked for the past year with a wide
> range of experts in the language tagging community to build consensus on a
> design for language tags that meets the needs and requirements of the user
> community. Language tags form a basic building block for natural language
> support in computer systems and content. The revision proposed in this
> specification addresses the needs of this community of users with a minimal
> impact on existing content and implementations, while providing a stable
>  basis for future development, expansion, and improvement.
> _______________________________________________
> IETF-Announce mailing list
> IETF-Announce@ietf.org
> https://www1.ietf.org/mailman/listinfo/ietf-announce
> -------------------------------------------------------

Michael Brauer                                Phone:  +49 40 23646 500
Technical Architect Software Engineering      Fax:    +49 40 23646 550
StarOffice Development
Sun Microsystems GmbH
Sachsenfeld 4
D-20097 Hamburg, Germany                e-mail: michael.brauer@sun.com

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]