Unicode, strings, and STIX

[John-Mark Gurney <jmg@newcontext.com>]

Hello,

In attempting to nail down the definition of the type string, there have been a few questions raised about the best definition.  I do not believe there is any disagreement that Unicode will be used for the string representation, it is more how to address some of the things about handling the string type.

You may have heard various talk about character vs code point vs glyph vs grapheme, and I found a good post answering the distinction between them at http://stackoverflow.com/questions/27331819/whats-the-difference-between-a-character-a-code-point-a-glyph-and-a-grapheme .  I will talk about encoding later.

So, at the most basic, a string is a sequence of Unicode code points.  Some strings may have more code points than others, though they are the equivalent, ø (1) vs o w/ combining long solidus overlay (2), though when normalized (NFC), they will be equal.  Sadly, some other code points are ligatures, which are not expanded when normalized (NFC) resulting in the fi ligature not being equal to the letters f followed by i when normalized (NFC).  NFKC will make them equal, but will destroy the meaning of other symbols, like 2 superscript becomes a normal 2.

1) Should we add length restrictions to (some?) fields?  For example, should the title field be restricted in it's length somehow?  Or should people be able to put unlimited length text in the field?  Some fields like description, I expect would possibly be unlimited sans some other overriding limit, such as total TLO size, etc.

2) If there are length limits, how should the length limit be defined?  Should it be number of graphemes displayed?  (Be careful of this, because of things like Zalgo (http://stackoverflow.com/questions/6579844/how-does-zalgo-text-work) make even a short ~25 grapheme string have ~292 code points, or 559 bytes when UTF-8 encoded.  Though no language will normally use so many combining code points, it is required to use more than one in some languages. Normalization can help reduce a string's number of code points, but does not always help.  Some languages, like Thai, will use more than one combining code point to make a single grapheme (consonant + vowel + tone mark for three code points for a single grapheme).

If graphemes are used, it would require a validator to have a detailed table to decide how many graphemes are in the string.  Using code points would not require as much work for the validator.

There is an additional issue of encoding, but this should be easy.  It should use the underlying serialization format's encoding of Unicode.  In the case of JSON, the default is UTF-8.  In the case of XML, it can be specified by the document itself, and may even be in a non-UTF encoding, but it is assumed that if the document is in a different character set, that the processor will convert to Unicode code points properly.

Additional Reading:

UNICODE TEXT SEGMENTATION http://unicode.org/reports/tr29/  -- has additional examples of grapheme and code points.

Internationalization for Turkish: Dotted and Dotless Letter "I" http://www.i18nguy.com/unicode/turkish-i18n.html -- More deals w/ complexities of locales than the above

Forms of Unicode http://www.icu-project.org/docs/papers/forms_of_unicode/ -- Good description of glyph vs characters vs ligatures and encoding info

My recommendations:

1) I do believe that limits should be defined for some fields.  Things like title should not have the description in them, and leaving it undefined will allow it to happen.

2) My personal view (as a programmer of many years) it to go the simple round and limit it by code points.  This is easiest for a programmer to do w/ existing tools.  It also gives a more clear storage space limit (see the Zalgo example above).

John-Mark

New Context