RE: [cti] Internationalization: lang field required or optional?

["Back, Greg" <gback@mitre.org>]

+1 for everything Allan said. I would prefer optional.

> -----Original Message-----
> From: cti@lists.oasis-open.org [mailto:cti@lists.oasis-open.org] On Behalf Of
> Allan Thomson
> Sent: Thursday, February 23, 2017 5:01 PM
> To: Bret Jordan <Bret_Jordan@symantec.com>; Wunder, John A.
> <jwunder@mitre.org>; cti@lists.oasis-open.org
> Subject: Re: [cti] Internationalization: lang field required or optional?
> 
> If you are expecting to use different language content then its required for
> interoperability reasons.
> 
> 
> 
> But by marking it required in the spec means that all content must have it
> even when most content is not multi-language.
> 
> 
> 
> I generally would prefer more tolerance in the spec level and let the
> products/market use good behavior to drive what fields are included or not.
> 
> 
> 
> If people care about language and multi-language support then they will use
> it. If they don’t then they wont be interoperable as that will be part of the
> test in the interop spec.
> 
> 
> 
> allan
> 
> 
> 
> From: Bret Jordan <Bret_Jordan@symantec.com>
> Date: Thursday, February 23, 2017 at 2:04 PM
> To: Allan Thomson <athomson@lookingglasscyber.com>, "Wunder, John"
> <jwunder@mitre.org>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
> Subject: Re: [cti] Internationalization: lang field required or optional?
> 
> 
> 
> My thoughts....
> 
> 
> 
> 1) In reality we are talking about a feature not a property.
> 
> 2) If it is property of this feature is optional, then the only products that will
> implement this feature, are those that care about internationalization.
> 
> 3) If it is required, then everyone will be forced to implement it.
> 
> 
> 
> Personally I see this as a data quality issue, not a STIX issue.  And I think both
> sides can suffer from it.
> 
> 
> 
> Problems with Required:
> 
> a) product or tool does not care, does not provide a UX for it, and just hard
> codes it to something, say "en"
> 
> b) product or tool does provide a UX for it, but analyst does not care and it
> just remains what ever the default is.
> 
> 
> 
> Problems with Optional:
> 
> a) product or tool does not care, does not provide a UX for it, and just leaves
> it out of the data.  So it is undef.
> 
> b) product or tool does care and provides a UX for it and the analyst does not
> care and leaves it blank.
> 
> c) Broker product or tool takes in data that has a lang tag, but they do not
> support that feature so they never implemented it.  So when the data goes
> back out the other side, the language tag is now missing.
> 
> 
> 
> I personally do not see the harm in requiring tools to support and populate
> the Lang tag.  In the spec we can define an "unknown" value, so if you are
> doing bulk loading of data and you honestly do not know the language, you
> could just flag it as "unknown".  Then at least as the consumer you would
> know that the producer did not know the language.  Versus getting an object
> where the language tag is omitted and you do not know if:
> 
> i) they did not know the language
> 
> ii) there tool did not support it
> 
> iii) they were just lazy and did not add it.
> 
> 
> 
> Once again, this is a data quality problem and if we make the lang field
> required, then it is a SUPER EASY interop test to see if they do it right.  If it is
> optional, then you are just at a guess all the time.
> 
> 
> 
> Bret
> 
> ________________________________
> 
> From: cti@lists.oasis-open.org <cti@lists.oasis-open.org> on behalf of Allan
> Thomson <athomson@lookingglasscyber.com>
> Sent: Thursday, February 23, 2017 2:29:59 PM
> To: Wunder, John A.; cti@lists.oasis-open.org
> Subject: Re: [cti] Internationalization: lang field required or optional?
> 
> 
> 
> Prefer optional.
> 
> 
> 
> From: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org> on behalf of
> "Wunder, John" <jwunder@mitre.org>
> Date: Thursday, February 23, 2017 at 12:59 PM
> To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
> Subject: [cti] Internationalization: lang field required or optional?
> 
> 
> 
> Hey everyone,
> 
> 
> 
> We’re getting very close to having a completed approach for
> internationalization, you can see the full writeup here:
> https://docs.google.com/document/d/15qD9KBQcVcY4FlG9n_VGhqacaeiLlN
> cQ7zVEjc8I3b4/edit#heading=h.61fy0hlsdirz
> <https://clicktime.symantec.com/a/1/qWBLZKnNvvyFG2i-
> ABJwnfDZ0ZhWYHOOGdoKaqw0c7E=?d=U3YT8E2S5XDUQtCKMuXppLuJy6_E
> m-ecWq7R2e8d52iTGAmdJbKuWkADrfPqe6CD0TlgkzXtCl-
> NRQEID5mE5XFllNRjrboocSIKAqU0FSeByZviKmWKK9uYxl36I4Xnzh4h4x_ALL_
> _2TA9RZEeOSRcjNP5RFhxIe6dlRx8vl1fy4DeZwnreWnCFJgdQd9xbHTeQrwbZ3
> xHhuDOcj_aRQAZYrum0lZjMSL4hQ-
> CwzHAJYFrjdCjnJdrCGBNqmK4QbHqq__XzNZwPut6RQKgT1cvS3HKT5GFWhq
> RNoaRViAl7CcqaEg4RC1YII8ma_7DlpXbApEjDV5v9iCMBCCVtZrwJB8FM-
> BD71POwUO80OE8ZMLTtRT-
> ilDNE2N8xAS52SAWx_wG6eGw5hlDmA8r5SKfNuH3pdzOA9VEJCyfXBmEsmn
> eIgH5JDutnAScTaJ7CRo6ivq3whzwPgmXbShfvvVf3_Nv42N1zXTJ0A%3D%3D&
> u=https%3A%2F%2Fdocs.google.com%2Fdocument%2Fd%2F15qD9KBQcVcY
> 4FlG9n_VGhqacaeiLlNcQ7zVEjc8I3b4%2Fedit%23heading%3Dh.61fy0hlsdirz>
> 
> 
> 
> We do have one remaining question before we can move forward though.
> As part of the proposal, every single top-level object has a “lang” field, that
> identifies the language of the text content in that object. What we need to
> decide is whether we make that field required or optional.
> 
> 
> 
> If we make the field required, every top-level object in STIX (SDOs and SROs)
> would have to have a “lang” field in it or it would be invalid STIX. If we make it
> optional, producers could either include the field or not.
> 
> 
> 
> Here are some thoughts:
> 
> 
> 
> Making it required:
> 
> 
> 
> 
> 
> -          All SDOs and SROs would have a language tag, so consumers could
> depend on it being there
> 
> -          It would encourage producers to actually fill it out, because they
> wouldn’t be creating valid STIX otherwise
> 
> -          It shows we have a commitment to internationalization
> 
> 
> 
> Making it optional:
> 
> 
> 
> -          Any SRO or SDO could have a language tag, so consumers could not
> depend on it
> 
> -          Producers would not have to create it
> 
> -          We do have a SHOULD requirement saying that it should be included
> 
> 
> 
> My opinion is that we should make it optional. If it’s required, I think people
> who don’t want to do internationalization (especially those creating one-off
> scripts or open source tools) will hardcode it to English and things will be
> mislabeled. If it’s optional, I think those who need/want to support
> internationalization and would do it right (most/all vendors, major open
> source projects) will populate it correctly regardless…because they need
> it…while those who couldn’t be bothered will be able to leave it off and we
> won’t have mis-labeled data. Also it’s almost not worth saying, but we
> already have a bunch of required fields on every SDO/SRO and I’ve already
> had one conversation with someone who said there’s a lot of bloat…would
> like to avoid adding to that.
> 
> 
> 
> Anyway, what does everyone think…required or optional?
> 
> 
> 
> John