Subject: Re: [cti] Questions about future changes to STIX
Between the two, I think that forward compatibility is more important. It’s what gives people incentive to start producing content in 2.1 even before tools are updated to support it. Ideally, 2.0 parsers would treat new 2.1 fields as “custom properties”; obviously there is a chance for semantic misunderstanding (which is something we should keep in mind).
I think it’s really hard for patterning (in its current form) to be forwards-compatible. Any “new” syntax will break existing parsers for the patterning grammar (there’s no concept of custom extensions to patterns). It’s much easier to keep the patterning spec backwards-compatible (so valid 2.0 patterns are also valid 2.1 patterns); I don’t think any of the proposed changes to patterning in 2.1 would be compatible with 2.0, except maybe the specific requirements for match capture groups in regular expressions (which is more about interpretation than syntax).
A recent discussion has come up about how to deal with future versions of STIX and what that means for backwards and forward compatibility.
In addition to this, we as a community need to decide how comfortable we are with changes that may break future versions. Just for clarification, OASIS has no rules about when a TC can do breaking changes as they view every release as standalone. So we as a TC need to decide, what does that means for us and when is it okay to make a breaking change and when is it not.
Some definitions to level set everyone.
STIX 2.1 data would be considered backward compatible with STIX 2.0 if every object that is valid under the 2.0 format is still valid in STIX 2.1. Meaning, a well-written STIX 2.1 parser can successfully parse STIX 2.0 content.
On the surface this seems like a good idea. However, this means that we can not add new required fields to objects in a later releases or change a previously defined optional field to now be required. Doing so would break the 2.1 parser if the content was not included in the original STIX 2.0 data (because it used to be optional). This could potentially also impact conformance clauses for future versions, so we would need to be very careful. This is potentially a big issue for the COA and Malware stubs that we put in to STIX 2.0. If we tried to maintain backwards compatibility, that means we could not add any required fields.
If we want to maintain backwards compatibility, we will need to remove the stub objects from STIX 2.0 and re-issue the CSD and public review.
What does this mean for patterning? Can patterning even be backwards compatible?
STIX 2.0 data would be forward compatible with STIX 2.1 if a product that can consume STIX 2.0 can "gracefully" process input designed for STIX 2.1+. Meaning, a well-written STIX 2.0 parser can successfully parse STIX 2.1+ content.
Content marked as required in STIX 2.0 can not be made optional in a later release.
What does this mean for patterning? Can patterning even be forwards compatible?
We will discuss this at the F2F, but I wanted to start the discussion on the email list so that eventually the TC can make a decision (since no decisions are made at the F2F). As I have thought about this from my perspective, I think the only thing we can really hope for is "forward compatibility". I personally do not think "backwards compatibility" is possible given the restriction on not adding additional required fields and what that will mean for Patterning.
But I am curious to hear and know everyone's thoughts on this. Hopefully we can get some discussion here over email, so we can have a better discussion at the F2F. How comfortable are you with future additions and changes? What is your appetite for changes that may be breaking either backward or forward compatibility?