Re: EXTERNAL: [cti-users] Re: [STIX] STIX in JSON

["Jordan, Bret" <bret.jordan@bluecoat.com>]

From a JSON standpoint it is not really that difficult to make the conversion, especially on the STIX side of the house, I have done it and others have done it. JSON even supports schema validation for those interested in that (schema validation works today for JSON TAXII). CybOX will be a bit harder to do, but with the clean up and simplification that Ivan and Trey are working on, this should be possible as well.  Imagine how much easier CybOX would be to deal with in JSON... :)   

The only other problem I see is with the way data markings are currently done.  However, you could argue that this problem exists outside of JSON and that XPath is nearly impossible to work with in code.  To solve this problem I have proposed that we look at making data markings a top level object that can just be referenced by the elements that need it. This should solve the 95% use case. For the rest, those with very elaborate data markings, we will need to make sure they are taken care of. 

I believe that adopting JSON will lower the cost of entry for vendors, product managers, and developers to the point where we will get across the chasm and go mainstream with STIX. From what I heard at RSA and Blackhat as I walked the floor and talked with vendors (CEOs, CVOs, CTOs) is that, "if we only had JSON support we would be more willing to adopt STIX".  

I envision a day when there are hundreds or thousands of apps on the various APP stores that support cyber threat intelligence and at least 15-20 new startups all dealing with CTI and my "SoC of the future".  I think we can make it happen.  We just need to lower the cost of entry and give the product managers and developers what they want, which is clear when they tell me "can I have anything but XML".  

So I push for JSON.  

Thanks,

Bret

Bret Jordan CISSP

Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

On Aug 20, 2015, at 14:34, Otto Fowler <ofowler@INDUSTRIALDEFENDER.COM> wrote:

Bret,

Can you speak to what you see as the current obstacles for supporting JSON as a complete binding?  Is it the security marking etc? 

From: <cti-users@lists.oasis-open.org> on behalf of "Jordan, Bret"
Date: Thursday, August 20, 2015 at 3:30 PM
To: "croblee@trustar.co", "cti-users@lists.oasis-open.org"
Subject: EXTERNAL: [cti-users] Re: [STIX] STIX in JSON

Great question Chris, and very timely.

As many of you know, I have been beating the drum for JSON support for a long time, nearly 18 months now. I seem to get several private emails a week from the community asking about it. Even Facebook is using JSON for their ThreatExchange platform.  

Where we stand today, to the best of my knowledge is:

1) STIX 1.2.1 will be ratified with OASIS using the current XML binding as the only reference implementation.  However, the STIX team as always maintained that you can do your own reference implementation.  And to that end several companies and groups have done just that.

2) There is support in the current Python APIs for a to_dict() to make it JSON.  I believe Soltra is using this on the back side as they store their STIX data in JSON format.  However, this to_dict() it is not a standardized form of JSON.  It was just put together by the developer of the API and some of the constructs are more XMLized JSON.

3) Several vendors and startups are doing their own STIX based JSON implementations for all of the reasons I have brought up in the past.  Intelworks is one of the few that have made public announcements about it.  The rest are still not public and thus I can not speak about them.  

4) JSON based TAXII is done and the specification can be found here [1].  I worked on this specification with lots of people and it has gone through several revisions in the TAXII community.  I even have fully working APIs for JSON based TAXII.

5) I am hoping that for STIX 2.0, we can have an official JSON binding for STIX that everyone can use.  But in the mean time, I would contact Intelworks and do what they are doing.  My project, FreeSTIX [2] is going to follow what Intelworks has done.

Everyone in this community that would like to see a JSON based STIX implementation should speak up.

[1] https://github.com/freetaxii/json-schema

[2] https://github.com/freestix 

Thanks,

Bret

Bret Jordan CISSP

Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

On Aug 20, 2015, at 10:53, Christopher Roblee <croblee@trustar.co> wrote:

Signed PGP part

I would like to resurrect this conversation on the new list.

Standard JSON bindings for STIX would greatly simplify interoperability
for us, and facilitate broader incorporation of the schema into our data
models. It sounds like this is the case for other vendors as well.

@Bret: could you please update the group as to where we are in terms of
getting JSON support in the standard libraries?

Thanks,
Chris


On 6/16/15 4:11 PM, Jordan, Bret wrote:
> I fully and 100% support your proposal as outlined below.
>
> Thanks,
>
> Bret
>
>
>
> *Bret Jordan CISSP*
> Director of Security Architecture and Standards | Office of the CTO
> Blue Coat Systems
> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
> "Without cryptography vihv vivc ce xhrnrw, however, the only thing
that can not be unscrambled is an egg."
>
>> On Jun 16, 2015, at 16:58, Aharon Chernin <achernin@soltra.com
<mailto:achernin@soltra.com>> wrote:
>>
>> I do not see this discussion as XML vs JSON. In fact, we are one of
those companies who use primarily JSON then convert to XML right before
it goes out our pipes. This is a discussion around STIX interoperability.
>>
>>
>> Consumers chose the winners. This choice is not necessarily related
to ease of vendor implementation. Some good examples of this are VHS vs
Betamax or the success of the SCAP standards outside of the US Federal
government. If we give users a poor STIX experience, by making STIX
incompatible with STIX, then users are going to spread negativity about
STIX within the user community. They will instead chose products that
"just work", instead of products who are doing the right thing by
adopting STIX. Don't get me wrong, I am all for reducing vendor
complexity, but user experience comes first.
>>
>> Let me give you a poor user experience with STIX:
>> 1) ​OASIS ratifies new JSON binding of STIX for STIX 1.x
>> 2) Consumers now begin to discover that tools that have "adopted STIX
1.x" cannot communicate with each other
>> 3) OASIS releases STIX 2.0, which is not backwards compatible with
STIX 1.x. (I support this btw)
>> 4) Users now have STIX 1.x tools that cannot talk to each other and
STIX 2.x tools that cannot talk to STIX 1.x enabled products
>>
>> Aren't we supposed to be increasing interoperability?
>>
>> Discussion suggestions:
>> *) We discuss the future of STIX 1.x in OASIS. We can discuss holding
off new features in STIX 1.x - and only bug fix. We leave the only
agreed upon binding as XML to allow all STIX 1.x client/servers to
continue communications. At this point we begin to immediately begin
work on STIX 2.x.
>> *) We discuss using JSON as the default binding type for STIX 2.0
(which I would support btw). We already assume that STIX 1.x and 2.x
would be incompatible. This is our perfect chance to change exchange
formats.
>>
>> The above proposal only breaks interoperability between major STIX
versions. Major version incompatibility can be easily explained to users
and in some cases is even expected by users.
>>
>>
>> Aharon Chernin
>> CTO
>> *SOLTRA* | An FS-ISAC & DTCC Company
>> 18301 Bermuda green Dr
>> Tampa, fl 33647
>> 813.470.2173 | achernin@soltra.com
<mailto:achernin@soltra.com><mailto:eguerrino@soltra.com>
>> www.soltra.com <http://www.soltra.com/>
>> -------------------------
>> *From:* Jordan, Bret <bret.jordan@bluecoat.com
<mailto:bret.jordan@bluecoat.com>>
>> *Sent:* Tuesday, June 16, 2015 4:25 PM
>> *To:* Aharon Chernin
>> *Cc:* STIX-DISCUSSION-LIST@LISTS.MITRE.ORG
<mailto:STIX-DISCUSSION-LIST@LISTS.MITRE.ORG>
>> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON
>>
>> The steps as I see them:
>>
>> 1: Finalize the JSON implementations for the various UML spec(s)
>> 2: Implement the JSON version in the various APIs, including the
MITRE Python/JAVA libraries (we also need libraries for a lot of other
languages to really go mainstream, not everyone uses or likes Python for
commercial products)
>>
>> At this point the market can decide which becomes the main method
people end up using.   My honest guess is it will be JSON, simple
because it will be easier and faster for web developers, app developers,
and open-source developers to use.  Most companies I have spoken with
today are already doing some sort of JSON based STIX on the back end and
then trying to do some conversion to XML to send it over the wire.  So
why not just send it over the wire in JSON format?  Seems like a way to
get people up and moving more quickly.
>>
>> When I look at it, if we sum up the total number of lines of code
that have been written so far for STIX, it will amount to less than 5%
of the sum total amount of code that has yet to be written.  One of the
major goals of the standard should be easy of implementation and use.
If the standard is too hard to implement, then people will not do it or
they will only do selective parts of it.
>>
>> The success of STIX and TAXII, in the end, will be based on the
number of platforms and projects that implement it.  If we get across
the chasm and go mainstream, then in my view we can expect to see
hundreds of apps on the various app stores and IMO, all of them will be
using JSON based STIX.
>>
>> Thanks,
>>
>> Bret
>>
>>
>>
>> *Bret Jordan CISSP*
>> Director of Security Architecture and Standards | Office of the CTO
>> Blue Coat Systems
>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing
that can not be unscrambled is an egg."
>>
>>> On Jun 16, 2015, at 13:55, Aharon Chernin <achernin@soltra.com
<mailto:achernin@soltra.com>> wrote:
>>>
>>> As it stands today, 99% of all STIX and TAXII traffic is XML based.
I am afraid we risk fragmenting the STIX community. My hope is that STIX
users (users: not vendors) don't have to experience a world where STIX
enabled products don't talk to each other due to one vendor picking JSON
over XML. Users won't blame the data exchange format, they will blame
STIX. We need to avoid that.
>>>
>>> One of the ways we can mitigate this is by including support for
JSON within the libraries used by most of the vendors who are already
creating STIX content. Mitre, are there plans to integrate the emerging
JSON format of STIX into PythonSTIX?
>>>
>>>
>>> Aharon Chernin
>>> CTO
>>> *SOLTRA* | An FS-ISAC & DTCC Company
>>> 18301 Bermuda green Dr
>>> Tampa, fl 33647
>>> 813.470.2173 | achernin@soltra.com
<mailto:achernin@soltra.com><mailto:eguerrino@soltra.com>
>>> www.soltra.com <http://www.soltra.com/>
>>> -------------------------
>>> *From:* Jordan, Bret <bret.jordan@BLUECOAT.COM
<mailto:bret.jordan@BLUECOAT.COM>>
>>> *Sent:* Tuesday, June 16, 2015 1:08 PM
>>> *To:* STIX-DISCUSSION-LIST@LISTS.MITRE.ORG
<mailto:STIX-DISCUSSION-LIST@LISTS.MITRE.ORG>
>>> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON
>>>
>>> This is such great news.  I will be working very closely with
Interworks and others interested in JSON to get my implementation in
perfect alignment.
>>>
>>> I would challenge all of you interested in JSON, or that have
already started using JSON based STIX / TAXII in your implementations to
join us.  We need this community to push for a JSON based working group
in OASIS and we need all of your best ideas to help make this successful
as quickly as possible.
>>>
>>>
>>> Thanks,
>>>
>>> Bret
>>>
>>>
>>>
>>> *Bret Jordan CISSP*
>>> Director of Security Architecture and Standards | Office of the CTO
>>> Blue Coat Systems
>>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303
>>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing
that can not be unscrambled is an egg."
>>>
>>>> On Jun 16, 2015, at 08:09, Joep Gommers <joep@intelworks.com
<mailto:joep@intelworks.com>> wrote:
>>>>
>>>> Dear all,
>>>>
>>>> With the excellent work going on from @Bret Jordan on STIX in JSON,
we thought it helpful to share Intelworks approach to STIX in JSON and
ensure the community learned from our mistakes and investments. Props to
list- and team member @Wouter Bolsterlee for his work on this!
>>>>
>>>> *In short, lessons learned*
>>>>
>>>>   * Compound structures are objects
>>>>   * Attributes and child elements are key/value pairs
>>>>   * Relations are nested objects (or arrays of objects)
>>>>   * Flat is better than nested
>>>>   * And some ID and corner cases – see below
>>>>
>>>> Full details further down in this email. Your feedback is much
appreciated.
>>>>
>>>> We do have work-in-progress libraries available for (store-less)
bi-directional transformation of XML, JSON and YAML notations – which
might help those implementing STIX in JSON down the road. If you’d like
to know more, please contact me off-list.
>>>>
>>>> Best regards,
>>>> Joep
>>>>
>>>> Founder & CEO
>>>> Intelworks – Intelligence Powered Defence
>>>> www.intelworks.com <http://www.intelworks.com/>
>>>>
>>>> Find me at
>>>> +31 615489825
>>>> @joepgommers
>>>>
>>>>
>>>> ====
>>>>
>>>> The STIX language uses quite a few advanced XML modelling
techniques (multiple namespaces, xsi:type substitutions in instance
documents, QName identifiers, and so on), making it quite complex to
work with/implement. The JSON format used by Intelworks tries to be much
simpler to work with. Structurally it mirrors most of the original XML
tree structure, but the resulting tree structures are not identical
since the JSON representation favours flat objects over nested structures.
>>>>
>>>>
>>>>       Compound structures are objects
>>>>
>>>> In general, each compound structure is converted into a JSON object
(dict in Python). These objects always have a|type| key to indicate the
type of the structure:
>>>>
>>>> {
>>>>   "type": "indicator",
>>>>   "...": "..."
>>>> }
>>>>
>>>> Each of the main STIX constructs (see the STIX architecture
<http://stixproject.github.io/getting-started/whitepaper/#architecture>)
is represented as a JSON object. The |type| keys used are:
>>>>
>>>> Defining schema    XML Schema type    Object |type| field
>>>> STIX (Core)    |STIXType|    |package|
>>>> STIX (Campaign)    |CampaignType|    |campaign|
>>>> STIX (Course of Action)    |CourseOfActionType|    |course-of-action|
>>>> STIX (Exploit Target)    |ExploitTargetType|    |exploit-target|
>>>> STIX (Incident)    |IncidentType|    |incident|
>>>> STIX (Indicator)    |IndicatorType|    |indicator|
>>>> STIX (TTP)    |TTPType|    |ttp|
>>>> STIX (Threat Actor)    |ThreatActorType|    |threat-actor|
>>>> CybOX    |ObservableType|    |observable|
>>>>
>>>> Secondary constructs use these additional types (this list is NON
EXHAUSTIVE! And just a representation of potential)
>>>>
>>>> Defining schema    XML Schema type    Object |type| field
>>>> STIX (Common)    |IdentityType|    |identity|
>>>> STIX (Common)    |InformationSourceType|    |information-source|
>>>> STIX (Common)    |StatementType|    |statement|
>>>> STIX (Course of Action)    |ObjectiveType|    |objective|
>>>> STIX (Indicator)    |ValidTimeType|    |valid-time|
>>>> STIX (Markings)    |MarkingSpecificationType|
|marking-specification|
>>>> STIX (Markings)    |MarkingStructureType| (and extensions)
|marking-structure|
>>>> STIX (TTP)    |InfrastructureType|    |infrastructure|
>>>> STIX (TTP)    |MalwareInstanceType|    |malware-instance|
>>>> STIX (TTP)    |ResourceType|    |resource|
>>>> STIX (TTP)    |ToolInformationType|    |tool-information|
>>>> STIX (TTP)    |VictimTargetingType|    |victim-targeting|
>>>> CybOX    |MeasureSourceType|    |measure-source|
>>>> CybOX    |ObjectType|    |cybox-object|
>>>> CybOX    |ToolInformationType|    |tool-information|
>>>>
>>>>
>>>>       Attributes and child elements are key/value pairs
>>>>
>>>> Both the attributes and child elements defined for a compound
structure usually map to additional key/value pairs of the JSON objects:
>>>>
>>>> {
>>>>   "type": "indicator",
>>>>   "negate": false,
>>>>   "title": "This is the title."
>>>> }
>>>>
>>>>
>>>>       Relations are nested objects (or arrays of objects)
>>>>
>>>> For one-to-one relations, the value is a nested object, and the key
is a singular noun (|observable| in the example):
>>>>
>>>> {
>>>>   "type": "indicator",
>>>>   "observable": {
>>>>     "type": "observable",
>>>>     "...": "..."
>>>>   },
>>>>   "...": "..."
>>>> }
>>>>
>>>> For one-to-many relations, the value is a JSON array containing the
child objects, and the key is a plural noun (|indicators| in the example):
>>>>
>>>> {
>>>>   "type": "package",
>>>>   "indicators": [
>>>>     {
>>>>       "type": "indicator",
>>>>       "...": "..."
>>>>     },
>>>>     {
>>>>       "type": "indicator",
>>>>       "...": "..."
>>>>     }
>>>>   ],
>>>>   "...": "..."
>>>> }
>>>>
>>>> Additionally, the many |RelatedXYZ| constructs (and the surrounding
container objects) in STIX are also flattened: the target of the
relation is the child object (or a list of those), and any additional
relationship information is embedded into the child object(s):
>>>>
>>>> {
>>>>   "type": "indicator",
>>>>   "indicated_ttps": [
>>>>     {
>>>>       "type": "ttp",
>>>>       "relationship": "...",
>>>>       "relationship_information_source": "...",
>>>>       "...": "..."
>>>>     },
>>>>     {
>>>>       "type": "ttp",
>>>>       "relationship": "...",
>>>>       "relationship_information_source": "...",
>>>>       "...": "..."
>>>>     }
>>>>   ],
>>>>   "...": "..."
>>>> }
>>>>
>>>> See also the notes about nesting below.
>>>>
>>>>
>>>>       Flat is better than nested
>>>>
>>>> The STIX XML representation is deeply nested, partly due to the way
XML is typically used. The JSON representation tries to be a bit more
pragmatic and adheres to the "flat is better than nested" adage.
>>>>
>>>> In practice, this means that nested container structures are
flattened as much as possible. Unnecessary container structures are
simply removed. For example, the |<stix:Indicators>| container structure
used in the XML representation does not exist as such in the JSON
representation, since using an array is sufficient.
>>>>
>>>> To further reduce the number of nested objects, various XML
constructs using container elements with (optional) attributes are
flattened into the parent object by using multiple related keys. This is
best explained using an example.
>>>>
>>>> For example, the |StructuredTextType| used in both STIX and CybOX
is basically a string that can optionally carry a|structuring_format|
attribute. A naive conversion would require a nested object to represent
this:
>>>>
>>>> {
>>>>   "type": "...",
>>>>   "description": {
>>>>     "structuring_format": "html",
>>>>     "value": "Description goes here."
>>>>   },
>>>>   "...": "..."
>>>> }
>>>>
>>>> Since the |structuring_format| is optional, this approach would
often result in a small nested object with only a single key/value pair
(the |value|). To avoid this, /objectivistix/ takes an alternative
approach using two related keys in the containing object:
>>>>
>>>> {
>>>>   "type": "...",
>>>>   "description": "Description goes here.",
>>>>   "description_structuring_format": "html",
>>>>   "...": "..."
>>>> }
>>>>
>>>> In case the |structuring_format| is not specified, the
|description_structuring_format| key/value pair would simply not be present:
>>>>
>>>> {
>>>>   "type": "...",
>>>>   "description": "Description goes here.",
>>>>   "...": "..."
>>>> }
>>>>
>>>>
>>>>       ID handling
>>>>
>>>> All |id| and |idref| attributes in STIX XML are not simply string
values, but qualified names (QName in XML), meaning that they contain a
namespace prefix which resolves to a namespace URI. To avoid any
explicit mappings for these prefixes and their associated namespace URI,
the JSON representation always expresses |id| and |idref| values in
their canonical form using the so-called Clark notation
<http://www.jclark.com/xml/xmlns.htm>, which looks like this:
|{http://example.com/ns/uri}local-name|.
>>>>
>>>> The top level object may optionally contain an |id_namespaces|
mapping that maps prefixes to namespace URIs. This mapping will be used
to determine the prefixes used for |id| and |idref| attribute values
when converting the object to XML, as illustrated by the example below:
>>>>
>>>> {
>>>>   "type": "package",
>>>>   "id":
"{http://example.org/}Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c",
>>>>   "id_namespaces": {
>>>>     "example": "http://example.org/"
>>>>   }
>>>> }
>>>> <stix:STIX_Package
>>>>   xmlns:stix="http://stix.mitre.org/stix-1"
>>>>   xmlns:example="http://example.com/"
>>>>   id="example:Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c">
>>>>   …
>>>> </stix:STIX_Package>
>>>>
>>>> In case no |id_namespaces| mapping is present, a unique namespace
prefix will be used instead. The |id_namespaces| can safely be left out
with no semantical loss, since the prefix is arbitrary and only used for
serialized XML data, and not for the in-memory model.
>>>>
>>>>
>>>>       Special conversion notes
>>>>
>>>>  *
>>>>
>>>>     STIX package header
>>>>
>>>>     The package header is not treated as a first-class structure.
Since the |STIX_Header| construct only applies to|STIX_Package|, it is
merged completely into the main |package| object (this avoids having an
additional nested object for the header):
>>>>
>>>>     {
>>>>       "type": "package",
>>>>       "description": "Description goes here.",
>>>>       "...": "..."
>>>>     }
>>>>  *
>>>>
>>>>     Structured text
>>>>
>>>>     The |StructuredTextType| construct is not transformed into a
child object. Instead, the keys |foo| and
(optionally)|foo_structuring_format| are added to the containing object.
>>>>
>>>>  *
>>>>
>>>>     Observable composition
>>>>
>>>>     An ‘observable composition’ structure does not result in a
nested object for the composition itself. Instead, the|composition| key
contains the child objects, and the |composition_operator| specifies the
operator:
>>>>
>>>>     {
>>>>       "type": "indicator",
>>>>       "observable": {
>>>>         "composition_operator": "or",
>>>>         "composition": [
>>>>           {
>>>>             "type": "observable",
>>>>             "...": "..."
>>>>           },
>>>>           {
>>>>             "type": "observable",
>>>>             "...": "..."
>>>>           }
>>>>         ]
>>>>       },
>>>>       "...": "..."
>>>>     }
>

--
Chris Roblee
Director of Engineering
TruSTAR Technology
Mobile: +1 781 248 2828
OpenPGP key ID: 2C9D0D20

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail