[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]
Subject: Re: [STIX] STIX in JSON
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 I would like to resurrect this conversation on the new list. Standard JSON bindings for STIX would greatly simplify interoperability for us, and facilitate broader incorporation of the schema into our data models. It sounds like this is the case for other vendors as well. @Bret: could you please update the group as to where we are in terms of getting JSON support in the standard libraries? Thanks, Chris On 6/16/15 4:11 PM, Jordan, Bret wrote: > I fully and 100% support your proposal as outlined below. > > Thanks, > > Bret > > > > *Bret Jordan CISSP* > Director of Security Architecture and Standards | Office of the CTO > Blue Coat Systems > PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303 > "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." > >> On Jun 16, 2015, at 16:58, Aharon Chernin <achernin@soltra.com <mailto:achernin@soltra.com>> wrote: >> >> I do not see this discussion as XML vs JSON. In fact, we are one of those companies who use primarily JSON then convert to XML right before it goes out our pipes. This is a discussion around STIX interoperability. >> >> >> Consumers chose the winners. This choice is not necessarily related to ease of vendor implementation. Some good examples of this are VHS vs Betamax or the success of the SCAP standards outside of the US Federal government. If we give users a poor STIX experience, by making STIX incompatible with STIX, then users are going to spread negativity about STIX within the user community. They will instead chose products that "just work", instead of products who are doing the right thing by adopting STIX. Don't get me wrong, I am all for reducing vendor complexity, but user experience comes first. >> >> Let me give you a poor user experience with STIX: >> 1) OASIS ratifies new JSON binding of STIX for STIX 1.x >> 2) Consumers now begin to discover that tools that have "adopted STIX 1.x" cannot communicate with each other >> 3) OASIS releases STIX 2.0, which is not backwards compatible with STIX 1.x. (I support this btw) >> 4) Users now have STIX 1.x tools that cannot talk to each other and STIX 2.x tools that cannot talk to STIX 1.x enabled products >> >> Aren't we supposed to be increasing interoperability? >> >> Discussion suggestions: >> *) We discuss the future of STIX 1.x in OASIS. We can discuss holding off new features in STIX 1.x - and only bug fix. We leave the only agreed upon binding as XML to allow all STIX 1.x client/servers to continue communications. At this point we begin to immediately begin work on STIX 2.x. >> *) We discuss using JSON as the default binding type for STIX 2.0 (which I would support btw). We already assume that STIX 1.x and 2.x would be incompatible. This is our perfect chance to change exchange formats. >> >> The above proposal only breaks interoperability between major STIX versions. Major version incompatibility can be easily explained to users and in some cases is even expected by users. >> >> >> Aharon Chernin >> CTO >> *SOLTRA* | An FS-ISAC & DTCC Company >> 18301 Bermuda green Dr >> Tampa, fl 33647 >> 813.470.2173 | achernin@soltra.com <mailto:achernin@soltra.com><mailto:eguerrino@soltra.com> >> www.soltra.com <http://www.soltra.com/> >> ------------------------- >> *From:* Jordan, Bret <bret.jordan@bluecoat.com <mailto:bret.jordan@bluecoat.com>> >> *Sent:* Tuesday, June 16, 2015 4:25 PM >> *To:* Aharon Chernin >> *Cc:* STIX-DISCUSSION-LIST@LISTS.MITRE.ORG <mailto:STIX-DISCUSSION-LIST@LISTS.MITRE.ORG> >> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON >> >> The steps as I see them: >> >> 1: Finalize the JSON implementations for the various UML spec(s) >> 2: Implement the JSON version in the various APIs, including the MITRE Python/JAVA libraries (we also need libraries for a lot of other languages to really go mainstream, not everyone uses or likes Python for commercial products) >> >> At this point the market can decide which becomes the main method people end up using. My honest guess is it will be JSON, simple because it will be easier and faster for web developers, app developers, and open-source developers to use. Most companies I have spoken with today are already doing some sort of JSON based STIX on the back end and then trying to do some conversion to XML to send it over the wire. So why not just send it over the wire in JSON format? Seems like a way to get people up and moving more quickly. >> >> When I look at it, if we sum up the total number of lines of code that have been written so far for STIX, it will amount to less than 5% of the sum total amount of code that has yet to be written. One of the major goals of the standard should be easy of implementation and use. If the standard is too hard to implement, then people will not do it or they will only do selective parts of it. >> >> The success of STIX and TAXII, in the end, will be based on the number of platforms and projects that implement it. If we get across the chasm and go mainstream, then in my view we can expect to see hundreds of apps on the various app stores and IMO, all of them will be using JSON based STIX. >> >> Thanks, >> >> Bret >> >> >> >> *Bret Jordan CISSP* >> Director of Security Architecture and Standards | Office of the CTO >> Blue Coat Systems >> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303 >> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." >> >>> On Jun 16, 2015, at 13:55, Aharon Chernin <achernin@soltra.com <mailto:achernin@soltra.com>> wrote: >>> >>> As it stands today, 99% of all STIX and TAXII traffic is XML based. I am afraid we risk fragmenting the STIX community. My hope is that STIX users (users: not vendors) don't have to experience a world where STIX enabled products don't talk to each other due to one vendor picking JSON over XML. Users won't blame the data exchange format, they will blame STIX. We need to avoid that. >>> >>> One of the ways we can mitigate this is by including support for JSON within the libraries used by most of the vendors who are already creating STIX content. Mitre, are there plans to integrate the emerging JSON format of STIX into PythonSTIX? >>> >>> >>> Aharon Chernin >>> CTO >>> *SOLTRA* | An FS-ISAC & DTCC Company >>> 18301 Bermuda green Dr >>> Tampa, fl 33647 >>> 813.470.2173 | achernin@soltra.com <mailto:achernin@soltra.com><mailto:eguerrino@soltra.com> >>> www.soltra.com <http://www.soltra.com/> >>> ------------------------- >>> *From:* Jordan, Bret <bret.jordan@BLUECOAT.COM <mailto:bret.jordan@BLUECOAT.COM>> >>> *Sent:* Tuesday, June 16, 2015 1:08 PM >>> *To:* STIX-DISCUSSION-LIST@LISTS.MITRE.ORG <mailto:STIX-DISCUSSION-LIST@LISTS.MITRE.ORG> >>> *Subject:* Re: [STIX] Intelworks implementation of STIX in JSON >>> >>> This is such great news. I will be working very closely with Interworks and others interested in JSON to get my implementation in perfect alignment. >>> >>> I would challenge all of you interested in JSON, or that have already started using JSON based STIX / TAXII in your implementations to join us. We need this community to push for a JSON based working group in OASIS and we need all of your best ideas to help make this successful as quickly as possible. >>> >>> >>> Thanks, >>> >>> Bret >>> >>> >>> >>> *Bret Jordan CISSP* >>> Director of Security Architecture and Standards | Office of the CTO >>> Blue Coat Systems >>> PGP Fingerprint: 62A6 5999 0F7D 0D61 4C66 D59C 2DB5 111D 63BC A303 >>> "Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." >>> >>>> On Jun 16, 2015, at 08:09, Joep Gommers <joep@intelworks.com <mailto:joep@intelworks.com>> wrote: >>>> >>>> Dear all, >>>> >>>> With the excellent work going on from @Bret Jordan on STIX in JSON, we thought it helpful to share Intelworks approach to STIX in JSON and ensure the community learned from our mistakes and investments. Props to list- and team member @Wouter Bolsterlee for his work on this! >>>> >>>> *In short, lessons learned* >>>> >>>> * Compound structures are objects >>>> * Attributes and child elements are key/value pairs >>>> * Relations are nested objects (or arrays of objects) >>>> * Flat is better than nested >>>> * And some ID and corner cases – see below >>>> >>>> Full details further down in this email. Your feedback is much appreciated. >>>> >>>> We do have work-in-progress libraries available for (store-less) bi-directional transformation of XML, JSON and YAML notations – which might help those implementing STIX in JSON down the road. If you’d like to know more, please contact me off-list. >>>> >>>> Best regards, >>>> Joep >>>> >>>> Founder & CEO >>>> Intelworks – Intelligence Powered Defence >>>> www.intelworks.com <http://www.intelworks.com/> >>>> >>>> Find me at >>>> +31 615489825 >>>> @joepgommers >>>> >>>> >>>> ==== >>>> >>>> The STIX language uses quite a few advanced XML modelling techniques (multiple namespaces, xsi:type substitutions in instance documents, QName identifiers, and so on), making it quite complex to work with/implement. The JSON format used by Intelworks tries to be much simpler to work with. Structurally it mirrors most of the original XML tree structure, but the resulting tree structures are not identical since the JSON representation favours flat objects over nested structures. >>>> >>>> >>>> Compound structures are objects >>>> >>>> In general, each compound structure is converted into a JSON object (dict in Python). These objects always have a|type| key to indicate the type of the structure: >>>> >>>> { >>>> "type": "indicator", >>>> "...": "..." >>>> } >>>> >>>> Each of the main STIX constructs (see the STIX architecture <http://stixproject.github.io/getting-started/whitepaper/#architecture>) is represented as a JSON object. The |type| keys used are: >>>> >>>> Defining schema XML Schema type Object |type| field >>>> STIX (Core) |STIXType| |package| >>>> STIX (Campaign) |CampaignType| |campaign| >>>> STIX (Course of Action) |CourseOfActionType| |course-of-action| >>>> STIX (Exploit Target) |ExploitTargetType| |exploit-target| >>>> STIX (Incident) |IncidentType| |incident| >>>> STIX (Indicator) |IndicatorType| |indicator| >>>> STIX (TTP) |TTPType| |ttp| >>>> STIX (Threat Actor) |ThreatActorType| |threat-actor| >>>> CybOX |ObservableType| |observable| >>>> >>>> Secondary constructs use these additional types (this list is NON EXHAUSTIVE! And just a representation of potential) >>>> >>>> Defining schema XML Schema type Object |type| field >>>> STIX (Common) |IdentityType| |identity| >>>> STIX (Common) |InformationSourceType| |information-source| >>>> STIX (Common) |StatementType| |statement| >>>> STIX (Course of Action) |ObjectiveType| |objective| >>>> STIX (Indicator) |ValidTimeType| |valid-time| >>>> STIX (Markings) |MarkingSpecificationType| |marking-specification| >>>> STIX (Markings) |MarkingStructureType| (and extensions) |marking-structure| >>>> STIX (TTP) |InfrastructureType| |infrastructure| >>>> STIX (TTP) |MalwareInstanceType| |malware-instance| >>>> STIX (TTP) |ResourceType| |resource| >>>> STIX (TTP) |ToolInformationType| |tool-information| >>>> STIX (TTP) |VictimTargetingType| |victim-targeting| >>>> CybOX |MeasureSourceType| |measure-source| >>>> CybOX |ObjectType| |cybox-object| >>>> CybOX |ToolInformationType| |tool-information| >>>> >>>> >>>> Attributes and child elements are key/value pairs >>>> >>>> Both the attributes and child elements defined for a compound structure usually map to additional key/value pairs of the JSON objects: >>>> >>>> { >>>> "type": "indicator", >>>> "negate": false, >>>> "title": "This is the title." >>>> } >>>> >>>> >>>> Relations are nested objects (or arrays of objects) >>>> >>>> For one-to-one relations, the value is a nested object, and the key is a singular noun (|observable| in the example): >>>> >>>> { >>>> "type": "indicator", >>>> "observable": { >>>> "type": "observable", >>>> "...": "..." >>>> }, >>>> "...": "..." >>>> } >>>> >>>> For one-to-many relations, the value is a JSON array containing the child objects, and the key is a plural noun (|indicators| in the example): >>>> >>>> { >>>> "type": "package", >>>> "indicators": [ >>>> { >>>> "type": "indicator", >>>> "...": "..." >>>> }, >>>> { >>>> "type": "indicator", >>>> "...": "..." >>>> } >>>> ], >>>> "...": "..." >>>> } >>>> >>>> Additionally, the many |RelatedXYZ| constructs (and the surrounding container objects) in STIX are also flattened: the target of the relation is the child object (or a list of those), and any additional relationship information is embedded into the child object(s): >>>> >>>> { >>>> "type": "indicator", >>>> "indicated_ttps": [ >>>> { >>>> "type": "ttp", >>>> "relationship": "...", >>>> "relationship_information_source": "...", >>>> "...": "..." >>>> }, >>>> { >>>> "type": "ttp", >>>> "relationship": "...", >>>> "relationship_information_source": "...", >>>> "...": "..." >>>> } >>>> ], >>>> "...": "..." >>>> } >>>> >>>> See also the notes about nesting below. >>>> >>>> >>>> Flat is better than nested >>>> >>>> The STIX XML representation is deeply nested, partly due to the way XML is typically used. The JSON representation tries to be a bit more pragmatic and adheres to the "flat is better than nested" adage. >>>> >>>> In practice, this means that nested container structures are flattened as much as possible. Unnecessary container structures are simply removed. For example, the |<stix:Indicators>| container structure used in the XML representation does not exist as such in the JSON representation, since using an array is sufficient. >>>> >>>> To further reduce the number of nested objects, various XML constructs using container elements with (optional) attributes are flattened into the parent object by using multiple related keys. This is best explained using an example. >>>> >>>> For example, the |StructuredTextType| used in both STIX and CybOX is basically a string that can optionally carry a|structuring_format| attribute. A naive conversion would require a nested object to represent this: >>>> >>>> { >>>> "type": "...", >>>> "description": { >>>> "structuring_format": "html", >>>> "value": "Description goes here." >>>> }, >>>> "...": "..." >>>> } >>>> >>>> Since the |structuring_format| is optional, this approach would often result in a small nested object with only a single key/value pair (the |value|). To avoid this, /objectivistix/ takes an alternative approach using two related keys in the containing object: >>>> >>>> { >>>> "type": "...", >>>> "description": "Description goes here.", >>>> "description_structuring_format": "html", >>>> "...": "..." >>>> } >>>> >>>> In case the |structuring_format| is not specified, the |description_structuring_format| key/value pair would simply not be present: >>>> >>>> { >>>> "type": "...", >>>> "description": "Description goes here.", >>>> "...": "..." >>>> } >>>> >>>> >>>> ID handling >>>> >>>> All |id| and |idref| attributes in STIX XML are not simply string values, but qualified names (QName in XML), meaning that they contain a namespace prefix which resolves to a namespace URI. To avoid any explicit mappings for these prefixes and their associated namespace URI, the JSON representation always expresses |id| and |idref| values in their canonical form using the so-called Clark notation <http://www.jclark.com/xml/xmlns.htm>, which looks like this: |{http://example.com/ns/uri}local-name|. >>>> >>>> The top level object may optionally contain an |id_namespaces| mapping that maps prefixes to namespace URIs. This mapping will be used to determine the prefixes used for |id| and |idref| attribute values when converting the object to XML, as illustrated by the example below: >>>> >>>> { >>>> "type": "package", >>>> "id": "{http://example.org/}Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c", >>>> "id_namespaces": { >>>> "example": "http://example.org/" >>>> } >>>> } >>>> <stix:STIX_Package >>>> xmlns:stix="http://stix.mitre.org/stix-1" >>>> xmlns:example="http://example.com/" >>>> id="example:Package-b3ba766b-d3e6-4d92-82b2-5940f0cb763c"> >>>> … >>>> </stix:STIX_Package> >>>> >>>> In case no |id_namespaces| mapping is present, a unique namespace prefix will be used instead. The |id_namespaces| can safely be left out with no semantical loss, since the prefix is arbitrary and only used for serialized XML data, and not for the in-memory model. >>>> >>>> >>>> Special conversion notes >>>> >>>> * >>>> >>>> STIX package header >>>> >>>> The package header is not treated as a first-class structure. Since the |STIX_Header| construct only applies to|STIX_Package|, it is merged completely into the main |package| object (this avoids having an additional nested object for the header): >>>> >>>> { >>>> "type": "package", >>>> "description": "Description goes here.", >>>> "...": "..." >>>> } >>>> * >>>> >>>> Structured text >>>> >>>> The |StructuredTextType| construct is not transformed into a child object. Instead, the keys |foo| and (optionally)|foo_structuring_format| are added to the containing object. >>>> >>>> * >>>> >>>> Observable composition >>>> >>>> An ‘observable composition’ structure does not result in a nested object for the composition itself. Instead, the|composition| key contains the child objects, and the |composition_operator| specifies the operator: >>>> >>>> { >>>> "type": "indicator", >>>> "observable": { >>>> "composition_operator": "or", >>>> "composition": [ >>>> { >>>> "type": "observable", >>>> "...": "..." >>>> }, >>>> { >>>> "type": "observable", >>>> "...": "..." >>>> } >>>> ] >>>> }, >>>> "...": "..." >>>> } > - -- Chris Roblee Director of Engineering TruSTAR Technology Mobile: +1 781 248 2828 OpenPGP key ID: 2C9D0D20 -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQIcBAEBCgAGBQJV1gX4AAoJELygnscsnQ0gxvIP/A6x3sfoX7+C1ihCQ0rj6P9c CJJ75Wuz7wGe0/A+9HK3keHxUGy89jfDZbXEMo07PDadcryuqNIRHSu3i1sSLpxY jjldgxM2+rNqBO9ZHGWGO4gdFKlLy8Vz7Z1AroFTWUl9OlfvD6kFbdOZh7sBXMZb 3iIE4cFhIfC9b7OuTLSDrA4S7hnDIhlSTBJKdvK3REJv3W92KrjT6fkHIQH6TDxj ayOuq023iHntMCwqJREWgXOwX+Y5E7vAD6xzqLRY3qWsUvh+/OsR8sK4wxUrGZiV 4IK+TFszxfngit6/UYxvSTpcziG1WxiToL+p8KGBx2Wz+xCS4bWdKqfkaXqaIEKZ iqfNVCP/cBrb5VAckKHFDMirg9Iuj5mlIJ+alwbLCPU17vVmJmFgya5Y8dAIKELM qEaC0aWWtWhJRiCEUIGjE+H594dPJXj0Xk8LoED+yjxwlqVqxdQflAlpsMimUvkb iCIuvmm1yTRsRxQnriqdDa+nmAjjVXROQCxmWzZLclsl8xnAejxN3vPBb3DHMAlb pQ9J7AJ/15PTeIETPYjBH+Jq7dmtz5bcbIzI239B5T5hueOHQyBIZwONjxzvkHrg x793iQ5qlmgn9VI5dINqxKGLqs2TYGCV2S7hvHp6xB+m8dW7N+MVVQ1Mq/uCTCy2 gw3gD7idC4QmL7Fb4YVi =0TI4 -----END PGP SIGNATURE-----
[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]