RE: Proposal for simple CybOX objects

["Kemp, David P" <dpkemp@nsa.gov>]

Thanks - I withdraw the suggestion, and will restructure our STIX COA examples to use containers instead of simple objects.

Dave


-----Original Message-----
From: cti-cybox@lists.oasis-open.org [mailto:cti-cybox@lists.oasis-open.org] On Behalf Of Kirillov, Ivan A.
Sent: Wednesday, August 17, 2016 4:05 PM
To: Kemp, David P <dpkemp@nsa.gov>; 'Mates, Jeffrey CIV DC3\DCCI' <Jeffrey.Mates@dc3.mil>; cti-cybox@lists.oasis-open.org
Subject: [cti-cybox] Re: Proposal for simple CybOX objects

This discussion (embedding vs. referencing) has come up in the past, and there have been valid arguments made on both sides. On the side of embedding, it’s true that this approach can lead smaller file sizes in some cases and “cleaner” output that may be easier to parse; on the other hand, it can lead to duplicate content, and also unbounded nesting (which was problematic in previous versions of CybOX). On the side of referencing, this approach means that there won’t be any duplicated content and is truer to a graph-based representation, though can lead to a “fragmented” serialization, especially when Objects are used only once. 

Since our mantra for CybOX 3.0 has been to support one, and only one, way of doing things we’ve had to pick an approach. Supporting both, as in CybOX 2.x, would lead us back to the same issues we had with producers creating inconsistent content and subsequently placing a large burden on consumers to parse and make sense of the data. Due to the above and also because STIX has settled on a graph-based serialization, we’ve decided to adopt a similar graph-based approach. 

I think at this point, given the fact that we’re nearing release of the first draft, it’s too late to change our approach in this regard. That said, deciding how individual Objects (e.g., Email Message) are modeled is definitely still on the table, so perhaps we can come up with a model that’s less fragmented.

Regards,
Ivan

On 8/17/16, 1:08 PM, "Kemp, David P" <dpkemp@nsa.gov> wrote:

>I'm not sure I understand the distinction.   In order to read any JSON document, every byte in the document must be "fully scanned" down to the closing brace, and the time that it takes to do the scan depends on the size of the document.  An application that counts IP addresses can build a flat symbol table of IP addresses (or of all 10 CybOX object types) in linear time either as the file is being read or after it is in memory as, e.g., a Javascript or Python datastructure.
>
>I understand CybOX and STIX to be interface standards - serialization formats that allow data to be exchanged between applications.  They are not intended to be API (in-memory) specifications, are they?  An API for counting IP addresses would want them all collected into one data structure.  But an ICD would most naturally encode them within the object where they occur.   Example 9.3.0.2, Complex MIME Message looks like a pointless slicing and dicing that would be more difficult for a receiving application to process than either the raw MIME message or a set of CybOX objects structured as they appear in the raw message.
>
>A graph would show, for example, all of the IP addresses that occurred in all of the email messages that flowed through a sensor, discovering how many addresses a sender used.   Is one CybOX document really intended to transfer such a complete graph between applications, or is it intended to define in a standard way all of the building blocks (observables) from which such a graph could be constructed.   I thought its purpose was the latter.
>
>Dave
>
>
>-----Original Message-----
>From: Mates, Jeffrey CIV DC3\DCCI [mailto:Jeffrey.Mates@dc3.mil]
>Sent: Wednesday, August 17, 2016 1:30 PM
>To: Kemp, David P <dpkemp@nsa.gov>; 'Kirillov, Ivan A.' <ikirillov@mitre.org>; cti-cybox@lists.oasis-open.org
>Subject: RE: Proposal for simple CybOX objects
>
>Allowing embedded objects was one of the biggest problems in the previous version of CybOX and if possible I really want to avoid this. Allowing mixed typing destroys the graph model that supports both STIX and CybOX information while also making relatively simple tasks, such as counting the number of distinct IP addresses that appear in your message, difficult as a full traversal will always be required for all queries.
>
>Consider a simple query against a CybOX document such as: Give me all of the IP addresses that appear in it.
>
>Without nesting all top level objects can be checked and any IP address objects will be returned.
>
>With nesting EVERY object must be fully scanned to see if an IP appears anywhere within any of its substructures.  For example a network connection to an FQDN to an IP.
>
>As such, I am strongly in favor of keeping it flat.
>
>Jeffrey Mates, Civ DC3/DCCI
>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>Computer Scientist
>Defense Cyber Crime Institute
>jeffrey.mates@dc3.mil
>410-694-4335
>
>
>-----Original Message-----
>From: cti-cybox@lists.oasis-open.org [mailto:cti-cybox@lists.oasis-open.org] On Behalf Of Kemp, David P
>Sent: Wednesday, August 17, 2016 12:49 PM
>To: 'Kirillov, Ivan A.'; cti-cybox@lists.oasis-open.org
>Subject: [Non-DoD Source] [cti-cybox] Proposal for simple CybOX objects
>
>I have added a comment on the Network Objects file requesting that subcommittee adopt a design goal of defining all CybOX objects as simple objects.  In practice, this means that properties that are defined indirectly (as references) have a direct (non-referenced) base property.  If there are use cases that require references, a  _ref property can be defined in addition to the direct property.  But in most cases, _refs should not be needed and the object can be used without having to wrap it in a cybox-container.
>
>For example in ipv4-address-object,  type and value are defined as a direct properties:
>
>{
>    "type": "ipv4-address-object",
>    "value": "1.2.3.4"
>}
>
>but resolves_to_refs and belongs_to_refs are defined indirectly.
>
>The proposal is to define equivalent direct properties that can be used in simple objects, e.g.:
>
>{
>    "type": "ipv4-address-object",
>    "value": "1.2.3.4",
>    "resolves_to": [{
>        "type": "mac-address-object",
>        "value": "01:23:45:67:89:ab"
>    }]
>}
>
>Fortunately mac-address-object is defined as a simple object so ipv4-address-object can use it without having to wrap it.   For simplicity and consistency, all other CybOX objects (including ipv4-address-object) should also be usable without wrapping them in cybox-containers.
>
>Thoughts?
>
>Dave Kemp
>
>
>
>-----Original Message-----
>From: cti-cybox@lists.oasis-open.org [mailto:cti-cybox@lists.oasis-open.org] On Behalf Of Kirillov, Ivan A.
>Sent: Wednesday, August 17, 2016 10:49 AM
>To: cti-cybox@lists.oasis-open.org
>Subject: Re: [cti-cybox] Recent CybOX Changes
>
>I just wanted to let everyone know where we stand on these issues. Trey and I had a discussion on this topic and decided to make the following changes:
>
>
>
>·         file_path in File Object/file-system-properties-type was changed to a string. As others had pointed out, there was no benefit to the delimited approach and it had the side effect of making patterns more complex.
>
>·         We’ve removed the string-with-encoding-type and reverted all uses back to string. We will similarly go back to our old approach of capturing observed encoding using the flattened “_enc” and “_b64” fields – for an example of this please see the file_name and corresponding fields [1]. We’ve also reverted our old section in CybOX Core on this topic [2] – please review and comment. Also, let us know which other fields in the CybOX Objects we should be adding this for.
>
>
>
>[1] https://docs.google.com/document/d/1DdS-NrVTjGJ3wvCJ7dbSlhYeiaWS6G6dOXu2F3POpUs/edit#heading=h.8qkuomuewogh <https://docs.google.com/document/d/1DdS-NrVTjGJ3wvCJ7dbSlhYeiaWS6G6dOXu2F3POpUs/edit#heading=h.8qkuomuewogh>
>
>[2] https://docs.google.com/document/d/1PSGv6Uvo3YyrK354cH0cvdn7gGedbhYJkgNVzwW9E6A/edit?pref=2&pli=1#heading=h.mqdyq1mo042h <https://docs.google.com/document/d/1PSGv6Uvo3YyrK354cH0cvdn7gGedbhYJkgNVzwW9E6A/edit?pref=2&pli=1#heading=h.mqdyq1mo042h>
>
>
>
>Regards,
>
>Ivan
>
>
>
>From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
>Date: Tuesday, August 2, 2016 at 3:28 PM
>To: Allan Thomson <athomson@lookingglasscyber.com>
>Cc: John-Mark Gurney <jmg@newcontext.com>, "Mates, Jeffrey CIV DC3\\DCCI" <Jeffrey.Mates@dc3.mil>, Ivan Kirillov <ikirillov@mitre.org>, John Wunder <jwunder@mitre.org>, Greg Back <gback@mitre.org>, "cti-cybox@lists.oasis-open.org" <cti-cybox@lists.oasis-open.org>
>Subject: Re: [cti-cybox] Recent CybOX Changes
>
>
>
>I agree, I am also unsure what the purpose of this field is... are there actual threat analytics use cases where knowing the encoding of a piece of text conveys useful information? Was this requested by a specific user group?
>
>-
>Jason Keirstead
>STSM, Product Architect, Security Intelligence, IBM Security Systems www.ibm.com/security | www.securityintelligence.com
>
>Without data, all you are is just another person with an opinion - Unknown
>
>
>nactive hide details for Allan Thomson ---08/02/2016 03:47:31 PM---That’sAllan Thomson ---08/02/2016 03:47:31 PM---That’s a good clarification and it makes me wonder why this attribute needs to be conveyed at all. W
>
>From: Allan Thomson <athomson@lookingglasscyber.com>
>To: John-Mark Gurney <jmg@newcontext.com>
>Cc: "Mates, Jeffrey CIV DC3\\DCCI" <Jeffrey.Mates@dc3.mil>, "'Kirillov, Ivan A.'" <ikirillov@mitre.org>, Jason Keirstead/CanEast/IBM@IBMCA, "Wunder, John A." <jwunder@mitre.org>, "Back, Greg" <gback@mitre.org>, "cti-cybox@lists.oasis-open.org" <cti-cybox@lists.oasis-open.org>
>Date: 08/02/2016 03:47 PM
>Subject: Re: [cti-cybox] Recent CybOX Changes Sent by: <cti-cybox@lists.oasis-open.org>
>
>________________________________
>
>
>
>
>That’s a good clarification and it makes me wonder why this attribute needs to be conveyed at all. What is a recipient expected to do with this information?
>
>Certainly combining it into a single field makes no sense now.
>
>Thanks for sharing this perspective. It changes my opinion of what this encoding field is attempting to convey and how it should be done.
>
>allan
>
>On 8/2/16, 10:47 AM, "John-Mark Gurney" <jmg@newcontext.com> wrote:
>
>   Allan Thomson wrote this message on Tue, Aug 02, 2016 at 14:02 +0000:
>   > Finally, *if* encoding must be conveyed when a field that may have different encodings possible is required. Then what is the most efficient and effective way to allow two systems to a) generate that package b) read that package efficiently and effectively.
>
>   I want to clarify this statement..  Fields only ever contain Unicode
>   data.  They never contain non-Unicode, or encoded data..  The encoding
>   is a way to convey what encoding the original data had before converting
>   to CybOX.
>
>   --
>   John-Mark
>
>
>
>
>
>