Re: [cti-stix] STIX Indicator Proposal

[John-Mark Gurney <jmg@newcontext.com>]

Katz, Gary CTR DC3\DCCI wrote this message on Thu, Jun 08, 2017 at 15:49 +0000:
> I would like to provide another alternative to cover the issues associated
> with analysts creating indicators and the objects they are based upon.
> There are definitely some cons to this approach, but I believe it does meet
> the core issue and at the least it should hopefully start some new
> discussion on the topic.
> 
> Problem:
> Currently analysts need to capture an IP address, email address or other
> atomic indicators in two ways, first as part of observed data or
> infrastructure and then again as an indicator within a pattern.  Not all
> IPs, FQDNs, email addresses, etc. are indicators but a large majority are
> and large volumes of these types of indicators are captured on a daily
> basis.  While individual products can be built to alleviate the extra work
> required to maintain this information in two locations (the indicator object
> and in the analysis (i.e. observed data, infrastructure, etc.), STIX should
> be designed to encourage good practices and minimize the need for systems to
> hide inefficiencies in the standard.
> 
> Proposed Solution:
> This solution does leave the opportunity for indicators to be included
> within a document in two ways, but encourages atomic indicators to be
> represented in one way, while complex indicators to be represented in
> another way.  Many properties within objects can also be atomic indicators.
> This proposal suggests that any properties that are also an atomic indicator
> have an 'i_' in front of the property name, while if that property was not
> an indicator it would not have an 'i_'.  The Indicator object would be used
> to store complex indicators (i.e. patterns), although atomic indicators
> could still also be stored using this approach (hence the two ways to
> represent an indicator).

I do not like this proposal at all.

The big issue w/ this is that this implies that an observed-data
will be modified after it is created.  As there is no way for, at
the time of capture (unless a pattern already exists for this, but
then why?), to know that a particular Cyber Object should be an
indicator or not.  observed-data's are considered a "fact" that
has happened.  Modifications of them are abnormal.

It also means that if you are trying to match against a pattern, you need
to match against both value and any i_*.  This is because any data
recorded on an observed-data is a "fact" so, it doesn't make sense to
add an indicator to an observed-data, but that you "convert" an existing
observed-data field into an indicator.

This also means that there will be two ways to do the same thing.  This
is too much like the old way of handing indicators which had tons of
issues in writting a proper matching engine for.

This also makes quering indicators from a TAXII feed much more
difficult.  W/ the dedicated object, you just asked for Indicators,
but for this, you'd need to ask for all observed-data that have
a i_* field, which will be a very complex query/search unless you
create a special index on it.

To know if a cyber object contains an indicator, we have to scan ALL
the properties of it.  Considering that we have 3+ level deep objects
currently (network-traffic->extensions->http-request-ext for example),
that's a lot of extra parsing/scanning just to figure out if an
observed data has an indicator on it or not.

IMO, it's not hard at ALL to convert an observed-data field into a
pattern.  If you really do have your example below, you simply create
a pattern as such:
[ $type:value = 'badguy@harpoons.com' ]

Yes, it does mean that you now have to create/publish a new SDO, but
you still had to do that w/ the proposal, as you had to publish a
modified SDO.

Doing the modification also presents an interesting problem of sharing
PII.  The proposed method automatically includes unneeded information
for matching, while programatically generating them
sanitizes information.

I do understand the desire to make indicators more simple, but IMO,
for simple indicators, it's simple enough to create a pattern, that
I don't see a reason to add a different way to do the same thing.

> For example:
> {
>   "type": "observed-data",
>   "id": "observed-data--b67d30ff-02ac-498a-92f9-32f845f448cf",
>   "created_by_ref": "identity--f431f809-377b-45e0-aa1c-6a4751cae5ff",
>   "created": "2016-04-06T19:58:16.000Z",
>   "modified": "2016-04-06T19:58:16.000Z",
>   "first_observed": "2015-12-21T19:00:00Z",
>   "last_observed": "2015-12-21T19:00:00Z",
>   "number_observed": 50,
>   "objects": {
>   	"0": {
> 	    "type": "email-addr",
> 	    "i_value": "badguy@harpoons.com",
> 	    "display_name": "Bad Guy"
>   	},
>   	"1": {
> 	    "type": "email-addr",
> 	    "value": "victim@clicksonanything.com",
> 	    "display_name": "Vic"
>   	},
>   }
> }
> 
> Two email addresses are represented within the objects.  badguy@harpoons.com
> is an indicator, while victim@clicksonanything.com is not an indicator.
> This allows users (or implementers) to easily create atomic indicators
> without having to represent the email address within the context of the
> event and then separately create an indicator object with the same email
> address, linking the two together.
> 
> In order for this approach to work, we would need to define how each data
> type (including complex datatypes, such as windows-registry-value-type)
> would be used as an indicator.  In most cases this would be simple, but in
> complex datatypes such as windows-registry-value-type, we would need to
> specify how the key and values are used to create an indicator.  
> 
> I do not believe this functionality would need to go in the 2.0 release of
> STIX.  In my view, this does not break backwards compatibility, although it
> is a fairly significant change that would break forwards compatibility.  
> It also is a little be it of a hack, which I'm not a fan of and puts extra 
> work onto the developer to support the capability.  Truthfully, I'd rather 
> put more work on the developer to make things easier on the users 
> rather than making it easier on the developer to implement something 
> that puts the burden onto the user.  
> 
> Interested in everyone's thoughts,

-- 
John-Mark