Re: [cti] IDs with uuidv5 too

As the original proposer of opening up flexibility to UUIDv5, I'd like to point out a simple solution that was part of the original proposal: prefix the seed with your Namespace (e.g., my company.com, cyberdivision1.my company.com)

All we are asking for is removing the MUST constraint to UUIDv4. Since the version numbers are encoded in the UUID (and therefore testable for compliance), the UUIDv4 constraint prevents other means of generating UUID that would provide value to many in the community.

The only functional STIX Requirement is that these UUIDs be unique. If we were to codify the use of organization namespaces as the prefix to any UUIDv5, we have a very low probability of collisions.

Patrick Maroney

Principal Engineer - Data Science & Analytics

Wapack Labs LLC

(609)841-5104

pmaroney@wapacklabs.com

Public Key: http://pgp.mit.edu/pks/lookup?op=get&search=0x7C810C9769BD29AF

On May 22, 2017, at 5:22 PM, Terry MacDonald <terry.macdonald@cosive.com> wrote:

Hi Jason,

It can't result in collisions if every vendor uses a different starter uuid. If you produce two UUIDv5 outputs, use a different starter uuid for each one yet use the same internal primary identifier each, you will still get two different UUIDv5 outputs.

My proposal goes even further than requiring different starter UUID's for each vendor, as I suggest we require each vendor to have their own different start UUID's for each type of object that they create. This makes the 'nsmespaces' even smaller, reducing the chance of collisions even further.

My proposal removes the requirement for vendors to make structural changes to their internal databases in order to bolt-on STIX support. They also don't need to sort and additional massive lookup table as you have done.

I've heard from a few vendors that this is how they are doing it already internally. I figured that if it's already being done then maybe it's a need that's currently missing from the standard. Hence my proposal.

I'd much rather have this officially allowed as an option for generating IDs rather than people doing it 'illegally'.

Cheers

Terry MacDonald

Cosive

On 23/05/2017 00:24, "Jason Keirstead" <Jason.Keirstead@ca.ibm.com> wrote:

Sorry got cut off...that should conclude..

"Your proposal is going to result in ID collisions"

--
Sent from my mobile device, please excuse any typos.

me --- Re: [cti] Re: IDs with uuidv5 too ---

From: "Jason Keirstead" <Jason.Keirstead@ca.ibm.com>
To: "Terry MacDonald" <terry.macdonald@cosive.com>
Cc: cti-stix@lists.oasis-open.org, "CTI TC Discussion List" <cti@lists.oasis-open.org>
Date: Mon, May 22, 2017 11:22 AM
Subject: Re: [cti] Re: IDs with uuidv5 too

We ran into this as well, and simply are going to maintain a lookup map and stick our internal ID on the object as a custom property. I fail to see why it is such a large problem... "Grow huge very quickly"... Define huge.. a lookup map with 100 million entries easily fits in memory.

Your proposal is

--
Sent from my mobile device, please excuse any typos.

Terry MacDonald --- [cti] Re: IDs with uuidv5 too ---

From: "Terry MacDonald" <terry.macdonald@cosive.com>
To: cti-stix@lists.oasis-open.org, "CTI TC Discussion List" <cti@lists.oasis-open.org>
Date: Fri, May 19, 2017 6:46 PM
Subject: [cti] Re: IDs with uuidv5 too

An important point to make, is that this doesn't affect versioning in any way. The use of UUIDv5 does not mean that the STIX object is immutable (which was the big discussion last year). The generated UUIDv5 is solely focused on the vendor's original internal record's primary key, which means that the contents of the STIX object can be changed/updated/versioned without changing its STIX ID. The STIX ID will change only when the Vendors internal primary key changes, which is supposed to happen.

Cheers

Terry MacDonald | Chief Product Officer

<cosive_mail_signature.png>

M: +64 211 918 814
E: terry.macdonald@cosive.com
W: www.cosive.com

On Sat, May 20, 2017 at 9:23 AM, Terry MacDonald <terry.macdonald@cosive.com> wrote:
Hi all,

As part of recent integration work we've been doing with a Threat Intel vendor, it's become apparent that the current restriction in uuidv4 generation of STIX IDs causes extra work for vendors. I firmly believe that we need to add the ability to generate uuidv5 based STIX IDs as well in order to make implementation simpler.

The problem:
Most Threat Intel vendors at present publish their threat intelligence through their own web based platform. Most vendors also support a JSON based REST API that they allow their authenticated users to access the same Threat Intel through. Many vendors are now adding STIX and TAXII support alongside their previous JSON API, and it's this Co-existence that's causing the issue.

All Threat Intel vendors I've spoken to use a proprietary identifier as their primary key. Any relationships that link their internal Intel together is done using that proprietary primary key.

This causes massive problems for the vendors when it comes time to translate their data into STIX. They need to somehow maintain a relationship between their internal primary id and the STIX id that is mandated by the standards.

This effectively forces the vendor to either:
maintain an ID translation table that records all STIX IDs sent out and the corresponding internal primary key it relates to, or
change their internal intelligence database to include STIX IDs generated when the data is first added.
The first suggestion becomes unwieldy very quickly, as the amount of data to track would grow huge very speedily. The second suggestion is the right longer term solution, a bit requires changing the vendors main data source which they are often very reluctant to do. Most Threat Intel vendors are just experimenting with STIX right now.

The solution:
We've found the quick and easy solution is to tell the that Intel vendor to generate uuidv5 STIX IDs based solely on the primary key. UUIDv5 takes a 'seed' uuid, and combines it with a known value (the primary key) and results in a resulting uuid that is derived from the primary key.

This makes the vendors job much easier, as they can now easily bolt-on STIX IDs during the STIXification of their Threat Intel data, and they don't need to make any big changes to their main Intel systems. This makes it much more likely that the vendor will look at supporting STIX and TAXII, as it's not so impactful on their current operations.

Recommendation:
We recommend that UUIDv5 is added to the six 2.1 standard alongside UUIDv4 as an acceptable way to generate STIX IDs. Vendors who choose to use UUIDv5 method of uuid creation must use their internal threat intelligence proprietary identifier (primary key) as the value that is passed into the UUIDv5 generation process. We also recommend that the seed uuid value is different for each STIX object type to minimize the chance of uuid collisions.

import uuid
# randomly selected starter UUID for Indicators
INDICATOR_NAMESPACE_SEED = uuid.UUID('a288ef91-8db3-46de-22ae-8c13fe286599')
# Vendor's internal ID for the record we are turning into a STIX Indicator
VENDOR_INTEL_INTERNAL_ID = "id-1.1.1.1"
# create Indicator STIX ID from the vendors internal ID using UUIDv5
indicator_stix_id = "indicator--" + str(uuid.uuid5(INDICATOR_NAMESPACE_SEED, VENDOR_INTEL_INTERNAL_ID))
# print the Indicator STIX ID
print (indicator_stix_id)
# The STIX ID for record 'id-1.1.1.1' will always be
# indicator--f5704f38-a888-58ed-9fc3-f6659c658847
# using the UUIDv5 method

By restricting the UUIDv5 so it's based on the primary key we will make it far easier for vendors to generate STIX from data they already have. This in turn will make it far more likely that they will.

Could this be discussed and agreed at the Face-2-face?

Cheers
Terry MacDonald
Cosive

cti-stix message