Re: [cti] CybOX Datatype Refactoring/Deprecation

Yup, I totally agree that having a single approach may render this capability unusable for some folks, but it would be interesting to hear directly from the analysts/users that require this capability in their day to day operations. I know Pat has spoken for DSIE, but I’m just wondering if it other communities have the same issues; would base64-based defanging be a deal breaker?

Regards,

Ivan

From: Sean Barnum <sbarnum@mitre.org>
Date: Monday, February 22, 2016 at 10:29 AM
To: Ivan Kirillov <ikirillov@mitre.org>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

If we attempted to force such a decision then yes it would require less fields.

I would worry about taking such an approach however given that the real-world community of practitioners do NOT have such a "ONE standard way of performing defanging/refanging on all fields” and it is not for lack of trying. My understanding is that many different parties over the years have tried to drive towards a single algorithm and have not been successful. Different communities/players have strong feelings that certain approaches offer necessary capabilities that others do not.

I know that Pat Maroney has been part of these sorts of discussion within DSIE and elsewhere so I would encourage him to comment here.

The danger of us thinking we can decide and enforce the ONE way of doing something even though the communities that we are attempting to support clearly do not yet agree is that we could be making STIX/CybOX untenable for whole swaths of players. I would think this issue potentially very much akin to the whole XML vs JSON issue.

I think it makes total sense to work to identify ONE default approach but don’t think we can then simply disallow all other approaches. I think 2 or 3 fields is an acceptable price to pay for something like this.

sean

From: Steve Cell <ikirillov@mitre.org>
Date: Monday, February 22, 2016 at 12:11 PM
To: "Barnum, Sean D." <sbarnum@mitre.org>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

Ah, I see – that makes sense. So I suppose the corollary to this is, what if we specified ONE standard way of performing defanging/refanging on all fields, such as by base64-encoding the data? Seems like this would make the specification simpler, and consumers would only need to understand/implement one refanging method.

Regards,

Ivan

From: Sean Barnum <sbarnum@mitre.org>
Date: Monday, February 22, 2016 at 10:02 AM
To: Ivan Kirillov <ikirillov@mitre.org>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

>[Ivan] That makes sense, though I do wonder if in such a case Party A will indeed provide refanging information if they do not have certainty about the defanging method used. >Also, the refanging field as designed before was meant to be a machine-parseable transform (e.g, a regex); is there still the expectation that it is actually used in this way (i.e., >that consumers would actually machine parse it and apply it to the data) in the wild?

I apologize if my comment on refanging was confusing. It was not intended to be tied directly to the case of no defanging algorithm provided.

It was just meant in general having the ability to specify a refanging algorithm. I would certainly think that having an explicit way to automatically refang content (within a GUI for example) would still be very useful. If everyone was using the same defanging algorithm then the refanging could likely be hardcoded and would not need a separate field but I do not believe that this is the reality we live in or are likely to live in for a while. Some communities or organizations use diverse defanging approaches and the only way to refang such content automatically would require explicit specification of a refanging algorithm.

</my-two-cents>

sean

From: Steve Cell <ikirillov@mitre.org>
Date: Monday, February 22, 2016 at 11:54 AM
To: "Barnum, Sean D." <sbarnum@mitre.org>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

1. Could we limit the properties associated with obfuscation and defanging/refanging to the following?
  1. obfuscation_algorithm_ref (URI): a reference to a description of the algorithm used to obfuscate the object property
  1. defanging_algorithm_ref (URI): a reference to a description of the algorithm used to defang the object property

[Ivan] Yes, that’s correct – John and I thought that we could likely remove the is_obfuscated/is_defanged fields, since the presence of the _ref fields is enough to state that a field is obfuscated and/or defanged. Do we as the community feel that not specifying a reference to the obfuscation/defanging algorithm used would be a common occurrence?

My understanding of a common use case is that Party A passes along data to Party B that was not originally produced by Party A where they have determined the data is obfuscated or defanged and wish to convey that to Party B but do not have certainty on the exact algorithm used.

Either way, I would think you will also need to include refanging

[Ivan] That makes sense, though I do wonder if in such a case Party A will indeed provide refanging information if they do not have certainty about the defanging method used. Also, the refanging field as designed before was meant to be a machine-parseable transform (e.g, a regex); is there still the expectation that it is actually used in this way (i.e., that consumers would actually machine parse it and apply it to the data) in the wild?

Regards,

Ivan

From: Sean Barnum <sbarnum@mitre.org>
Date: Tuesday, February 16, 2016 at 7:43 AM
To: Ivan Kirillov <ikirillov@mitre.org>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

Comments inline

From: <cti@lists.oasis-open.org> on behalf of Steve Cell <ikirillov@mitre.org>
Date: Monday, February 15, 2016 at 10:45 AM
To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

Are there any additional thoughts on this topic? Just trying to see if have any consensus on:

This overall approach, i.e., supporting specific metadata properties as part of CybOX Object fields (implemented via JSON Objects)
The specific set of metadata properties that we should support
1. Does it make sense to include an “is_random” property for assertions of randomness in the data of a particular field?

[sean]I have no dog in this hunt but know that this field came from request came from digital forensics use cases where they were fully aware that this was an observation and not a pattern. We should ask Eoghan for his thoughts on this.

1. Could we limit the properties associated with obfuscation and defanging/refanging to the following?
  1. obfuscation_algorithm_ref (URI): a reference to a description of the algorithm used to obfuscate the object property
  1. defanging_algorithm_ref (URI): a reference to a description of the algorithm used to defang the object property

[sean]So for these I assume you are suggesting that the presence of these fields implicitly means that the field is obfuscated or defanged thereby negating the need for the explicit field. The only potential issue I see with that is when the field is obfuscated or defanged but no algorithm is provided or available. My understanding of a common use case is that Party A passes along data to Party B that was not originally produced by Party A where they have determined the data is obfuscated or defanged and wish to convey that to Party B but do not have certainty on the exact algorithm used.

Either way, I would think you will also need to include refanging

Regards,

Ivan

From: Terry MacDonald <terry@soltra.com>
Date: Tuesday, February 9, 2016 at 4:49 PM
To: Ivan Kirillov <ikirillov@mitre.org>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: RE: [cti] CybOX Datatype Refactoring/Deprecation

What about using the relationships to handle the levels of obfuscation? That would allow description of obfuscated and encrypted objects within other obfuscated objects.

In other words to describe a PDF containing obfuscated _javascript_ that decodes and extracts a windows executable from within the PDF we could do something like this:

https://docs.google.com/drawings/d/1ozVBXQcFWja5yMAKDRfqkKz_1KfDZwZoVf5IyO7ElRg/pub?w=667&h=560

Would that sort of thing work? BTW the relationship thing seems pretty darn powerful when used like this….

Cheers

Terry MacDonald

Senior STIX Subject Matter Expert

SOLTRA | An FS-ISAC and DTCC Company

+61 (407) 203 206 | terry@soltra.com

From:cti@lists.oasis-open.org [mailto:cti@lists.oasis-open.org] On Behalf Of Kirillov, Ivan A.
Sent: Wednesday, 10 February 2016 8:16 AM
To: cti@lists.oasis-open.org
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

>> As much as I would like fields to just be the value, I agree that we probably need this layer of indirection to support these extra properties.

I was also hoping for Observations/Objects to just hold the value, but some of these fields do seem to be necessary. I think the only other way to do this would be via relationships, but then you’d have to assign an ID to each field, and that would just get messy quickly without any real benefit.

>> We’ll need to decide what that means for patterning, if anything, by the way.

Yup, agreed. My initial thinking is that these properties don’t (or shouldn’t) have any significance for patterning.

>>Given that we’re talking about actual observations here (instances), is “appears_random” appropriate? That seems to be a statement about a pattern of behavior over several observations rather than a single observation.

I’ve wondered the same – this does seem to be an assertion that one wouldn’t typically make based on a single data point. Also, it seems like you’d want to be able to specify some level of confidence and perhaps other properties with this as well, so I’m also wondering if this belongs here or perhaps in some separate structure (which could also be used for related observations such as entropy, etc.)?

>>Can we combine the “obfuscation” fields into some single “obfuscation_algorithm”, representing a URI or something? Then if the field is there, it’s obfuscated using that. If it’s not, then it’s not obfuscated.

Yeah, that makes good sense to me. I think having the “is_defanged” boolean isn’t strictly necessary, as you can infer that something is obfuscated if the corresponding “obfuscation_algorithm” field is populated. Having a URI in there seems reasonable.

>> Given that we’re now talking about machine-to-machine transmissions, is defanging still important? Can’t the raw data be exchanged as it is and then let tools to present it to users defanged however they want? It seems weird to me to transmit it defanged if it’s M2M (vs. a PDF that people might accidentally click).

I had the same question when we were first implementing defanging/fanging; it does seem like something that should be done at the data ingest/GUI layer rather than at the data specification layer. It’s definitely odd to support it if there is no expectation that humans will actually look at the data.

>> If we do need to include it, can we similarly collapse these attributes into one “defanging_algorithm” URI? They seem very redundant right now.

I’m also onboard with this. So we’d get something like:

obfuscation_algorithm_ref (URI): a reference to a description of the algorithm used to obfuscate the object property
defanging_algorithm_ref (URI): a reference to a description of the algorithm used to defang the object property
refanging_transform (string): an automated transform that can be applied to the object property in order to refang it to its original format

Regards,

Ivan

From: <cti@lists.oasis-open.org> on behalf of John Wunder <jwunder@mitre.org>
Date: Tuesday, February 9, 2016 at 1:56 PM
To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] CybOX Datatype Refactoring/Deprecation

As much as I would like fields to just be the value, I agree that we probably need this layer of indirection to support these extra properties. We’ll need to decide what that means for patterning, if anything, by the way.

On the attributes you listed, I just have a few questions:

Given that we’re talking about actual observations here (instances), is “appears_random” appropriate? That seems to be a statement about a pattern of behavior over several observations rather than a single observation.
Can we combine the “obfuscation” fields into some single “obfuscation_algorithm”, representing a URI or something? Then if the field is there, it’s obfuscated using that. If it’s not, then it’s not obfuscated.
For defanging, I have a two part question:

Given that we’re now talking about machine-to-machine transmissions, is defanging still important? Can’t the raw data be exchanged as it is and then let tools to present it to users defanged however they want? It seems weird to me to transmit it defanged if it’s M2M (vs. a PDF that people might accidentally click).
If we do need to include it, can we similarly collapse these attributes into one “defanging_algorithm” URI? They seem very redundant right now.

John

From: <cti@lists.oasis-open.org> on behalf of Ivan Kirillov <ikirillov@mitre.org>
Date: Tuesday, February 9, 2016 at 12:21 PM
To: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: [cti] CybOX Datatype Refactoring/Deprecation

Sending this to the broader CTI since it’s part of the STIX/CybOX Indicator tranche.

Given that we’ll be splitting out embedded patterning from CybOX, there are changes that we need to make to the base datatypes used currently for specifying properties in CybOX Objects. At a high level, this involves the following:

Deprecating any types/enums associated with patterning
Refactoring the base BaseObjectPropertyType (the base type for ALL Object properties) to remove patterning and permit more accurate datatype based field specification in Objects (e.g., an integer is represented as an integer and not a string). Accordingly, this would entail creating a new BaseStringObjectPropertyType and BaseNumberObjectPropertyType that extend from the refactored BaseObjectPropertyType.
Updating the existing ObjectPropertyTypes (e.g., StringObjectPropertyType, DoubleObjectPropertyType, etc.) to extend from the new BaseString/Number ObjectPropertyTypes

More details on these proposed changes can be found here: https://github.com/CybOXProject/schemas/issues/416#issuecomment-181916815

Accordingly, this would result in CybOX instance content that looks like the following; each property is an Object (so that it can specify metadata about refanging/defanging/etc) and thus the “value” field MUST be included in every property to capture its actual value.

   "size": {"value": 23134},

   "file-system-properties" : {"file_name": {"value":"test.dll",

                                             "observed_encoding":"utf-8"}}

Some open questions:

Does this approach seem reasonable?
Do we need all of the properties defined on the BaseObjectPropertyType (see below), or are there other properties that we should consider adding?

appears_random (boolean): whether the property appears random in nature
is_obfuscated (boolean): whether the property has been obfuscated
obfuscation_algorithm_ref (string): a reference to a description of the algorithm used to obfuscate the object property
is_defanged (boolean): whether the property has been defanged
defanging_algorithm_ref (string): a reference to a description of the algorithm used to defang the object property
refanging_transform_type (string): the type of refanging transform specified in the refanging_transform property
refanging_transform (string): an automated transform that can be applied to the object property in order to refang it to its original format
observed_encoding (string): the encoding of the string as observed

For the extensions of the BaseStringObjectPropertyType and BaseNumberObjectPropertyType, such as PositiveIntegerPropertyType, should we define additional patterns (i.e. regular expressions) for field-level validation of these types? E.g., in this example, to make sure that the number is a valid positive integer?

Regards,

Ivan

cti message