OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

cti message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [cti] Re: Observable Patterning


I would agree Mark, I do think the topics are intertwined. Even after looking at this for some time, I still couldn't decide whether patterning should sit within CybOX, STIX or both. So I decided to try and take a step back to look at what the bigger picture.

I decided to look at the type of scenarios that one would want to describe, and see which patterning option would best cover those scenarios. As I see it there are a few major types of scenarios one would want to capture using patterns:
I used the above scenarios along with the following guidance found within the STIX and CybOX sites:

I then tried to pull these together to describe the 3 different options we have for where patterns sit. 

At this point, I prefer Option 1, but I would like to see what others think.

-----

Option 1: Patterns only in CybOX, removed from STIX

This option changes the Indicator object to only have a single Cybox Pattern description in it. That single pattern contains as little or as much information in it to describe the scenarios listed above. Using the great posts within the Github issue (https://github.com/CybOXProject/schemas/issues/381) I've attempted to describe the 4 scenarios I listed earlier using a JSON style notation, as if the patterns and sequencing was done within CybOX. THe resulting psuedo code is below. Please note: The cybox patterns would all sit within a STIX Indicator object. I haven't put the STIX Indicator in the psuedo code.

Single item:

pattern: {
        id: id1,
        objects: [{
                id: obj1 
                type: DomainObject
                properties: [{
                        path: /type
                        condition: EQUALS  
                        value: FQDN
                },{
                        path: /value
                        condition: EQUALS  
                        value: www.badsite.com
                }]
        }],
        condition {
                obj1
        }
 }

Single regex:

pattern: {
        id: id2,
        objects: [{
                id: obj1 
                type: DomainObject
                properties: [{
                        path: /type
                        condition: EQUALS  
                        value: FQDN
                },{
                        path: /value
                        condition: MATCHES  
                        value: xx3\[a-b]{3}.*\.com
                }]
        }],
        condition {
                obj1
        }
 }

Group of Patterns:

pattern: {
        id: id3,
        objects: [{
                id: obj1 
                type: FileObject
                properties: [{
                        path: /hashes/hash/simple_hash_value
                        condition: EQUALS  
                        value: c38862b4835729d979e7940d72a48172
        },{
                id: obj2 
                type: FileObject
                properties: [{
                        path: /filename
                        condition: CONTAINS  
                        value: malicious.dll
                }]
        }],
        condition {
                obj1 OR obj2
        }
 }

Sequence of Group of Patterns:

pattern: {
        id: id4,
        objects: [{
                id: obj1 
                type: DomainObject
                properties: [{
                        path: /type
                        condition: EQUALS  
                        value: FQDN
                },{
                        path: /value
                        condition: EQUALS  
                        value: www.badsite.com
                }]
        }],
        condition {
                obj1
        }
 },
pattern: {
        id: id5,
        objects: [{
                id: obj1 
                type: FileObject
                properties: [{
                        path: /hashes/hash/simple_hash_value
                        condition: EQUALS  
                        value: c38862b4835729d979e7940d72a48172
        },{
                id: obj2 
                type: FileObject
                properties: [{
                        path: /filename
                        condition: CONTAINS  
                        value: malicious.dll
                }]
        }],
        condition {
                obj1 OR obj2
        }
 },
pattern_sequence: {
        id: seqid1,
        sequence: [{
                idref: id4,
                order: 1
        },{
                idref: id5,
                order: 2,
                after: PT5S   #ISO8601 duration standard format (after 5 secs)
                within: PT1M   #ISO8601 duration standard format (within 
        }]

}

Pros:
Cons: 
-----

Option 2: Patterns only in STIX, uses CybOX as it is used now

This option reuses the Indicator object's Observable Composition as the thing that offers the patterning. This is trying to use the existing way of doing things but in a pattern style. Using a JSON style notation, the resulting psuedo code is below. Please note, there isn't any way of sequencing indicators easily at present within STIX, so I took the liberty of adding the indicator_sequence section (although there is bound to be a better way of doing it). There are Observable Events/Actions that could be used instead to potentially describe the temporal nature of the indicator sequence but I couldnt work out how to make that easily discernable.

Single item:

indicator: {
        id: id1,
        observable: {
                object: [{
                        id: obj1 
                        type: DomainObject
                        properties: [{
                                path: /type
                                condition: EQUALS  
                                value: FQDN
                        },{
                                path: /value
                                condition: EQUALS  
                                value: www.badsite.com
                        }]
                }],
        }
 }

Single regex:

indicator: {
        id: id2,
        observable: {
                object: [{
                        id: obj1 
                        type: DomainObject
                        properties: [{
                                path: /type
                                condition: EQUALS  
                                value: FQDN
                        },{
                                path: /value
                                condition: EQUALS  
                                value: xx3\[a-b]{3}.*\.com
                        }]
                }],
        }
 }

Group of Patterns:

indicator: {
        id: id3,
        observable: {
                observable_composition: {
                        operator: OR
                        observable: {
                                object: {
                                        id: obj1 
                                        type: FileObject
                                        properties: [{
                                                path: /hashes/hash/simple_hash_value
                                                condition: EQUALS  
                                                value: c38862b4835729d979e7940d72a48172
                                        }]
                                }
                        },
                        observable: {
                                object: {
                                        id: obj1 
                                        type: FileObject
                                        properties: [{
                                                path: /hashes/hash/simple_hash_value
                                                condition: EQUALS  
                                                value: c38862b4835729d979e7940d72a48172
                                        }]
                                }
                        }
                }
        }
}

Sequence of Group of Patterns:

indicator: {
        id: id4,
        observable: {
                object: [{
                        id: obj1 
                        type: DomainObject
                        properties: [{
                                path: /type
                                condition: EQUALS  
                                value: FQDN
                        },{
                                path: /value
                                condition: EQUALS  
                                value: xx3\[a-b]{3}.*\.com
                        }]
                }],
        }
 },
indicator: {
        id: id3,
        observable: {
                observable_composition: {
                        operator: OR
                        observable: {
                                object: {
                                        id: obj1 
                                        type: FileObject
                                        properties: [{
                                                path: /hashes/hash/simple_hash_value
                                                condition: EQUALS  
                                                value: c38862b4835729d979e7940d72a48172
                                        }]
                                }
                        },
                        observable: {
                                object: {
                                        id: obj1 
                                        type: FileObject
                                        properties: [{
                                                path: /hashes/hash/simple_hash_value
                                                condition: EQUALS  
                                                value: c38862b4835729d979e7940d72a48172
                                        }]
                                }
                        }
                }
        }
},
indicator_sequence: {
        id: seqid1,
        sequence: [{
                idref: id4,
                order: 1
        },{
                idref: id5,
                order: 2,
                after: PT5S   #ISO8601 duration standard format (after 5 secs)
                within: PT1M   #ISO8601 duration standard format (within 
        }]

}

Pros:
  • reuses existing Indicator Observable_Composition objects
Cons: 
  • patterns are only described within STIX. Not available to any other languages such as MAEC or CAPEC
  • Patterns don't live within cybox packages, so might be harder to keep STIX and CybOX in step.

-----

Option 3: Patterns only in STIX, no CybOX

This option would make patterns only in the STIX Indicator object and not use CybOX objects at all, which in my opinion is wrong. So I didn't even bother fleshing out that scenario :).



Cheers

Terry MacDonald | STIX, TAXII, CybOX Consultant




Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers.

On 29 September 2015 at 02:34, Davidson II, Mark S <mdavidson@mitre.org> wrote:

I think there’s two intertwined topics to disambiguate:

 

·         Querying STIX information, where some see STIX as a graph-based model

o   E.g., Tell me all the Indicators related to this TTP

·         CybOX patterns, which are much more signature-like

o   E.g., Find the file with Hash=0x1234

 

When I think of a CybOX pattern, I tend to think of something that is a peer to signature/Test Mechanism.

 

Thank you.

-Mark

 

From: cti@lists.oasis-open.org [mailto:cti@lists.oasis-open.org] On Behalf Of Joep Gommers
Sent: Monday, September 28, 2015 11:58 AM
To: Kirillov, Ivan A. <ikirillov@mitre.org>; Jerome Athias <athiasjerome@gmail.com>; Patrick Maroney <Pmaroney@specere.org>
Cc: Terry MacDonald <terry.macdonald@threatloop.com>; cti@lists.oasis-open.org


Subject: Re: [cti] Re: Observable Patterning

 

Hi Ivan,

 

I’d be interested what the team feels about the relationship between these patterns and test mechanism concept. Please correct me if wrong;

- indicator patterns described a pattern of potentially malicious activity that is intended to convey the pattern in a way that a human can understand and that machines can translate into action-based use-cases (like detection)

- test mechanisms are a pre-computed or created pattern specific to a certain use-case (like snort detection, yara, etc.)

 

Since in both cases that might be conditional relationships and nested conditional calculations (e.g. If you don’t see X, then the previous variable in de flow is +1, if you do seeX, then the previous variable in the flow is –1) - it makes it ill-suited for graph representation in the context of graph querying.

 

J-

 

From: <Kirillov>, "Ivan A." <ikirillov@mitre.org>
Date: Monday, September 28, 2015 at 3:36 PM
To: Jerome Athias <athiasjerome@gmail.com>, Patrick Maroney <Pmaroney@specere.org>
Cc: Terry MacDonald <terry.macdonald@threatloop.com>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] Re: Observable Patterning

 

>- Human readable

     Agreed.

>- Machine friendly (parsing/computation, algorithmically friendly)

     Agreed.

>- Data graph friendly

     Not so sure on this one. 

 

While I agree that STIX/TAXII querying (wherever it ends up) is inherently graph-based, I don’t believe the same is true for Indicator patterning. The main difference is that, in my view, the majority of Indicator patterns are meant to be parsed and executed against data that is inherently flat – lists of files, lists of processes, lists of IP addresses, etc. Thus while the two may overlap in certain places, I’d be very hesitant about overloading an Indicator patterning structure to support graph based querying, though perhaps it could be extended to do so. Again, I think a primary focus here should be to keep Indicator patterns SIMPLE, so that they can easily be written and consumed by analysts.

 

Regards,

Ivan

 

From: Jerome Athias
Date: Saturday, September 26, 2015 at 12:38 PM
To: Patrick Maroney
Cc: Terry MacDonald, "cti@lists.oasis-open.org", Ivan Kirillov
Subject: Re: [cti] Re: Observable Patterning

 

so should we try to capture the requirements in one place for this change request?

Quickly:

- Human readable

- Machine friendly (parsing/computation, algorithmically friendly)

- Data graph friendly

 

2015-09-26 17:57 GMT+03:00 Patrick Maroney <Pmaroney@specere.org>:

Probably not clear that I've similarly argued to keep TAXII focused on the transport of packages.  In the query method suggested for consideration, the Pattern would be passed as a STIX (Query/RFI) package and results similarly returned (again as a STIX package).

 

It's important to visualize this a Data Graph that maps to the Conceptual CTI model (which is the basis for suggesting Cypher as an example representation).  One can describe/query precise graphs of Objects/Relationships, any Nodes/Edges off of a Root Node, etc.

 

Hope this makes sense (at least in terms of framing the proposal).

Sent from Outlook

 



On Fri, Sep 25, 2015 at 6:01 PM -0700, "Terry MacDonald" <terry.macdonald@threatloop.com> wrote:

I agree with Patrick's comments about ensuring pattern support for querying, but not the idea querying should be done in TAXII.

Querying and answering should be done within STIX, so that the querying and answering done by the language contained within TAXII, and so that TAXII is free to concentrate on only delivering content. We should instead develop a STIX query object, and a STIX reply object in order to support question and answer directly within STIX. I won't go into the specific argument why here and clog this thread, but it has been discussed on the TAXII list recently.  

Cheers


Terry MacDonald | STIX, TAXII, CybOX Consultant

 

 

 

Disclaimer: The opinions expressed within this email do not represent the sentiment of any other party except my own. My views do not necessarily reflect those of my employers.

 

On 26 September 2015 at 01:33, Patrick Maroney <Pmaroney@specere.org> wrote:

I would like to add a couple of conceptual ideas to this discussion (along with a specific illustrative reference):

 

(1)  If we architect our flexible "patterning" model correctly it 'should' be directly applicable to our TAXII Query Language.

 

(2) Graph data models provide some interesting correlations/visualizations.  As previously suggested neo4j's Cypher Query language might serve as a good conceptual model for how to express "patterns"

 

http://assets.neo4j.org/download/Neo4j_CheatSheet_v3.pdf

Patrick Maroney
President
Integrated Networking Technologies, Inc.
Desk: (856)983-0001
Cell: (609)841-5104
Email: pmaroney@specere.org

 



On Fri, Sep 25, 2015 at 8:12 AM -0700, "Patrick Maroney" <Pmaroney@Specere.org> wrote:

A general [+1] for everything @Ivan said below (although agnostic on some of the specifics)

 

To this "flexible" patterning construct, advocate inclusion of fixed/relative temporal relationship _expression_ (highly simplified examples:  "a" followed by "b" , "a" during "b" ) along with compound boolean and set theory _expression_.

Patrick Maroney
President
Integrated Networking Technologies, Inc.
Desk: (856)983-0001
Cell: (609)841-5104
Email: pmaroney@specere.org

 



On Fri, Sep 25, 2015 at 6:13 AM -0700, "Kirillov, Ivan A." <ikirillov@mitre.org> wrote:

To Terry’s point about tool-specific versus generic patterns  - to me at least, this is rather relative. While it’s true that there is often a hard dependency between a particular pattern language and engine/tool that consumes it, this is not always the case; for example, YARA rules can be consumed by several different tools, including Cuckoo Sandbox. Thus, going back to Mark’s point below, I don’t see a true semantic distinction between STIX/CybOX patterns and those expressed in other forms (test mechanisms) – ultimately they are meant for detecting badness, and either your tool(s) can parse and make use of them, or they cannot. 

 

I’m of the opinion that the value that CybOX offers in terms of patterning is with regards to expressing patterns against its Objects and their corresponding data models (very similar to OVAL in that respect, as Jerome alluded to :)), and enhancing this capability should be our primary focus. Trying to create a monolithic pattern language that subsumes all others is untenable (not that I think anyone is really suggesting this), and I think having Snort, YARA, and the like out there is a great thing; they do what they do very well, and thus we should leave their capabilities to them. I don’t view it as a negative if an Indicator pattern can be better expressed, say, through a YARA rule than through a CybOX pattern.

 

Also, I’m supportive of Bernd’s idea of a top-level Pattern structure, and also one that is relatively abstract and amenable to being used in different domains (e.g., fraud). Ultimately, I think this structure needs to be able to express some set of constraints against a data model and its fields, whether it’s a CybOX Object or something else, like a structured representation of a bank account (as an example). Therefore, perhaps it could even live as a separate effort/work product – Generic LangUage for Patterning (GLUP), anyone? :-) 

 

So, to offer up another straw man (and I think Jason had something very similar), it would be interesting to see if we can converge on a YARA-like patterning structure that is easy for humans to read and write, and also machine-parseable:

 

pattern example_1 

{

  objects:

$OBJ1 = {type = AddressObject,

         fields = [{“category”:”ipv4-addr”},

                   {“address_value”:”192.168.2.3”}]} 

  

$OBJ2 = {type = AddressObject,

         fields = [{“category”:”ipv4-addr”},

                   {“address_value”:”192.168.4.7”}]} 

condition:

    OBJ1 and OBJ2

}

 

Regards,

Ivan

 

On 9/25/15, 8:02 AM, "Davidson II, Mark S" <mdavidson@mitre.org> wrote:

 

Adding my thoughts to the Test_Mechanism idea.

 

In line with 'one way of doing things', my preference would be for all signature-ish things to be thought of in the same way. In my mind, "signature-ish things" includes SNORT, YARA, and STIX/CybOX patterns. This is a little different than what we have today, where CybOX patterns are separate from Test Mechanisms [1], and Test Mechanisms are intended to identify the CybOX pattern(s). The way I'm thinking would have all signature-ish things become peers.

 

If I'm whiteboarding a use-case for signature/Indicator exchange, it would be something along the lines of:

 

Title: Send a signature

 

Description: An analyst creates a signature/indicator and chooses to share it with partner organizations.

 

Main Success Scenario:

 

1. Org 1's analyst develops a signature (e.g., SNORT, CybOX pattern)

2. Org 1's analyst decides to share the signature; posts the signature to a TAXII Channel (perhaps as a STIX Indicator)

3. Org 2, which is subscribed to the TAXII Channel, receives the signature (or STIX Indicator) and

3a. The signature (or STIX Indicator) is entered into Org 2's internal review queue to decide whether to deploy the signature (or STIX Indicator)

4. The signature (or STIX Indicator) is acted on - either manually or automatedly - and is discarded or deployed (or some other action)

 

Each one of these steps probably has a one-level-deeper set of steps associated with them (and a bunch of dependiencies/requirement), but at least for me, this is how I understand the workflow from Indicator creation to information sharing.

 

Thank you.

-Mark

 

 

-----Original Message-----

From: Grobauer, Bernd [mailto:Bernd.Grobauer@siemens.com]

Sent: Friday, September 25, 2015 7:02 AM

Subject: RE: [cti] Re: Observable Patterning

 

Hi,

 

I don't really see Test_Mechanisms as a candidate for reusing the

patterning that we've been discussing. Each Test_Mechanism rule is a

rule derived from the Indicators, and written in the tools native

language.

 

The principle of having a test-mechanism derive from what is described

within an indicator using observable patterns is the intended meaning,

but this principle is not always followed: it may be quite complicated

to describe in STIX/CybOX, what can be succinctly described in a

certain pattern language or the authoring party simply does not want to

bother with this -- so the STIX indicator is also used as a vehicle to

exchange such a test mechanism, enriched with contextual information

contained in the indicator and referenced/referencing STIX objects,

but not containing corresponding observable patterns. I would guess

that we will continue to see such use of a TestMechanism within an

otherwise rather empty indicator (empty with respect to Observables),

even if the pattern language is evolved such that in theory it could

describe everything that is expressible in any thinkable test

mechanism.

 

It doesn't make sense to me to have patterning in the

Test_Mechanism unless it is natively in the tool that uses the rules.

 

I had made that proposal in one of my emails with the intent of encapsulating

complexity. I can see why one would not want to use a test mechanism

as container for STIX/CybOX containers -- after all, no other test mechanism

would have references out of the test-mechanism blob, whereas a

STIX pattern could have many references to objects/object properties

defined elsewhere.

 

But I really would like to get rid of AND/OR/...-composition on

the level of indicators and push the whole pattern language

into (oh no, here it comes) yet another STIX top level object

STIX_Pattern or whatever. Then allow STIX_Patterns to reference

other patterns for composition, but not Indicators.

 

The reason is that the whole logical and temporal composition

is really complicated. The Indicator is absolutely central

to STIX, but like I said above, there may be use-cases to

communicate things to look out for *without* the use of

CybOX and CybOX patterning but via a test mechanism (or,

I haven't given up on that idea, a STIX-standardized key/value-pair

mechanism for communicating patterns).

 

Now, if we encapsulate the complications of the pattern language

in something that is removed from the Indicator level, then

one can implement the non-patterning-part of STIX much easier

in order to use it for use cases as described above. Also,

using STIX for other domains (fraud and what not) as has

been mentioned on the list yesterday, could become easier, since

for these other domains, the full might of STIX patterns

may not be needed ... or such domains may even need something

else entirely. (Hm, I just realize that this reasoning

would actually support making patterning part of CybOX

as a CybOX-pattern-top-level object.)

 

In closing, because I have proposed yet another top-level STIX entity:

in one of his mails about a possible top-level targeting object,

Aharon was concerned about making things too complicated by adding

more and more top-level objects. That is a valid concern, but on the

other hand, having huge schema definitions for objects that encompass

all kinds of substructures, is also quite complicated.  So, while we

certainly should have good reasons for introducing new top-level

objects, I would say that given the right circumstances, new top-level

objects will actually decrease complexity.

 

So, in short:

 

- I fully agree that decoupling patterning and instances

  is the right way to go.

 

- I would like to encapsulate patterns in a top-level construct

  that may be referenced by STIX indicators while removing

  not/and/or composition for indicators.

 

- I am not quite sure yet whether the 'Pattern' top-level object

  would better be part of STIX or part of CybOX.

 

  If we think the patterning is reusable also for extended domains such

  as a possible SocEngOX (Social Engineering Observable EXchange)

  language and we think we will want to write things like

 

  (SocEng-Observation "Phone call from Mr. Evil") THEN

  (CybOX Email Object with sender "lucifer.evil@hell.com)

 

  then patterning is probably part of STIX rather than CybOX; otherwise,

  as a cyber-focused language, it might well be part of CybOX.

 

 

Kind regards,

 

Bernd

 

------

 

Bernd Grobauer, Siemens CERT

 

 

 

 

 




[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]