Hello,
I'm new to STIX and I've been evaluating the use of STIX 2.0RC3 Patterns ([0]) for some use-cases. I find it to be quite a powerful tool. However, there are a couple concepts I don't know how to express. I'm hoping the community might be able to help me out -- either there's a usage I don't see (missed in my reading) or there's an oversight in the language. For any of the latter, I hope I'm not too late for my suggestions to be considered for 2.0 timelines.
(1) Dictionary key absence. For instance, looking for some network-traffic property AND the absence of a specific HTTP header. Do I treat the dictionary as a collection and use NOT IN comparison operator? That seems to violate the grammar rules, which say the LHS (left-hand side) of an IN clause is the Object Path and the RHS (right-hand side) is a set of constant values.
(2) Related to (1), the same syntax question is raised for optional Object Paths (properties) which are not present. For example, a path like "file:size" describes an optional property of the File Object. How to check for (the absence of) that field? Is there a "null/nil/None" object value that can be checked for? What's the syntax for the check? What primitive types support it? Some candidate syntax comes to mind: "[file:size != nil]" or maybe "[file:size IS NOT null]". Where is that discussed in the spec?
(3) Also related to (1), the comparison _expression_ is always Object Path on the LHS, literal value on the RHS. This is inflexible, and means there's no way to compare two Object Paths to each other. It also means I cannot check to see if Object Path A is present in Object Path B where B is a collection.
(4) Speaking of collections, collection object lookup syntax is divergent. For list types (Part 4, Section 5.2), we have a 0-based index with square-brackets ('[]'). We also have a convenience syntax of "list_property[*]" as syntactic sugar for the logical ANY operator from SQL. However, dictionary types (Part 4, Section 5.3) are referenced with "dot-notation", just like any object property. This obscures the property's type for a casual reader and restricts (or at least confuses) the body of syntax available for expressions on dictionary elements. For example, there's no ANY equivalent for matching dictionary members like we can for lists.
(5) Related to (4), there appears to be no syntax for the other collection-based logical operators provided by SQL -- ALL, SOME, EXISTS. As mentioned in (2), (3), there is IN syntax, but it's not available for Object Path element collections, only constant set literals.
(6) While talking about logical operators, I haven't noticed the equivalent of a BETWEEN _expression_. One must say, for example "[type:property > A AND type:property < B]". I haven't thought in depth on this topic, could be other non-numerical types for which it's less obvious how to express value constraints, for which BETWEEN syntax goes from not just "nice to have" but "required for _expression_" -- ie, where the semantics of greater-than, less-than do not make sense but BETWEEN does.
(7) Is there any thought around a library of functions for primitive types? For instance, a length() function that operates on strings.
(8) The next logical question from (7) is where would UDF's or implementation-specific extensions be installed into the syntax? It would be very powerful to have a Function Object (including a function implementation) that can be exported along with the Indicators that contain Patterns that make use of that function.
(9) Also from (7), any consideration for aggregation operators/functions? "Match when the combined file size of all attachments on the Email Message is greater than 5mb". Assuming the Email Message Object had a property "attachment_refs" of type object-ref list that's restricted to File Object types, that might look like "[sum(email-message:attachment_refs[*].size) > 5 * 1024 * 1024]".
(10) It's not clear to me if the RHS from the example in (9) is even valid -- are arbitrary mathematical expressions legal for either LHS or RHS of comparison expressions?
(11) Capability for backward references. I'd like to be able to refer to the value that matched a previous _expression_ later in the pattern. For instance, "match when an email contains a URL hosted on
baddomain.com and subsequent http traffic contains a 200 success request for that URL". One approach might be to borrow group backreferences from regex syntax, "[(email-message:url_refs[*].value MATCHES '*.
baddomain.com/.*\.docx')] FOLLOWED BY [network-traffic:extensions.http-request-ext.request_value = \1 AND network-traffic:extensions.x-examplecom-http-request-ext.status_code = 200]". I don't quite know how this would work -- you'd want to define capturing groups of arbitrary expressions and also allow for capturing groups to be embedded into the RHS of a MATCHES _expression_. Tricky.
Thanks a lot!
Nick