OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

cti-stix message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [cti-stix] Applying data markings


Answering Jason's question (“What does a TAXII server do if half the document is not public/available for a particular consumer?”) but here because it’s as good a place as any to answer it.

I am biased, but some Burger fellow did one solution for messaging gateways. The issue is isomorphic to the issue here. Namely, if you send a multipart document from a first messaging network to a second messaging network, what do you do if the target messaging network does not understand one of the body parts? The question in the email context is, “If a target network does not understand a particular body part, what should the messaging gateway do?” Options are to fail the message and tell the sender which part(s) are unknown on the other side, so the sender can fix it, silently fail the message, or just drop the parts of the message the recipient will not understand and pass through the remainder of the message.

The isomorphism is the producer of the STIX document is the first network and the consumer of the STIX information (note I said information and not document) is not privy to all parts of the STIX document. The messaging gateway in email is the TAXII server in CTI. ‘Understanding’ of the body part in email is whether the receiver is eligible to view the STIX information.s


Our solution was to say the sender marked parts of the message as critical, meaning if the recipient cannot understand them, then fail the entire message. Otherwise, if the sender marks a part as not critical, the messaging gateway will silently drop that part. The use case in the messaging world is when you send “Here is the PowerPoint” and the recipient cannot grok PowerPoints, you probably want the message to fail. On the other hand, when you send “Hello World” and Outlook attaches the friendly tnef.dat file, you really don’t care if it gets killed, because you had no idea it got inserted in the first place.

In the CTI case, the sender most likely is marking things RED or some other restriction knowing the rest of the data is useful. Otherwise, they would have marked the whole thing RED. However, maybe they are in a mixed user environment, and they mark something AMBER, such that they don’t want to mark the whole thing AMBER or RED, but if you are not at least the AMBER level, then you don’t get anything, even the GREEN stuff.

See https://tools.ietf.org/html/rfc3459


On Dec 11, 2015, at 5:26 PM, Taylor, Marlon <Marlon.Taylor@hq.dhs.gov> wrote:

Hi STIX SC,

New numbering this does not continue from the previous list. I suggest we note #1 (and address it at a later time) and focus on #2 to bring us closer to conclusion. 

1. It looks like we came across a folk (interdependency topic) in this approach, not a problem. 

Embedded vs Referenced content - FINAL DECISION TBD; this marking approach leans towards referenced contents but that doesn't mean it won't work with embedded. We can split the fork and go down both paths and see what happens at each end OR defer the difference for now.

I'm leaning towards referenced content for all things but there could be some suggestive use-cases for embedded (THIS SHOULD BE ADDRESSED IN A SEPARATE THREAD).

2. Expected behavior for unexpected Paths expressions: What should we do in the following unexpected scenarios. These are separate cases however you may suggest the same behavior.
- if the path _expression_ is invalid
- if the path _expression_ is a valid _expression_ but yields no result for the given document

Committee, how do you suggest we handle the scenarios above?

-Marlon 

 

From: cti-stix@lists.oasis-open.org on behalf of Wunder, John A.
Sent: Friday, December 11, 2015 1:10:44 PM
To: Taylor, Marlon; Barnum, Sean D.; Jason Keirstead; Aharon Chernin
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] Applying data markings

PS: Thanks for pulling this thread, identifying the error conditions is very important.

I think we now have 3 open issues:
  • What is the expected behavior in error scenarios?
  • Can you mark at the package level?
  • Should level 1 markings be MTI?

My impression is that the answer to the last one can be “no, for now”. So that leaves the top two. What do people think? The package level markings one seems to be a relatively even split at this point, does anyone have critical needs/desires either way?

John

From: <cti-stix@lists.oasis-open.org> on behalf of "Wunder, John A." <jwunder@mitre.org>
Date: Friday, December 11, 2015 at 1:07 PM
To: Marlon Taylor <Marlon.Taylor@hq.dhs.gov>, Sean Barnum <sbarnum@mitre.org>, Jason Keirstead <Jason.Keirstead@ca.ibm.com>, Aharon Chernin <achernin@soltra.com>
Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Subject: Re: [cti-stix] Applying data markings

  1. Agreed, an example of marking scope would really help.
  2. Good point, we need to specify error scenarios like you get at in (4) in the conformance requirements.
  3. Yep.
  4. Yes, though I would say the actual error conditions are slightly different:
    1. The _expression_ is invalid (incorrect syntax)
    2. The _expression_ doesn’t select anything (your case is a subset of this)
  5. For this writeup I assumed that a markable object cannot be embedded in a markable object based on us moving to reference-only. If we end up with another decision there we need to come back to this and handle this scenario.

From: Marlon Taylor <Marlon.Taylor@hq.dhs.gov>
Date: Thursday, December 10, 2015 at 6:12 PM
To: "Wunder, John A." <jwunder@mitre.org>, Sean Barnum <sbarnum@mitre.org>, Jason Keirstead <Jason.Keirstead@ca.ibm.com>, Aharon Chernin <achernin@soltra.com>
Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Subject: RE: [cti-stix] Applying data markings

1.       Rewording and examples (like your response for #3) will go a long way. See #5.
2.       No technical-enforcement on full path result, got it. I would like to dive deeper into the expected behavior for incorrect (malicious/invalid) path expressions with my reply in #4.
3.       Your snippet of the active/current document for the given path expressions shows everyone what should be evaluated for a given L2 scope.
4.       As for “throwing an error” as mentioned in #2, I have some additional questions around the expected behavior for the scenarios sublisted below. Should the consumer log and continue/skip or stop/break?
a.       If the given path _expression_ is invalid (incorrect syntax)
b.      If the given path goes outside the scope (returns 0 results for the given scope)
5.       Going deeper into scoping I have the following scenario. I expanded the previous indicator[0] with composite_indicator but this can be any referenced/embedded object:
{
      "stix_version": "2.0",
      "marking_definitions": [{
            "id": "us-cert:TLP-RED",
            "marking_type": "tlp",
            "tlp": "RED"
      }, {
            "id": "us-cert:TLP-GREEN",
            "marking_type": "tlp",
            "tlp": "GREEN"
      }],
      "marking_refs": ["us-cert:TLP-GREEN"],
      "indicators": [{
            "id": "indicator-1234",
            "structured_markings": [{
                  "controlled_structure": [
                        "@.title",
                        "@.composite_indicators.indicators[0].description"
                  ],
                  "marking_refs": ["us-cert:TLP-RED"]
            }],
            "title": "First indicator",
            "composite_indicators": {
                  "logical_expression": "OR",
                  "indicators": [{
                        "id": "indicator-5678",
                        "structured_markings": [{
                              "controlled_structure": [
                                    "@.title"
                              ],
                              "marking_refs": ["us-cert:TLP-RED"]
                        }],
                        "title": "First Composite indicator",
                        "description": "This is how it all began ..."
                  }]
            }
      }, {
            "id": "indicator-4321",
            "title": "Second indicator"
      }]
}
This would result in Indicator-5678 being all TLP-RED, correct?

 

6.       Would the answer/result for #5 change depending on whether the nested Indicator was referenced or embedded?

 

-Marlon
From: Wunder, John A. [mailto:jwunder@mitre.org] 
Sent: Thursday, December 10, 2015 4:00 PM
To: Taylor, Marlon; Barnum, Sean D.; Jason Keirstead; Aharon Chernin
Cc: cti-stix@lists.oasis-open.org
Subject: RE: [cti-stix] Applying data markings

 

So here’s my perspective:

 

1.       In the specification, the line is “When used on a top-level object, the root of the controlled structure specification is the object” (maybe that should be clarified to, “the root of the controlled structure specification is the top-level object”). In other words, when used within an indicator, the controlled_structure should be evaluated as if the document is JUST that indicator. Same for a TTP, etc. When used in a package, obviously the package is the root.
2.       By making the root the same as the scope, it makes it very natural to do the right thing. In order to get the right answer you need to evaluate it as if the object is the root, and thus you should not evaluate it with the whole document (you should pull that object out of the document and evaluate it). Obviously, you can write invalid JSONPath expressions, either maliciously or accidentally. These just have to throw an error.
3.       For the purposes of that evaluation, the document will look like this:

 

{
"id": "indicator-1234",
      "structured_markings":
[
{
          "controlled_structure": [
            "$.indicators[1].title"
          ],
          "marking_refs": ["us-cert:TLP-RED"]
}
]
"title": "First indicator"
}

 

Thus, evaluating that JSONPath will lead to no results, because there’s no indicators key in the document. Yes, a consumer could potentially evaluate it in the context of the document, but even for well-formed queries that will almost always lead to the wrong result. So while we’re not validating it in schema (t’s impossible to do so, either in XML or JSON) we are making it very natural and easy to do the right thing (unlike the current approach, which is asking for this type of problem).

 

John

 

From: Taylor, Marlon [mailto:Marlon.Taylor@hq.dhs.gov] 
Sent: Thursday, December 10, 2015 2:46 PM
To: Wunder, John A. <jwunder@mitre.org>; Barnum, Sean D. <sbarnum@mitre.org>; Jason Keirstead <Jason.Keirstead@ca.ibm.com>; Aharon Chernin <achernin@soltra.com>
Cc: cti-stix@lists.oasis-open.org
Subject: RE: [cti-stix] Applying data markings

 

Grabbed this example from the option 5 approach overriding-package-markings-1 example (https://github.com/johnwunder/data-markings#overriding-package-markings-1):
{
  "stix_version": "2.0",
  "marking_definitions": [
    {
      "id": "us-cert:TLP-RED",
      "marking_type": "tlp",
      "tlp": "RED"
    },
    {
      "id": "us-cert:TLP-GREEN",
      "marking_type": "tlp",
      "tlp": "GREEN"
    }
  ],
  "marking_refs": ["us-cert:TLP-GREEN"],
  "indicators": [
    {
      "id": "indicator-1234",
      "structured_markings": [
        {
          "controlled_structure": [
            "@.title"
          ],
          "marking_refs": ["us-cert:TLP-RED"]
        }
      ],
      "title": "First indicator"
    },
    {
      "id": "indicator-4321",
      "title": "Second indicator"
    }
  ]
}

 

The following questions are referencing the first indicator, indicators[0], which uses L2 markings.

 

1.       Scoping: What is the scope"controlled_structure": is in  "structured_markings": which is a valid JSON object. This is valid JSON but is this the intended scope for the L2 marking?
2.       Validation: How can we restrict/enforce the path is limited to the intended object?
3.       Exceptions: What is the expected behavior if we receive something referencing something outside of the current object?
    {
      "id": "indicator-1234",
      "structured_markings": [
        {
          "controlled_structure": [
            "$.indicators[1].title"
          ],
          "marking_refs": ["us-cert:TLP-RED"]
        }
      ],
      "title": "First indicator"
    },
    {
      "id": "indicator-4321",
      "title": "Second indicator"
   }

 

-Marlon
From: Wunder, John A. [mailto:jwunder@mitre.org] 
Sent: Thursday, December 10, 2015 10:00 AM
To: Taylor, Marlon; Barnum, Sean D.; Jason Keirstead; Aharon Chernin
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] Applying data markings

 

That’s a great point that totally slipped my mind.

 

In STIX 1.2, we said that the “marking scope” was limited to the construct itself. We didn’t say anything about the root of the document for purposes of evaluation (I just checked) so effectively it’s the root of the full package. Thus, as you say, it was possible to mark things outside the actual scope. I don’t know why we did this…whether we just didn’t think about it or whether it was because it’s (relatively) hard to pull a snippet of XML out of a document and have it live on its own (due to namespace mappings and things like that “higher” in the document that impact things at “lower” levels). So, in STIX 1.2, it’s up to the implementation to ensure that the things it marks are limited to the correct marking scope.

 

When defining this approach for STIX 1.2 I took a queue from JSON/JSONPath and considered just “re-rooting” the document. It’s very easy to do this in JSON because things higher in the document don’t impact things lower in the document, you can just pull that chunk out and it will always be valid JSON. Given that freedom in the (presumed) MTI, for this proposal I defined it differently: the actual evaluation root for the controlled structure is the object itself, and therefore when evaluating it you can/should just pull the chunk out completely and evaluate against that. That would prevent people from marking things outside the scope without requiring extra work on the implementation side.

 

If we develop an XML binding, we’ll have to figure out whether we need to override that and go back to the old approach. I would GUESS that this approach will still be better in XML even though you have to deal with namespace prefixes, but I’m not 100% sure. We could always use one for JSON and a different one for XML though.

 

That actually brings up another important point: if we assume JSON MTI is one format among several, tools will have to evaluate data markings and then track them in their repositories separate from the MTI approach, then be able to build and retransmit those markings in another format. In other words, if I take in JSON MTI I would need to evaluate that on the incoming JSON and store it in my database somehow. If I want to transmit into protobuf, I can’t just use those JSONPath markings because they’re no longer valid…they have to be redone in whatever we define for Protobuf. Just something to think about regarding the “storing STIX natively” topic.

 

John

 

From: Marlon Taylor <Marlon.Taylor@hq.dhs.gov>
Date: Thursday, December 10, 2015 at 9:33 AM
To: Sean Barnum <sbarnum@mitre.org>, "Wunder, John A." <jwunder@mitre.org>, Jason Keirstead <Jason.Keirstead@ca.ibm.com>, Aharon Chernin <achernin@soltra.com>
Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Subject: RE: [cti-stix] Applying data markings

 


Moving to go deeper on L2 markings I would like more information on the restriction/enforcement of the scope of the specific field path.

Background and Questions: The L2 path is supposed to be limited to the "current" object however path syntax allow you reference other (eg parent, sibling, non-self) objects.  Even given a regex to require the first characters reference the "current" object

a) how do we ensure the full path doesn't go outside of the current object?
b) if the full path goes outside of the current object, and what is the expected behavior?

Let's continue to narrow this discussion to conclusion.

-Marlon

 

From:cti-stix@lists.oasis-open.org on behalf of Barnum, Sean D.
Sent: Tuesday, December 08, 2015 2:33:39 PM
To: Wunder, John A.; Jason Keirstead; Aharon Chernin
Cc: cti-stix@lists.oasis-open.org
Subject: Re: [cti-stix] Applying data markings

I agree with John’s characterization here.

 

sean

 

From: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org> on behalf of John Wunder <jwunder@mitre.org>
Date: Tuesday, December 8, 2015 at 2:15 PM
To: Jason Keirstead <Jason.Keirstead@ca.ibm.com>, Aharon Chernin <achernin@soltra.com>
Cc: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Subject: Re: [cti-stix] Applying data markings

 

Nope, for any of these approaches we rely on the client software to do the right thing and preserve data markings (as well as honor them in the first place).

 

For this reason, it seems to me that as long as we define the semantics for how markings are applied at the package level we don’t need to have them exist directly on the object. We can just define that they’re semantically equivalent and allow people who are retransmitting objects to move them between the package and the object as they want (assuming of course that they preserve those semantics).

 

In other words, to me this would be a valid retransmission:

 

Org A => Org B
·         Package (TLP:RED)
o    Object 1
o    Object 2
Org B => Org C
·         Package
o    Object 1 (TLP:RED)
o    Object 2 (TLP:RED)
Yes, Org B is re-writing the object in the sense that they’re moving a marking there. But since semantically the marking was already there when it received it from Org A this seems fine to me. IMO resharers should be free to do things like this so long as they don’t change the actual content of what they’re retransmitting.

 

OTOH if this is not something the community thinks is valid then I would lean more towards what Aharon is suggesting and require that people mark the objects directly, otherwise he’s right that they’ll get “stuck” because you can’t retransmit them outside the context of the package that they came in.

 

John

 

From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Date: Tuesday, December 8, 2015 at 1:44 PM
To: Aharon Chernin <achernin@soltra.com>
Cc: "Wunder, John A." <jwunder@mitre.org>, "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Subject: Re: [cti-stix] Applying data markings

 

" If you mark at the package level there is no enforcement within the object that it’s a specific TLP. I could take that object and use it in another document and document mark that object as TLP White. 

This would work if and only if IDs are mandated to always be derrived based on hashing all of it's content, including any and all makings. 

Is that something that was decided?


-
Jason Keirstead
Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security | www.securityintelligence.com

Without data, all you are is just another person with an opinion - Unknown 


<image001.gif>
Aharon Chernin ---12/07/2015 05:26:45 PM---I ask the group this, if your markings are at the package level, how do you reuse the objects? If yo

From: Aharon Chernin <achernin@soltra.com>
To: "Wunder, John A." <jwunder@mitre.org>, "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Date: 12/07/2015 05:26 PM
Subject: Re: [cti-stix] Applying data markings
Sent by: <cti-stix@lists.oasis-open.org>





I ask the group this, if your markings are at the package level, how do you reuse the objects? If you mark at the package level there is no enforcement within the object that it’s a specific TLP. I could take that object and use it in another document and document mark that object as TLP White. 

If we require markings at the object level, then you are protected by object revisioning rules, and potentially ID hashing. You could never change the TLP of the object and reuse that object without breaking the revisioning rules or the ID hash.

I am not going to put up a huge fight here, but it just seems so obvious to me.

Aharon

From: <cti-stix@lists.oasis-open.org> on behalf of "Wunder, John A." <jwunder@mitre.org>
Date: 
Monday, December 7, 2015 at 3:42 PM
To: 
"
cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Subject: 
Re: [cti-stix] Applying data markings


All,

I updated the proposal based on the comments from last week: https://github.com/johnwunder/data-markings.

A few comments (on the comments…):

- Mark suggested renaming “Level 1” to “Data Markings” and “Level 2” to “Field Level Data Markings”. I initially made that change but found the language got very confusing so I switched it back.

- Per Aharon’s comments, this proposal does have markings at the package level for both L1 and L2. This allows you to mark everything in a package as TLP:RED in a single statement (not duplicated across each indicator). OTOH it also means that if you discard the package you would need to have some way of tracking the markings separately. IMO that’s totally fine…it’s metadata about an object so it’s not like you’re changing the object itself if you add them to the object, and in any case per Eric’s earlier e-mail I would imagine that after ingest your tool would probably be dealing with markings in something outside of STIX anyway (I certainly would, in particular for L2).

- Aharon commented that we were getting rid of XPath and that’s an advantage of this proposal. That’s true……...kind of. Replacing XML with JSON means we use JSONPath instead of XPath, but fundamentally it’s still a controlled structure based approach with all the implementation complexity that it brings (as Bryan Worrell can tell you).

- That said, JSON is more straightforward than XML and so I think using it will minimize the occurrence of bugs like the ones we’ve had in the past with misunderstanding XPath.

To summarize things, here’s the open questions that I can think of (beyond “is this proposal good”):

1. Should the ability to handle Level 1 markings be MTI (mandatory to implement)? (
2. Should we spend further time investigating better approaches for L2 markings (Option 3, 4, others) or just go with this (Option 5)?
3. Should we keep markings at the package level or move all markings to the individual top-level objects?

My answers:

1. Not sure to be honest. Leaning towards not MTI.
2. We should use L2 as-is…it’s hard, but marking fields is hard and so IMO that’s fine. Most people will use L1 anyway.
3. Keep markings at the package level. This allows you to represent a “default” marking and override it. I also *think* (maybe I’m wrong) that certain entities in USG require package-level markings to represent “highest-classification”, which means other gov’t may as well.

John
On Dec 7, 2015, at 10:43 AM, Cory Casanave <cory-c@modeldriven.com> wrote:

This does look like a good approach for message level markings. 
Any information exchange, except for open data, is done under some explicit or implicit agreement between the parties. Various forms of such “exchange agreements” have been around for years. For example, the “Collaborative Partner Profile Agreement” in EbXML, which came out of IBM. Having a reference to an exchange agreement provides the basis for trust between specific parties as well as an explicit representation of any categorization and/or restrictions on any exchange under that agreement. For example, asserting that telephone numbers are personally identifiable information and that personally identifiable information shall not be stored. Markings in the “instance” document augment the exchange agreement. It would simply not be practical or trustworthy to expect all such “markings” to be in every instance, so reference to an exchange agreement provides that level of trust and a place to put “generic” restrictions and categorizations. I suggest exchange agreements be considered for CTI. The exchange agreement reference can be a kind of marking.
-Cory

From:cti-stix@lists.oasis-open.org [mailto:cti-stix@lists.oasis-open.org] On Behalf Of Barnum, Sean D.
Sent:
 Monday, December 07, 2015 9:49 AM
To:
 Wunder, John A.; 
cti-stix@lists.oasis-open.org
Subject:
 Re: [cti-stix] Applying data markings

I fully support this option as would be expected.
Kudos to John for such a great writeup explaining the idea.

sean

From: "cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org> on behalf of John Wunder <jwunder@mitre.org>
Date: 
Thursday, December 3, 2015 at 3:11 PM
To: 
"
cti-stix@lists.oasis-open.org" <cti-stix@lists.oasis-open.org>
Subject: 
[cti-stix] Applying data markings


All, 

I developed this proposal to handle the application of data markings in STIX 2.0: https://github.com/johnwunder/data-markings. Note: it doesn't address the format of the markings themselves (improvements to TLP, the work in FIRST, etc), just how those markings get applied to content.

I this this meets the need for simplicity for object-level markings as we’ve talked about many times while still allowing for more complicated field-level markings for those that need them. Please review the proposal and let’s talk about feedback. If this looks good to everyone we could use it as the solution for issue #231 (currently #2 on our roadmap).

John

Attachment: smime.p7s
Description: S/MIME cryptographic signature



[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]