> I think the thing that needs to be clear here is there are far more use cases for TAXII than simply querying against a large repository of data. Statements like "The messages that it delivers are ephemeral and are subject to be deleted at the discretion of the TAXII server operator" assume that the TAXII server is acting as such a repository
My statement is actually supposed to mean the opposite of what you said. By saying that the messages are ephemeral then I am saying there is likely a very small repository that only holds messages for a very short duration (perhaps only seconds). The purpose of the TAXII server is to hold the messages for the shortest duration until it believes all clients should have requested the STIX objects. The flow of events is:
1. TAXII client sends POST message to send STIX object for distribution
2. TAXII server receives POST message and temporarily stores the object in some sort of repository/database
3. TAXII clients sends GET message to list all objects
4. TAXII server receives GET message and returns a list of all objects in repository/database
5. TAXII client issues GET message to download STIX object
6. TAXII server receives GET request, looks up object in repository/database, and sends to client
7. TAXII server timer expires for POSTed message and STIX object is removed
When can I no longer retrieve STIX objects POSTed to a TAXII server? How are they removed from the TAXII server? Does a timer like the one above exist is the question I have? It sounds like yes based on the discussions here, but I would like to have confirmation of that because then it would answer my question of when is a STIX object deleted from a TAXII server?
> An NGFW device, or a proxy server, or an endpoint agent, is highly unlikely to maintain any kind of historical database of threat intelligence - some of these may not store any data at all - but it can indeed act as a TAXII server endpoint for many other reasons.
Could you further explain this? I don’t understand how something can be a TAXII server but not have a historical database. I feel that it must have some repository/database of information to store the POSTed objects temporarily until a TAXII client does a GET request. Otherwise where do you get the STIX objects that are used in steps 4 and 6 above? A proxy I get, but that is really just a middle-man between a real TAXII server.
-Nate
From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Date: Thursday, May 4, 2017 at 11:32 AM
To: "Reller, Nathan S." <Nathan.Reller@jhuapl.edu>
Cc: Terry MacDonald <terry.macdonald@cosive.com>, Allan Thomson <athomson@lookingglasscyber.com>, Bret Jordan <Bret_Jordan@symantec.com>, "Struse, Richard" <Richard.Struse@hq.dhs.gov>, John-Mark Gurney <jmg@newcontext.com>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] Re: [EXT] [cti] Embedded Relationships
I think the thing that needs to be clear here is there are far more use cases for TAXII than simply querying against a large repository of data. Statements like "The messages that it delivers are ephemeral and are subject to be deleted at the discretion of the TAXII server operator" assume that the TAXII server is acting as such a repository, which true. Many of the consumers and producers of threat intelligence will be operating under different personas than that, acting as message brokers, live sources of data, and real-time consumers of data to act on.
An NGFW device, or a proxy server, or an endpoint agent, is highly unlikely to maintain any kind of historical database of threat intelligence - some of these may not store any data at all - but it can indeed act as a TAXII server endpoint for many other reasons.
-
Jason Keirstead
STSM, Product Architect, Security Intelligence, IBM Security Systems
www.ibm.com/security
Without data, all you are is just another person with an opinion - Unknown
From: "Reller, Nathan S." <Nathan.Reller@jhuapl.edu>
To: Terry MacDonald <terry.macdonald@cosive.com>, Allan Thomson <athomson@lookingglasscyber.com>
Cc: Jason Keirstead/CanEast/IBM@IBMCA, Bret Jordan <Bret_Jordan@symantec.com>, "Struse, Richard" <Richard.Struse@hq.dhs.gov>, John-Mark Gurney <jmg@newcontext.com>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Date: 05/03/2017 05:29 PM
Subject: Re: [cti] Re: [EXT] [cti] Embedded Relationships
Sent by: <cti@lists.oasis-open.org>
> A TAXII community is not a database.
I think this is the biggest misconception that I have had since reading the specs. I’m new to the group and STIX/TAXII, and my first task was to review the specs from a software engineer’s perspective. From my perspective of having just read the specs I think (or I did think before having many, many conversations) that the TAXII server is a REST service around a database. When someone tells me they have a REST service (which is what the TAXII document says) then I instantly think they have an HTTP server running with a RESTful architecture plugin that is accessing a database to manage the collections and objects in the collections. That framework will almost always have GET calls to retrieve object, POST to store objects, and DELETE to remove them. They may also likely support PUT to update them.
I think the TAXII document should state that a TAXII server is a “messaging framework” that uses POST to send messages and subscribers must use GET to receive messages. The messages that it delivers are ephemeral and are subject to be deleted at the discretion of the TAXII server operator. The TAXII server may hold messages for a day or maybe just an hour. This means that if you query for all objects at time t0 and again at time t1 where t1 != t0 then a completely different set of objects may be returned. It might also be helpful to state why a REST-service-like architecture was made and why a messaging queue like AMPQ, ZeroMQ, RabbitMQ, etc was not used.
Are those true statements?
-Nate
From: Terry MacDonald <terry.macdonald@cosive.com>
Date: Wednesday, May 3, 2017 at 3:51 PM
To: Allan Thomson <athomson@lookingglasscyber.com>
Cc: Jason Keirstead <Jason.Keirstead@ca.ibm.com>, Bret Jordan <Bret_Jordan@symantec.com>, "Reller, Nathan S." <Nathan.Reller@jhuapl.edu>, "Struse, Richard" <Richard.Struse@hq.dhs.gov>, John-Mark Gurney <jmg@newcontext.com>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
Subject: Re: [cti] Re: [EXT] [cti] Embedded Relationships
I think it's important to note here that a TAXII community doesn't necessarily agree on a common set of 'agreed truth'. It's always up to each consumer to determine what they want to believe. Threat intelligence sharing is just someone sharing their assertion to other members in the community.
A TAXII community is not a database. It's a sharing group. Once a producer has shared data with the community they have effectively lost control of the data ... And instead trust that other members of the year Intel sharing community will treat the data in the way they requested through the data marking that was applied to the assertions they made. A delete function doesn't make sense to me in this construct - at least at a 'community' level. Once data has been shared, it's been shared. The most we can communicate to recipients after that is that 'the data is worthless, so don't use it'.
At the moment TAXII operates through a single server for a community, but this is not the long term goal. Our long term goal is to spread the TAXII community over multiple TAXII servers in multiple different locations (newsgroup style), such that a single TAXII server can be part of multiple communities.
Cheers
Terry MacDonald
Cosive
On 4/05/2017 05:03, "Allan Thomson" <athomson@lookingglasscyber.com> wrote:
It would be better to just have timestamp when the relationship no longer is ‘active’.
That fits better with other immutable data processing for TI in general.
allan
From: "cti@lists.oasis-open.org" <cti@lists.oasis-open.org> on behalf of Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Date: Wednesday, May 3, 2017 at 9:19 AM
To: Bret Jordan <Bret_Jordan@symantec.com>
Cc: "Reller, Nathan S." <Nathan.Reller@jhuapl.edu>, John-Mark Gurney <jmg@newcontext.com>, "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>, "Struse, Richard" <Richard.Struse@hq.dhs.gov>
Subject: Re: [cti] Re: [EXT] [cti] Embedded Relationships
DELETE assumes the data is being stored in a repository. It would have to be optional to implement...
--
Sent from my mobile device, please excuse any typos.
Bret Jordan --- Re: [cti] Re: [EXT] [cti] Embedded Relationships ---
Nate,
Great questions. In regards to the DELETE function, I think we can easily add this in TAXII 2.1. Please help us identify other gaps.
One of the gaps you mentioned is how to get the rest of the objects. We have talked about adding an ability in TAXII to automatically deterrence embedded relationships. Maybe this would help.
As for the identity aspect, I think your internal tools inside your trust group can do what they need. You can always use custom properties to track an object as it moves through your work flow and object lifecycle. One thing we need to add to STIX and TAXII is the ability to send suggested updates. I think this would also help you.
Please keep sending questions and ideas and help us fill in these gaps for the next release.
Bret
Sent from my Commodore 64
PGP Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
On May 1, 2017, at 7:59 PM, Reller, Nathan S. <Nathan.Reller@jhuapl.edu> wrote:
It assumed that whom ever you got the relationship will know how to get the other objects referenced by the relationship...
Why is that assumed? Shouldn’t the STIX/TAXII specs say how to do that? If it’s not consistent on how to do that then that is bad for consumers, and it seems like the point of the specs is to make it consistent for how to share information.
there isn't a delete operation in STIX
Why is there not a delete operation? How can the objects live in perpetuity?
In this case, they may choose to use a single "identity" for this group to produce their objects, and anyone in the group is allowed to use it.
Now there are no current technical measures to prevent you from doing this.
There are a couple of challenges with this approach. I’m curious how you plan to solve them.
1. Identity management
I have a lot of potential identities. Every combination of collaboration must be supported. That is exponential growth in the number of groups. While the number of groups is concerning the real heartburn will be when a user needs to select one. They will need to choose from a large set of groups.
For instance, consider the Navy UARC community. There is APL, PSU, UT, UW, and UH. As time goes I expect every possible combination of collaboration to develop. That means my “identity” for creating an object could be [(APL), (APL, PSU), (APL, UT), (APL, UW), (APL, UH), (APL, PSU, UT), (APL, PSU, UW), (APL, PSU, UH), … That’s kind of annoying for users to scroll through the long list to select.
2. Group selection
How do you know which group should be used? In the use case I gave the DHS analyst creating the first event has no idea who will contribute to the group. How would he know that only APL and FBI would contribute? He cannot know that and perhaps tomorrow another organization will want to contribute. Perhaps only DHS will contribute to the object.
I see a couple of options. He can either use an “everybody” group that allows anyone to contribute, or he can create a new group for each new SDO. The “everybody” group allows for anyone to contribute to the object. That is good, but all edits to the object will be attributed to “everyone” in which case you don’t know who made the object. That’s not good.
The other option is to create a new group for each new object. This would allow the group to add and expand members as more requests to contribute are received. The downside is that I now have a new group for each object, and the Identity must be further parsed so I can determine who is in the group. Group names become meaningless. The other downside is that I still don’t know who made what revision of the object. All I know is that someone in the group produced the last version.
Currently, they would need to contact the DHS to update their object
This is the alternate to the single identity approach. The biggest downside to this approach is object attribution. If I create an object then I would be hesitant to update it based upon a request from someone else unless I have a very close relationship with them. Going back to my example with DHS, FBI, and APL let’s assume FBI sends a request to DHS to change the object. Would DHS likely just accept the request? Probably not because anything in that object is attributed to them, and what are the chances that the DHS person that created the object knows and trusts the FBI agent making the request? I would be nervous of bad information being attributed to me. And APL being a consumer may want to know who added which parts to the object.
-Nate