OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

cti-taxii message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: RE: [cti-taxii] HTTP REST API for TAXII 2.0


I had a quick conversation offline w/ John, and I realized there is a terminology collision on HTTP pipelining. HTTP pipelining is the technique of sending multiple HTTP requests on a single TCP connection [1]. When I used the term pipelining, I meant multiple messages in a single HTTP request or response. Facebook’s ThreatExchange uses the term batching [2], so maybe it’s worth just using that term?

 

> allowing for multiple content blocks in the POST would be another (simpler to implement) way to do that.

John and I were both talking about this concept, just using different terms.

 

I like the EventSource concept – I see it as an alternative to HTTP Long Polling.

 

I also like exploring the idea of getting rid of the group concept, as long as we can keep all the functionality we need.

 

Thank you.

-Mark

 

[1] https://en.wikipedia.org/wiki/HTTP_pipelining

[2] https://developers.facebook.com/docs/threat-exchange/best-practices/v2.4

 

From: cti-taxii@lists.oasis-open.org [mailto:cti-taxii@lists.oasis-open.org] On Behalf Of Wunder, John A.
Sent: Tuesday, September 22, 2015 8:53 AM
To: cti-taxii@lists.oasis-open.org
Subject: Re: [cti-taxii] HTTP REST API for TAXII 2.0

 

To pull the thread on a couple of those:

 

Groups

I can see the use case you’re aiming for, but another option to consider here is not implementing groups in TAXII and just making it clear that you can host multiple TAXII “channel” endpoints. You would lose the ability of users of the TAXII server to stand up new groups via the standard API, but as a trade-off you wouldn’t have to deal with the groups concept at all in TAXII itself.

 

Just throwing it out there as an idea.

 

Permissions

It seems like if you support setting permissions on a channel or group, then you should support modifying them?

 

Pipelining

HTTP pipelining is one solution to the connection overhead problem, but just allowing for multiple content blocks in the POST would be another (simpler to implement) way to do that.

 

Implementability

I still think this sounds implementable, the two things that seem like the long poles are HTTP pipelining and the EventSource capability (assume this is roughly what you’re talking about, minus the specific _javascript_ API: https://developer.mozilla.org/en-US/docs/Web/API/EventSource).

 

TTL

Makes sense.

 

John

 

On Sep 22, 2015, at 7:55 AM, Davidson II, Mark S <mdavidson@mitre.org> wrote:

 

Here are my thoughts on the questions (only responding where I have something to add):

 

Bret explained the current reasoning for groups. If there’s a better way to design/implement the concept, or if there are open questions, please raise them!

 

+1 on Message TTL – I think message expiration is an important aspect of system health.

 

RE: Content Types. I see a channel as potentially having multiple message types permitted on it (e.g., both an Indicator object and a Sighting object). If we’re going native HTTP, I feel like we should strongly consider only using HTTP’s built in Content-Type header (MIME types could be something like application/stix-2.0-json+sighting). For multiple messages in a single HTTP response, maybe that’s a MIME multipart response?

 

I like the idea of being able to POST multiple messages at once. This also represents something where a client could start simple/naive (e.g., send one message per POST) then as the client matures, implement advanced features (send multiple messages per POST).

 

RE: Permissions. It’s an open question for me whether there should be TAXII-level functionality for modifying group/channel permissions, or if that’s an “exercise left to the implementer” and TAXII only defines the error codes for e.g., permission denied.

 

+1 that indicator TTL (which is separate than message TTL) should not be in TAXII; I see that as specific to the payload.

 

I’ll send a separate email with the currently envisioned Channel concept.

 

Thank you.

-Mark

 

From: cti-taxii@lists.oasis-open.org [mailto:cti-taxii@lists.oasis-open.org] On Behalf Of Jordan, Bret
Sent: Monday, September 21, 2015 5:58 PM
To: Wunder, John A. <jwunder@mitre.org>
Cc: cti-taxii@lists.oasis-open.org
Subject: Re: [cti-taxii] HTTP REST API for TAXII 2.0

 

That for the questions.   Comments in-line...

 

Thanks,

 

Bret

 

 

 

Bret Jordan CISSP

Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

 

On Sep 21, 2015, at 13:39, Wunder, John A. <jwunder@mitre.org> wrote:

 

OK you got me. Here are my thoughts:

 

What additional information do you need to understand the design?

- Is there some explanation for what the group concept is supposed to represent? I guess I get it conceptually, but it’s not clear to me what use case it’s solving vs. just running separate instances. In other words, why do groups need to be defined by TAXII?

 

Bret: This requirement was brought up by Terry and others as a way of improving separation between trust groups running the same channels on one system.  The general idea is that various working trust groups or groups of interest will all have an indicator channel.  And the logical way of making that separation seemed to be using a group structure.  Each TAXII server will be default have at least one group, a default group.  They can, if they choose, have additional groups for different things.  

 

- Do you plan to prescribe caching strategies or will it be up to individual servers to implement that?

 

Bret: That is a good question.  What we have talked about is having a TTL value for a message channel and if a client has not pulled from the channel within that time, the message will disappear.  So for very low volume systems, that might be in the order of days or weeks.  In super high throughput system, it maybe in the matter of 10s of minutes.  We were thinking that the default value might be 24 hours.  But that is always open to discussion.  

 




For your use case, does it contain anything really bad/unworkable?

- Should the content type be set at the channel level or the message level? It seems like you might want to have several different types of content on the same channel. If you set it at the message level, this field can go away.

 

Bret: I do not know the answer to this question yet.  It seems to be something that will get decided later.  My gut response is that it would be part of the message.  But I am fully open to people telling me that I am wrong and that it should be done on the channel.  

 




- This doesn’t seem targeted at a request/response use case. That’s not really a problem, doing both that and pub/sub in the same spec would get complicated, but I figured it was worth calling out.

 

Bret: In general it is a pub/sub model.  The planned implementation appears to be HTTP with Long Polling of a REST API. 




- Are you planning to support update and destroy (PUT and DELETE)? Obviously renaming channels would be complicated and have weird side effects given the name is in the URL, but changing permissions for existing channels is probably a requirement.

 

Bret: I think some operations might have a PUT or DELETE feature.  I can see the creation and tear down of channels being one use-case.  I can see the creation and tear down of groups being another.  I can also see RBAC for a group and the transfer of ownership or the addition of an admin on a group.  Some of these things are really server implementation specific and some are things that would go in to the spec.  Not sure where that tear line is yet, and I am hoping the community will let us know where that should be.  I am also going to defer to our IBM folks and others on how best to build a RESTful API.  Lets find the people in our community that build these all day every day, and have them tell us what it should look and feel like. 




- The messages POST request seems to say you’ll create one message at a time, but the GET request lets you get multiple messages at once. That seems inconsistent. Maybe a POST should let you create multiple? That would cut down on the number of individual HTTP requests required (assuming clients queue and send)

 

Bret: I do not think there is any limitation right now that says you can not do pipelining on creation.  I think we could easily do that.  I know we have already talked about doing pipelining on GETs.  The idea would be that a client could tell the server that it can consume up to a certain size of data in one blob.  And the server will try its best to honor that.  The one problem use-case is if the client says it only wants 1K of data at a time and the next message is 2M in size.  The current thinking is that the server will NOT fragment but rather send one single message.  So when the client tell the server how much it can take, it is a guide not a hard and fast rule.  Now there may be use cases where we need to look in to that.  But that is a problem we have today and is not something new to this pub/sub model.  So it might be something for TAXII 2.1.  If anyone has any ideas of how this should work, please bring them up. 

 




If you’re an implementor, could you implement it?

Yeah, as far as I can tell.

 

Bret: That is good to know.  We want to be very cognizant of the people that have to implement this.  

 




 

Other important questions to ask

- Did you consider having the server auto-assign identifiers for channels and groups? I haven’t thought through it a ton and don’t have any problems with names but it’s an option.

 

Bret: If you do auth assign or auto generation of channel names then you have to provide a way for discovery to take plus for 100% of the use-cases.  If you have some well known names, then it is very simple to get a base level system up and running.  I think the general idea we are holding fast to, is TAXII 2.0 should be something that just works.  Organizations should not have to twiddle with to just get it working.  If it is in a network, other networking vendors should be able to just use it.

 




- Did you consider message-level permissions?

 

Bret: another great question.  We have started discussion permissions and policy stuff.  But a lot of that will be impacted on these other decisions.  So we understand the conceptual level, but we are waiting to see if the community is okay with REST before we move much further up the application stack.  IMO, and only my opinion, the TAXII spec will allow for something like a message handler.  That message handler may be only on the ingress point of a channel, or may also be on the egress point.  It could do lot of things we permission checking and possible filtering or rewrite rules, etc.   Mark has a list of verbs that the message handler should support to start the discussion.  And maybe that should be the discussion post REST that we start to talk about. 

 




- How about message level TTLs? I might want some indicators to live longer than others, for example.

 

Bret: I think that kind of TTL really belongs in STIX.  But push back if you think I am wrong.  The TTLs on the message channel, IMO, deal with how long they sit not the channel before they are consumed.  Now, you could easily create a long-lived Indicator channel, where you publish certain things that are available for a longer period of time.  That could be interesting for certain use cases.

 

Once again, thanks for the question, really great questions.  To everyone else, please speak up, please review the REST document that is out there so far and give us your feedback.  

 

 

Bret

 




John

 

On Sep 21, 2015, at 3:00 PM, Davidson II, Mark S <mdavidson@mitre.org> wrote:

 

All,

 

Since it worked well last time, I’ll threaten to interpret your silence as overwhelming agreement that the proposed REST API looks pretty awesome.

 

As you review the REST API, please think about the following:

 

·         What additional information do you need to understand the design?

o   If you have no context for what it is or why it’s being proposed, ask some questions! You can ask them to Bret and I directly and we’ll anonymously post them.

·         For your use case, does it contain anything really bad/unworkable?

·         If you’re an implementer – could you implement it?

·         Are there other important questions to ask?

 

Thank you.

-Mark

 

From: cti-taxii@lists.oasis-open.org [mailto:cti-taxii@lists.oasis-open.org] On Behalf Of Jordan, Bret
Sent: Thursday, September 17, 2015 2:16 PM
To: cti-taxii@lists.oasis-open.org
Subject: [cti-taxii] HTTP REST API for TAXII 2.0

 

All,

 

Given the strawman MTI discussion for using HTTP, the slack channel over the past few weeks has had a very lively discussion about what it would look like if we implemented a RESTful API for TAXII 2.0 on top of HTTP.  As this discussion has started to come to a steady state, we feel that it is important to move it out of Slack and in to the formal email discussion space to see if it holds water and hopeful flesh out further ideas.  

 

If we were to use a RESTful API over HTTP for TAXII 2.0, this would not represent a radical change form TAXII 1.1, but rather, IMO, a natural evolution of the TAXII 1.1 services concept.  I also believe this will enable us to build a foundation from what future versions of TAXII, beyond 2.0, can iterate and add functionality. 

 

The current mock up on the HTTP REST API can be found here:

  

 

NOTE: remember anyone that wants to join our Slack channel, please let us know, just be mindful that it can be very high volume.  

 

 

 

Thanks,

 

Bret

 

 

 

Bret Jordan CISSP

Director of Security Architecture and Standards | Office of the CTO

Blue Coat Systems

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050

"Without cryptography vihv vivc ce xhrnrw, however, the only thing that can not be unscrambled is an egg." 

 

 



[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]