OASIS Mailing List ArchivesView the OASIS mailing list archive below
or browse/search using MarkMail.

 


Help: OASIS Mailing Lists Help | MarkMail Help

cti-taxii message

[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]


Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search


It really feels like the information-request and the Observed Data search are totally different thing than normal RESTful queries that allow pivoting.  

While the decision is not mine, I generally prefer simple things being simple. 

In the long-term view I would see us having a set of RESTful endpoints that do really basic queries. I would also see us having actual search endpoints for more complicated stuff later on that some implementations will not support.  Meaning, the search end points will be OPTIONAL to implement. 

Examples of what we have today

{api-root}/collections/{id}/objects/?match[{key}]={value} 
{api-root}/collections/{id}/objects/versions/

Examples of what we could add

{api-root}/collections/{id}/relationships/related/
{api-root}/collections/{id}/search/
{api-root}/search/

The last two would be for the more detailed query object support, like what Jason and Terry have proposed.  But that is going to take us a lot longer to figure out and get working.  A simple RESTful path would be much easier to do in the short term, I think.  If I am wrong, and the majority of the TC thinks we should do the other way, please correct me. 

It seems like the processing of an arbitrary query object is a LOT more work, than parsing a simple string value off of a URL parameter. If we do not want to do:

{api-root}/collections/{id}/relationships/related/

We could always just do:
{api-root}/collections/{id}/objects/?match[type]=relationship&match[related]={stixid}

Bret




From: Matt Pladna <mpladna@lookingglasscyber.com>
Sent: Monday, November 26, 2018 10:28:28 AM
To: Bret Jordan; Jason Keirstead
Cc: Gary Jay Katz; cti-taxii@lists.oasis-open.org
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
 

Would it be tenable to have a hybrid approach where we take the proposal here https://docs.google.com/document/d/1Cy_9Bh5tKEkDHGg2iv5c3AwriqVr7ygbKXWOv4-uHxs/edit#heading=h.1jcqb6vc5y7z and scope it down to only handle relationship (specifically related-to) queries?

 

Meaning the google doc will have one information-request type, which will be related-to query requests.  It can still be extended later with more types.  Rather than return a specific response type (information-response) what if it just returned the STIX content as well (in a bundle? Stream?)?  If a TAXII type of information-response is necessary later, we can add it and support STIX 2 for backward compatibility.

 

I think having a type dedicated to taking a query request is a good idea.  JSON query/information request objects can be extended in the future and I think will give us more options than relying on endpoints and parameters in the URL.

 

If an endpoint of ‘<api-root>/collections/<id>/information-request/’ is too generic, we could make the types explicit in the URL:

<api-root>/collections/<id>/information-request/related-to’

 

I’m not sure which is better, but does this help reduce scope to something we can do soon AND leave us in a place to grow?

 

From: <cti-taxii@lists.oasis-open.org> on behalf of Bret Jordan <Bret_Jordan@symantec.com>
Date: Tuesday, November 20, 2018 at 09:02
To: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc: Gary Jay Katz <gary.katz@FireEye.com>, "cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search

 

Jason,

 

If you remember we had a global search in some of the early drafts of 2.0.  But it was pulled because several people did not like it.  I think any kind of search like that would need to be done at both the collection level and at the api-root level, where the api-root level is optional to implement.  

 

All,

 

So far we have only heard from 6 people in the TC so far.  I would like to hear what everyone else thinks.  

 

I plan on talking about this on next weeks working call, just FYI.

 

Bret 

Sent from my Commodore 64 

 

PGP Fingerprint: 63B4 FC53 680A 6B7D 1447  F2C0 74F8 ACAE 7415 0050


On Nov 20, 2018, at 6:25 AM, Jason Keirstead <Jason.Keirstead@ca.ibm.com> wrote:

I have no issue with the concept of getting an MVP, out so long as the future is considered so we don't box ourselves into a corner. I have already given very specific future-facing examples of problems this MVP introduces, that need to be thought through. We have already been down this path, with several breaking changes between TAXII 2.0 and TAXII 2.1, and parties coming to the fore saying they are "waiting" for 2.1 so they don't have to re-do implementations.

If TAXII 2.2 introduces more breaking changes because we did not consider all the use cases and tried to rush things out, then we are going to rapidly start to loose faith in the marketplace.

I'll also again re-iterate that it is very much debatable even if this is the actual MVP we need at this point in time. It depends highly on where in the intelligence ecosystem you sit. I will take a FireEye product as an example - if I have a file hash and during a hunt I want to query Helix using it's API for all of the sightings of that file hash, this MVP does not help me. We have no way to do this extremely basic use case in STIX today. I think this global query use case is *more* critical than being able to de-reference a relationship only inside a single collection... collections after all are often mirrored wholesale.

RE query - this is the exact use case that the STIX Extension process is meant to solve, by forcing folks down a path where proposals are implemented and tested and vetted in the wild before they make it into the standard. I think that anything we do along the lines of Query should have to be proven in the market before it makes it into the spec along with the potential for future breakages.

-
Jason Keirstead
Lead Architect - IBM Security Connect
www.ibm.com/security

"Things may come to those who wait, but only the things left by those who hustle." - Unknown




From:        Gary Jay Katz <gary.katz@FireEye.com>
To:        Bret Jordan <Bret_Jordan@symantec.com>, Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc:        "cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Date:        11/19/2018 07:14 PM
Subject:        Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Sent by:        <cti-taxii@lists.oasis-open.org>





Jason,
    That’s a good question.  One which I can provide my response, but I believe requires a larger discussion, which is part of the reason why I believe we should get an MVP out now, rather than waiting.  To Bret’s point, I’d rather not wait 6-12 months to get a release out.
 
     Currently we have ways to easily search for a STIX object.  It does not allow for STIX querying or the complex use cases we have identified, but it does allow someone to easily retrieve an SDO.  And that is a good thing.  There should be an easy way to perform a search, whether that is for an object or a relationship and it should be easy for a consumer to use that functionality to traverse the graph and easy for a producer to implement such functionality.  Not every producer will have a backend database (or the requirements) to allow for complex querying.  I truthfully do not see the issue with allowing restful methods to easily traverse the graph and then later add in full query support where you could accomplish the same thing in addition to much more powerful queries.  People may disagree, and we can and should debate this as TAXII query is designed, but I personally do not believe that the full TAXII query support needs to follow the same pattern as is defined for retrieving an SDO or SRO.  
 
Just my opinion, but hope it helps explain where my thoughts are.
     -Gary
 
 
From: Bret Jordan <Bret_Jordan@symantec.com>
Date:
Monday, November 19, 2018 at 4:41 PM
To:
Jason Keirstead <Jason.Keirstead@ca.ibm.com>, Gary Katz <gary.katz@FireEye.com>
Cc:
"cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Subject:
Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search

 

This provides a solution that we can have now. At the current rate any more advanced solution is at least 6-12 more months away.  Yes, we may need to do things differently in the future, but that is okay.  We can always deprecate features as needed. But I figured a more RESTful design would get us up and going sooner rather than later.

 

By my count the following individuals support the proposal I have submitted:

Myself

Drew

Nicholas

Sean

Gary

 

The individuals that do not support it:

Jason

 

I am curious to hear what others have to say. Can everyone please chime in on this debate.

 

Bret



From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent:
Monday, November 19, 2018 2:25:37 PM
To:
Gary Jay Katz
Cc:
Bret Jordan; cti-taxii@lists.oasis-open.org
Subject:
Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search

 
Gary - how would you propose in the future we can extend or expand this approach, without redoing it from scratch?

This is the problem with having query-specific endpoints or query parameters such as "related to" - they can't be extended or added to in any logical way.

It will result in either no way to actually do combinations of parameters, or an unworkable mishmash of normative text quantifying all of the edge cases of which endpoints can be combined with which others to get what behaviour.

Sent from IBM Verse

Gary Jay Katz --- Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search ---
 

From:

"Gary Jay Katz" <gary.katz@FireEye.com>

To:

"Jason Keirstead" <Jason.Keirstead@ca.ibm.com>, "Bret Jordan" <Bret_Jordan@symantec.com>

Cc:

cti-taxii@lists.oasis-open.org

Date:

Mon, Nov 19, 2018 4:55 PM

Subject:

Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search

 



 
Jason,
    I believe you are correct, there are a number of complex use cases that this would not meet.  For those that remember back to some of the early face-to-face meetings, I pushed for trying to find a solution that met all of these solutions in the first release.  I now believe that this needs to be taken in parts.  There may be certain TAXII implementations that only need to and are able to support basic search and pivoting, while others will be built to perform complex querying such as what you are suggesting.
 
     Having an ability to easily query relationships and pivot off of them which does not support the more complex use cases, does not mean that the capability needs to be rewritten or that those complex use cases are invalid.  We just need ways to do the simple things simply, while eventually put in place the ability to do the complex capabilities.  
 
    Currently we have no ability to query relationships or pivot.  This is a blocker for anyone trying to provide the baseline capabilities to support the STIX model using TAXII.  I would suggest we get the baseline capabilities in place to allow releasing a minimal viable product in the near-term while continuing to work towards more advanced query capabilities in a future release.
 
-Gary
 
From: <cti-taxii@lists.oasis-open.org> on behalf of Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Date:
Monday, November 19, 2018 at 3:38 PM
To:
Bret Jordan <Bret_Jordan@symantec.com>
Cc:
"cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Subject:
[cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search

 
The problem with this approach of "getting something done in a a few days", is it is highly likely it will have to be re-done in the future because it isn't considering the other use cases.

For example - In the future how can I find all sightings related to Object Foo that match the indicator 1.2.3.4, with your proposal? Seeing how any future "match the indicator" portion would be an entirely different endpoint than the "related-to" endpoint, I now can't combine them - and therefore would have to make multiple - perhaps dozens, hundreds, or even thousands of round-trips to the server to collect this information.


-
Jason Keirstead
Lead Architect - IBM Security Connect

www.ibm.com/security

"Things may come to those who wait, but only the things left by those who hustle." - Unknown





From:        
Bret Jordan <Bret_Jordan@symantec.com>
To:        
Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc:        
"cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Date:        
11/19/2018 04:27 PM
Subject:        
Re: [EXT] Re: [cti-taxii] TAXII Query / Search






Jason, the proposal you have submitted is valid, and one potential solution.  However, it represents a fair bit of complexity on the specification and on implementations. There are also a lot of unanswered questions about how it would actually be done. To support this, we would also need to create several conformance levels for it, which is non-trivial.


Per your own statements, over the past 12 months I have seen very little to no public support for it. As such, I am trying to find a simpler solution that we can do now and then look at the more complicated solution later.  I fear if we try to do a more complicated solution now, then TAXII 2.1 will still be many months away from being done.


The proposal I have worked on with Drew and Gary is something we can do very easily in a matter of days, which could enable us to finish TAXII 2.1 within the next month.


But all of this is up to the TC to decide.  We can see what kind of feedback we get, and see where the majority of TC members sit. But at this stage, I would personally (not as a chair or editor) prefer a more simple solution that can be used now.  


Bret




From:
Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent:
Monday, November 19, 2018 1:06:11 PM
To:
Bret Jordan
Cc:
cti-taxii@lists.oasis-open.org
Subject:
[EXT] Re: [cti-taxii] TAXII Query / Search


Hi Brett - Terry and I have an alternative proposal we've been asking the TC to have a look at here


https://docs.google.com/document/d/1Cy_9Bh5tKEkDHGg2iv5c3AwriqVr7ygbKXWOv4-uHxs/edit#heading=h.1jcqb6vc5y7z

As we lay out in the document, there are a number of issues TAXII has with query, and I think the proposal you have below is insufficient to solve the use cases we have in Github:

We first started working on this last Christmas - so it's been almost an entire year. We've been struggling to get feedback on the document and would like the TC to comment on it. It is in my opinion a very simple proposal - yet it allows query to be extensible for many future use cases.

We need to avoid endpoint pollution as one tries to bolt-on new endpoints and arguments for every kind of query parameter under the sun without any plan for the future use cases we have to meet.

     
Issue 15 - As a User, I want to traverse the STIX graph over TAXII in an efficient manner...
Issue 7 - Need ability to request related objects in one request to a distance of 1(?)
Issue 5 - Add support to get embedded objects in a single request, dereference...
Issue 6 - Add ability to find all objects related to a particular STIX object ID
Issue 4 - TAXII Observed Data Query  

Also, I think "the biggest element holding people back" is very much open to interpretation and what persona you fill. From the perspective of most any SOAR-based security tool integrating with a TAXII server, the biggest element holding me back is ability to find indicators that match an observation, or find observation that match an indicator.

Finding out who is related to a STIX object is very much a TIP-centric use case (although I will add that that use case is also addressed in our above proposal as well).

My point is that we should not be trying to come out with a lot of piecemeal-endpoints without thinking of the bigger picture of query.

-
Jason Keirstead
Lead Architect - IBM Security Connect

www.ibm.com/security

"Things may come to those who wait, but only the things left by those who hustle." - Unknown





From:        
Bret Jordan <Bret_Jordan@symantec.com>
To:        
"cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Date:        
11/19/2018 03:26 PM
Subject:        
[cti-taxii] TAXII Query / Search
Sent by:        
<cti-taxii@lists.oasis-open.org>






All,

Over the past year I have heard from several people that they are unable to implement TAXII because there is no way to search for relationships. Yes, there are lots of other things that people would like to do with a query request, but the ability to search for a relationship seems to be the single biggest missing element holding people back from implementing TAXII.

I have talked with Drew and Gary about how we could solve this in the very short term and would like to propose the following solution to the TC. If the TC agrees with this, Drew and I can add this to Working Draft 05 in the next week and submit Working Draft 05 to the TC for Review and CSD ballot the first of December.

Proposal

1) We create an endpoint called:
<api-root>/collections/<id>/relationships/related-to/<stix-id>/

This endpoint would return any SROs where the source_ref or target_ref matched the supplied STIX ID. This would be a very simple database query, and would not require a lot of computation work.  This endpoint would be mandatory to implement for all TAXII Servers as defined in the conformance section.


2) We add an optional URL parameter for the relationships endpoint called:
?deref=true

This optional URL parameter would tell the server to automatically send the objects that are referenced in the SROs that are being returned. From a conformance standpoint, this URL parameter would be optional to implement.

If a client makes a request with that URL parameter and the server has not implemented it, the server would respond with an HTTP 501 Not Implemented error code and a TAXII error message that says the URL parameter is not implemented.  

From a database performance standpoint, this URL parameter would require that the server perform multiple database queries for each request and would require the server to do some book keeping to ensure that it does not send the same object multiple times. However, this feature would eliminate the overhead of parsing multiple RESTful requests from the client as it comes back and asks for each of the objects one at a time.

Conclusion

We have some support for doing this already as this would be easy to implement in the specification and in code, and would solve a major blocker that has been identified. I would be curious to know if anyone else would support this or be against this.  This does not mean that we will not look at a more elaborate pattern based / property based query solution in the future.

Bret









This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

 



[Date Prev] | [Thread Prev] | [Thread Next] | [Date Next] -- [Date Index] | [Thread Index] | [List Home]