cti-taxii message
[Date Prev]
| [Thread Prev]
| [Thread Next]
| [Date Next]
--
[Date Index]
| [Thread Index]
| [List Home]
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
- From: "Jason Keirstead" <Jason.Keirstead@ca.ibm.com>
- To: Matt Pladna <mpladna@lookingglasscyber.com>
- Date: Mon, 26 Nov 2018 18:32:50 +0000
I am open to the idea of removing some
of them from the spec - however I will point out that all of them are optional
to implement, for exactly that reason. The relationship one as well
must be optional to implement, as not all TAXII profiles would be able
to return that information. So if we remove all of the other types then
we will have one type which is optional, which is a bit odd.
I am also open to your URI change suggestion.
Can you make these suggestions in the
doc ?
-
Jason Keirstead
Lead Architect - IBM Security Connect
www.ibm.com/security
"Things may come to those who wait, but only the things left by those
who hustle." - Unknown
From:
Matt Pladna <mpladna@lookingglasscyber.com>
To:
Bret Jordan <Bret_Jordan@symantec.com>,
Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc:
Gary Jay Katz <gary.katz@FireEye.com>,
"cti-taxii@lists.oasis-open.org" <cti-taxii@lists.oasis-open.org>
Date:
11/26/2018 01:28 PM
Subject:
Re: [cti-taxii]
Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Sent by:
<cti-taxii@lists.oasis-open.org>
Would it be tenable to have a hybrid
approach where we take the proposal here https://docs.google.com/document/d/1Cy_9Bh5tKEkDHGg2iv5c3AwriqVr7ygbKXWOv4-uHxs/edit#heading=h.1jcqb6vc5y7zand scope it down to only handle relationship (specifically related-to)
queries?
Meaning the google doc will have one
information-request type, which will be related-to query requests. It
can still be extended later with more types. Rather than return a
specific response type (information-response) what if it just returned
the STIX content as well (in a bundle? Stream?)? If a TAXII type
of information-response is necessary later, we can add it and support STIX
2 for backward compatibility.
I think having a type dedicated to taking
a query request is a good idea. JSON query/information request objects
can be extended in the future and I think will give us more options than
relying on endpoints and parameters in the URL.
If an endpoint of â<api-root>/collections/<id>/information-request/â
is too generic, we could make the types explicit in the URL:
â<api-root>/collections/<id>/information-request/related-toâ
Iâm not sure which is better, but does this help reduce
scope to something we can do soon AND leave us in a place to grow?
From: <cti-taxii@lists.oasis-open.org>
on behalf of Bret Jordan <Bret_Jordan@symantec.com>
Date: Tuesday, November 20, 2018 at 09:02
To: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc: Gary Jay Katz <gary.katz@FireEye.com>, "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Jason,
If you remember we had a global search
in some of the early drafts of 2.0. But it was pulled because several
people did not like it. I think any kind of search like that would
need to be done at both the collection level and at the api-root level,
where the api-root level is optional to implement.
All,
So far we have only heard from 6 people
in the TC so far. I would like to hear what everyone else thinks.
I plan on talking about this on next
weeks working call, just FYI.
Bret
Sent from my Commodore 64
PGP Fingerprint: 63B4 FC53 680A 6B7D
1447 F2C0 74F8 ACAE 7415 0050
On Nov 20, 2018, at 6:25 AM, Jason Keirstead <Jason.Keirstead@ca.ibm.com>
wrote:
I have no issue with the concept of getting
an MVP, out so long as the future is considered so we don't box ourselves
into a corner. I have already given very specific future-facing examples
of problems this MVP introduces, that need to be thought through. We have
already been down this path, with several breaking changes between TAXII
2.0 and TAXII 2.1, and parties coming to the fore saying they are "waiting"
for 2.1 so they don't have to re-do implementations.
If TAXII 2.2 introduces more breaking changes because we did not consider
all the use cases and tried to rush things out, then we are going to rapidly
start to loose faith in the marketplace.
I'll also again re-iterate that it is very much debatable even if this
is the actual MVP we need at this point in time. It depends highly on where
in the intelligence ecosystem you sit. I will take a FireEye product as
an example - if I have a file hash and during a hunt I want to query Helix
using it's API for all of the sightings of that file hash, this MVP does
not help me. We have no way to do this extremely basic use case in STIX
today. I think this global query use case is *more* critical than being
able to de-reference a relationship only inside a single collection...
collections after all are often mirrored wholesale.
RE query - this is the exact use case that the STIX Extension process is
meant to solve, by forcing folks down a path where proposals are implemented
and tested and vetted in the wild before they make it into the standard.
I think that anything we do along the lines of Query should have to be
proven in the market before it makes it into the spec along with the potential
for future breakages.
-
Jason Keirstead
Lead Architect - IBM Security Connect
www.ibm.com/security
"Things may come to those who wait, but only the things left by those
who hustle." - Unknown
From: Gary
Jay Katz <gary.katz@FireEye.com>
To: Bret Jordan
<Bret_Jordan@symantec.com>,
Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc: "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Date: 11/19/2018
07:14 PM
Subject: Re:
[cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Sent by: <cti-taxii@lists.oasis-open.org>
Jason,
Thatâs a good question. One which I can provide my
response, but I believe requires a larger discussion, which is part of
the reason why I believe we should get an MVP out now, rather than waiting.
To Bretâs point, Iâd rather not wait 6-12 months to get a release
out.
Currently we have ways to easily search for a STIX object.
It does not allow for STIX querying or the complex use cases we have
identified, but it does allow someone to easily retrieve an SDO. And
that is a good thing. There should be an easy way to perform a search,
whether that is for an object or a relationship and it should be easy for
a consumer to use that functionality to traverse the graph and easy for
a producer to implement such functionality. Not every producer will
have a backend database (or the requirements) to allow for complex querying.
I truthfully do not see the issue with allowing restful methods to
easily traverse the graph and then later add in full query support where
you could accomplish the same thing in addition to much more powerful queries.
People may disagree, and we can and should debate this as TAXII query
is designed, but I personally do not believe that the full TAXII query
support needs to follow the same pattern as is defined for retrieving an
SDO or SRO.
Just my opinion, but hope it helps explain where my thoughts are.
-Gary
From: Bret Jordan <Bret_Jordan@symantec.com>
Date: Monday, November 19, 2018 at 4:41 PM
To: Jason Keirstead <Jason.Keirstead@ca.ibm.com>,
Gary Katz <gary.katz@FireEye.com>
Cc: "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
This provides a solution that we can have now. At the current
rate any more advanced solution is at least 6-12 more months away. Yes,
we may need to do things differently in the future, but that is okay. We
can always deprecate features as needed. But I figured a more RESTful design
would get us up and going sooner rather than later.
By my count the following individuals support the proposal
I have submitted:
Myself
Drew
Nicholas
Sean
Gary
The individuals that do not support it:
Jason
I am curious to hear what others have to say. Can everyone
please chime in on this debate.
Bret
From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent: Monday, November 19, 2018 2:25:37 PM
To: Gary Jay Katz
Cc: Bret Jordan; cti-taxii@lists.oasis-open.org
Subject: Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
Gary - how would you propose in the future we can extend or expand this
approach, without redoing it from scratch?
This is the problem with having query-specific endpoints or query parameters
such as "related to" - they can't be extended or added to in
any logical way.
It will result in either no way to actually do combinations of parameters,
or an unworkable mishmash of normative text quantifying all of the edge
cases of which endpoints can be combined with which others to get what
behaviour.
Sent from IBM Verse
Gary Jay Katz --- Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query
/ Search ---
Jason,
I believe you are correct, there are a number of complex
use cases that this would not meet. For those that remember back
to some of the early face-to-face meetings, I pushed for trying to find
a solution that met all of these solutions in the first release. I
now believe that this needs to be taken in parts. There may be certain
TAXII implementations that only need to and are able to support basic search
and pivoting, while others will be built to perform complex querying such
as what you are suggesting.
Having an ability to easily query relationships and pivot
off of them which does not support the more complex use cases, does not
mean that the capability needs to be rewritten or that those complex use
cases are invalid. We just need ways to do the simple things simply,
while eventually put in place the ability to do the complex capabilities.
Currently we have no ability to query relationships or pivot.
This is a blocker for anyone trying to provide the baseline capabilities
to support the STIX model using TAXII. I would suggest we get the
baseline capabilities in place to allow releasing a minimal viable product
in the near-term while continuing to work towards more advanced query capabilities
in a future release.
-Gary
From: <cti-taxii@lists.oasis-open.org>
on behalf of Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Date: Monday, November 19, 2018 at 3:38 PM
To: Bret Jordan <Bret_Jordan@symantec.com>
Cc: "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Subject: [cti-taxii] Re: [EXT] Re: [cti-taxii] TAXII Query / Search
The problem with this approach of "getting something done in a a few
days", is it is highly likely it will have to be re-done in the future
because it isn't considering the other use cases.
For example - In the future how can I find all sightings related to Object
Foo that match the indicator 1.2.3.4, with your proposal? Seeing how any
future "match the indicator" portion would be an entirely different
endpoint than the "related-to" endpoint, I now can't combine
them - and therefore would have to make multiple - perhaps dozens, hundreds,
or even thousands of round-trips to the server to collect this information.
-
Jason Keirstead
Lead Architect - IBM Security Connect
www.ibm.com/security
"Things may come to those who wait, but only the things left by those
who hustle." - Unknown
From: Bret
Jordan <Bret_Jordan@symantec.com>
To: Jason Keirstead
<Jason.Keirstead@ca.ibm.com>
Cc: "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Date: 11/19/2018
04:27 PM
Subject: Re:
[EXT] Re: [cti-taxii] TAXII Query / Search
Jason, the proposal you have submitted is valid, and one potential solution.
However, it represents a fair bit of complexity on the specification
and on implementations. There are also a lot of unanswered questions about
how it would actually be done. To support this, we would also need to create
several conformance levels for it, which is non-trivial.
Per your own statements, over the past 12 months I have seen very little
to no public support for it. As such, I am trying to find a simpler
solution that we can do now and then look at the more complicated
solution later. I fear if we try to do a more complicated solution
now, then TAXII 2.1 will still be many months away from being done.
The proposal I have worked on with Drew and Gary is something we can do
very easily in a matter of days, which could enable us to finish TAXII
2.1 within the next month.
But all of this is up to the TC to decide. We can see what kind of
feedback we get, and see where the majority of TC members sit. But at this
stage, I would personally (not as a chair or editor) prefer a more simple
solution that can be used now.
Bret
From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent: Monday, November 19, 2018 1:06:11 PM
To: Bret Jordan
Cc: cti-taxii@lists.oasis-open.org
Subject: [EXT] Re: [cti-taxii] TAXII Query / Search
Hi Brett - Terry and I have an alternative proposal we've been asking the
TC to have a look at here
https://docs.google.com/document/d/1Cy_9Bh5tKEkDHGg2iv5c3AwriqVr7ygbKXWOv4-uHxs/edit#heading=h.1jcqb6vc5y7z
As we lay out in the document, there are a number of issues TAXII has with
query, and I think the proposal you have below is insufficient to solve
the use cases we have in Github:
We first started working on this last Christmas - so it's been almost an
entire year. We've been struggling to get feedback on the document and
would like the TC to comment on it. It is in my opinion a very simple proposal
- yet it allows query to be extensible for many future use cases.
We need to avoid endpoint pollution as one tries to bolt-on new endpoints
and arguments for every kind of query parameter under the sun without any
plan for the future use cases we have to meet.
Issue
15 - As a User, I want to traverse the STIX graph over TAXII in an efficient
manner...
Issue
7 - Need ability to request related objects in one request to a distance
of 1(?)
Issue
5 - Add support to get embedded objects in a single request, dereference...
Issue
6 - Add ability to find all objects related to a particular STIX object
ID
Issue
4 - TAXII Observed Data Query
Also, I think "the biggest element holding people back" is very
much open to interpretation and what persona you fill. From the perspective
of most any SOAR-based security tool integrating with a TAXII server, the
biggest element holding me back is ability to find indicators that match
an observation, or find observation that match an indicator.
Finding out who is related to a STIX object is very much a TIP-centric
use case (although I will add that that use case is also addressed in our
above proposal as well).
My point is that we should not be trying to come out with a lot of piecemeal-endpoints
without thinking of the bigger picture of query.
-
Jason Keirstead
Lead Architect - IBM Security Connect
www.ibm.com/security
"Things may come to those who wait, but only the things left by those
who hustle." - Unknown
From: Bret
Jordan <Bret_Jordan@symantec.com>
To: "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Date: 11/19/2018
03:26 PM
Subject: [cti-taxii]
TAXII Query / Search
Sent by: <cti-taxii@lists.oasis-open.org>
All,
Over the past year I have heard from several people that they are unable
to implement TAXII because there is no way to search for relationships.
Yes, there are lots of other things that people would like to do with a
query request, but the ability to search for a relationship seems to be
the single biggest missing element holding people back from implementing
TAXII.
I have talked with Drew and Gary about how we could solve this in the very
short term and would like to propose the following solution to the TC.
If the TC agrees with this, Drew and I can add this to Working Draft 05
in the next week and submit Working Draft 05 to the TC for Review and CSD
ballot the first of December.
Proposal
1) We create an endpoint called:
<api-root>/collections/<id>/relationships/related-to/<stix-id>/
This endpoint would return any SROs where the source_ref or target_ref
matched the supplied STIX ID. This would be a very simple database query,
and would not require a lot of computation work. This endpoint would
be mandatory to implement for all TAXII Servers as defined in the conformance
section.
2) We add an optional URL parameter for the relationships endpoint called:
?deref=true
This optional URL parameter would tell the server to automatically send
the objects that are referenced in the SROs that are being returned. From
a conformance standpoint, this URL parameter would be optional to implement.
If a client makes a request with that URL parameter and the server has
not implemented it, the server would respond with an HTTP 501 Not Implemented
error code and a TAXII error message that says the URL parameter is not
implemented.
From a database performance standpoint, this URL parameter would require
that the server perform multiple database queries for each request and
would require the server to do some book keeping to ensure that it does
not send the same object multiple times. However, this feature would eliminate
the overhead of parsing multiple RESTful requests from the client as it
comes back and asks for each of the objects one at a time.
Conclusion
We have some support for doing this already as this would be easy to implement
in the specification and in code, and would solve a major blocker that
has been identified. I would be curious to know if anyone else would support
this or be against this. This does not mean that we will not look
at a more elaborate pattern based / property based query solution in the
future.
Bret
This email and any attachments thereto may contain private, confidential,
and/or privileged material for the sole use of the intended recipient.
Any review, copying, or distribution of this email (or any attachments
thereto) by others is strictly prohibited. If you are not the intended
recipient, please contact the sender immediately and permanently delete
the original and any copies of this email and any attachments thereto.
This email and any attachments thereto may contain private,
confidential, and/or privileged material for the sole use of the intended
recipient. Any review, copying, or distribution of this email (or any attachments
thereto) by others is strictly prohibited. If you are not the intended
recipient, please contact the sender immediately and permanently delete
the original and any copies of this email and any attachments thereto.
[Date Prev]
| [Thread Prev]
| [Thread Next]
| [Date Next]
--
[Date Index]
| [Thread Index]
| [List Home]