Re: [cti] Re: [EXT] Re: [cti] TAXII Pagination Example Text

The problem with this approach is that it works great for systems where
data has this information at rest. If we have sensors that we want to
query for data, taxii is immediately not suitable because of this. We'd
be converting the data on the fly from other formats, meaning that there
is no insertion date. We'd have to map the timestamp to whatever the
data store on the far side has available.

Should the recommendation be that none of these tools are suitable
candidates to be used with TAXII?

Best regards,
Andras

On 07.09.19 13:11, Bret Jordan wrote:
> I would like to stress that we have had this discussion many times.
> Some of the discussions have run for months. In the end, when we review
> all of the use cases and the 90/10 most common needs, we have always
> come back to the simple pagination by the date an object was added to
> the TAXII server. This design makes it super simple for client and server.
>
> For those constantly advocating for a different method, please remember
> that content may rapidly be changing on the system due to the very
> nature of threat intelligence. As such, you really need a canonical
> unchanging entry point to pull / sync data from, thus the date_added to
> the system.
>
> To Jeff's concerns, once you have written the object to one of your
> TAXII servers, you can just sync that entire record, including the date
> added value, to other TAXII servers. This way you do not need to worry
> about jitter.
>
> To Andras' concerns, the TAXII date added timestamp is NOT part of the
> STIX / content data and one really must not use the STIX modified
> timestamp as the date added value. We call that out very explicitly in
> the TAXII spec.
>
> If you have designed your system to bulk load data using the same date
> added value, then I might suggest you look at changing that. You should
> be able to simply use date_added+.000001 for each record as it is added.
>
> Bret
> ------------------------------------------------------------------------
> *From:* cti@lists.oasis-open.org <cti@lists.oasis-open.org> on behalf of
> Wesley Brown <wbrown@lookingglasscyber.com>
> *Sent:* Friday, September 6, 2019 11:54 AM
> *To:* drew.varner@ninefx.com <drew.varner@ninefx.com>; Jason Keirstead
> <Jason.Keirstead@ca.ibm.com>
> *Cc:* Allan Thomson <athomson@lookingglasscyber.com>; Andras Iklody
> <andras.iklody@circl.lu>; Bret Jordan <Bret_Jordan@symantec.com>;
> cti@lists.oasis-open.org <cti@lists.oasis-open.org>
> *Subject:* Re: [cti] Re: [EXT] Re: [cti] TAXII Pagination Example Text
>
>
> I would add that there’s some implied guarantees behind cursor support:
>
>   * The ‘view’ that the cursor is created from remains consistent and
>     unchanging.
>   * Sorting of view, and if we allow new items to be inserted, then
>     where does the insertion happen?
>   * Maintaining state between requests.
>
>
>
> While it makes things easier for the client, it imposes a heavy cost on
> server infrastructure that is not obvious – I’ve seen cases where we had
> thousands of cursors open concurrently, and the backend in this case
> (ElasticSearch) could not ever do any compaction or deletion of shards
> because it needed to maintain snapshots for each of the cursors in question.
>
>
>
> -
>
> Wes Brown
>
> Distinguished Engineer
>
> Lookingglass Cyber Solutions
>
>
>
>
>
> *From: *<cti@lists.oasis-open.org> on behalf of "drew.varner@ninefx.com"
> <drew.varner@ninefx.com>
> *Date: *Friday, September 6, 2019 at 1:04 PM
> *To: *Jason Keirstead <Jason.Keirstead@ca.ibm.com>
> *Cc: *Allan Thomson <athomson@lookingglasscyber.com>, Andras Iklody
> <andras.iklody@circl.lu>, Bret Jordan <Bret_Jordan@symantec.com>,
> "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
> *Subject: *Re: [cti] Re: [EXT] Re: [cti] TAXII Pagination Example Text
>
>
>
> I believe we have enough information to provide a deterministic sorting
> token for every type of item a TAXII server returns, assuming we’re
> limiting the discussion to STIX results. Sometimes, this token must be a
> compound key, such as the combination of an object ID and it’s
> modification time.
> See https://clicktime.symantec.com/3M64TFS9SXjA2JBNWmZPYSN7Vc?u=https%3A%2F%2Fgithub.com%2Foasis-tcs%2Fcti-taxii2%2Fissues%2F50%23issuecomment-367505271
> <https://clicktime.symantec.com/3qny6zSUj9ZAkjrtg6Jwna7Vc?u=https%3A%2F%2Fgithub.com%2Foasis-tcs%2Fcti-taxii2%2Fissues%2F50%23issuecomment-367505271>
>
>
>
>
> I think that stateless approach from a scalability perspective than
> maintaining cursor state between requests.
>
>
> On Sep 6, 2019, at 12:30 PM, Jason Keirstead <Jason.Keirstead@ca.ibm.com
> <mailto:Jason.Keirstead@ca.ibm.com>> wrote:
>
>     Cloud APIs such as AWS, Azure, Cloudant, etc all handle this problem
>     using a system where the response to your initial request contains
>     at the end of it a continuation marker/token (UUID), and allow you
>     to fetch the next set of records using that marker/token.
>
>     This obviously requires the implementer of a TAXII server to have
>     some form of server-side cursor/result set management. But it's a
>     lot more elegant for the consumer and avoids all these
>     timestamp-related problems.
>
>     -
>     Jason Keirstead
>     Chief Architect - IBM Security Threat Management
>     https://clicktime.symantec.com/32icskAfVT3rR3YvXgWEnku7Vc?u=www.ibm.com%2Fsecurity
>     <https://clicktime.symantec.com/3QwfrzEEx4EPu35vppaQiuV7Vc?u=www.ibm.com%2Fsecurity>
>
>     "Would you like me to give you a formula for success? It's quite
>     simple, really. Double your rate of failure."
>
>     - Thomas J. Watson
>
>
>
>     From: Allan Thomson <athomson@lookingglasscyber.com
>     <mailto:athomson@lookingglasscyber.com>>
>     To: Andras Iklody <andras.iklody@circl.lu
>     <mailto:andras.iklody@circl.lu>>, Bret Jordan
>     <Bret_Jordan@symantec.com <mailto:Bret_Jordan@symantec.com>>,
>     "cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>"
>     <cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org>>
>     Date: 09/06/2019 11:10 AM
>     Subject: [EXTERNAL] Re: [cti] Re: [EXT] Re: [cti] TAXII
>     Pagination Example Text
>     Sent by: <cti@lists.oasis-open.org
>     <mailto:cti@lists.oasis-open.org>>
>
>     ------------------------------------------------------------------------
>
>
>
>
>     If you using the timestamps in Passive DNS (I know at least one
>     vendor that has stated they don’t want or provide accurate enough
>     timestamps) then you need to use the insertion timestamp local to
>     the TAXII server not the timestamp in the data itself.
>
>     And technically you need to do that on all data because the
>     timestamps in TAXII for the records are not the timestamps of the
>     data but rather the timestamp of addition to TAXII.
>
>     This is required because you could have older Intel that is months
>     old being added to a TAXII server today and if someone is sync-ing
>     to that server then they need to know what was added today.
>
>     Allan
>
>     On 9/6/19, 4:49 AM, "cti@lists.oasis-open.org
>     <mailto:cti@lists.oasis-open.org> on behalf of Andras Iklody"
>     <cti@lists.oasis-open.org <mailto:cti@lists.oasis-open.org> on
>     behalf of andras.iklody@circl.lu <mailto:andras.iklody@circl.lu>> wrote:
>
>     Hello Bret,
>
>     the problem with this is that TAXII / STIX are meant to work on the
>     transport layer, not necessarily the native storage behind it. For
>     example, if we deal with a passiveDNS system behind a TAXII
>     connector,
>     having microsecond precision would be absolutely meaningless,
>     hence the
>     second-precision values are converted to
>     "YYYY-MM-DDTHH:MM:SS.ssssssZ"
>     on the fly by padding everything beyond seconds.
>
>     This would mean that for systems such as this pagination would not be
>     possible if more values exist than the limit / page.
>
>     Best regards,
>     Andras
>
>     On 04.09.19 23:36, Bret Jordan wrote:
>     > Hi Andras,
>     >
>     > In TAXII we define the timestamp to be
>     "YYYY-MM-DDTHH:MM:SS.ssssssZ",
>     > aka microsecond precision. This timestamp is used for all
>     records as
>     > they are added to the TAXII Server. So under normal conditions
>     > microsecond precision should give ample amount of space per
>     second for
>     > new records coming in.
>     >
>     > Now there is a possibility that one may try to bulk load
>     records and
>     > give every new record the same timestamp. This would be a less
>     than
>     > ideal design. However, if this is what you have, and someone
>     requests
>     > more records than you can give, then you would probably respond
>     with an
>     > error message telling the client that you can not complete the
>     request
>     > since there are more records with the exact same microsecond
>     timestamp
>     > than the client requested.
>     >
>     > Bret
>     >
>     >
>     ------------------------------------------------------------------------
>     > *From:* cti@lists.oasis-open.org
>     <mailto:cti@lists.oasis-open.org> <cti@lists.oasis-open.org
>     <mailto:cti@lists.oasis-open.org>> on behalf of
>     > Andras Iklody <andras.iklody@circl.lu
>     <mailto:andras.iklody@circl.lu>>
>     > *Sent:* Wednesday, September 4, 2019 1:01 AM
>     > *To:* cti@lists.oasis-open.org
>     <mailto:cti@lists.oasis-open.org> <cti@lists.oasis-open.org
>     <mailto:cti@lists.oasis-open.org>>
>     > *Subject:* [EXT] Re: [cti] TAXII Pagination Example Text
>     >
>     > Hello Bret,
>     >
>     > just curious, how should we deal with more than 100 records
>     that were
>     > added at the same time?
>     >
>     > Best regards,
>     > Andras
>     >
>     > On 03.09.19 21:59, Bret Jordan wrote:
>     >> All,
>     >>
>     >> Here is the text we talked about on the working call today.
>     Please send
>     >> any changes or suggestions to the list by end of day next
>     Tuesday the
>     >> 10th. After we get all suggestions and changes, Drew and I
>     will add
>     >> this to TAXII.
>     >>
>     >>
>     >> TAXII 2.1 supports pagination of large result sets on certain
>     endpoints.
>     >> These endpoints return results sorted in ascending order by
>     the date
>     >> they were added to the collection (see section 3.3). The
>     server may
>     >> limit the number of responses in result to a query, either as
>     the result
>     >> of a server-specified limit, or in response to a limit
>     parameter passed
>     >> by the client as part of a query (see section 3.4). If more
>     records are
>     >> available than are returned, the client may paginate through the
>     >> remaining records by using the added_after filter parameter
>     and the
>     >> date/time value from the X-TAXII-Date-Added-Last header.
>     >>
>     >> Example:
>     >>
>     >> 1. Collection High-Value-Indicators has 1000 records in it.
>     >> 2. The client or server has limited all responses to 100
>     records at a time.
>     >> 3. A client will make a request and the server will respond
>     with the
>     >> first 100 records.
>     >> 4. The server will also populate the two X headers for TAXII,
>     >> X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last.
>     These headers
>     >> will contain the date/time value of when the first and
>     last records
>     >> were added to the TAXII server.
>     >> 5. The server will also set the “more” property to a value of
>     true on
>     >> the TAXII envelope.
>     >> 6. When a client wants to obtain the next 100 records, the
>     client will
>     >> populate the added_after filter with the value from the
>     previous
>     >> results X-TAXII-Date-Added-Last header. This will ensure
>     that the
>     >> client starts requests the records immediate following the
>     data that
>     >> was returned in their last request.
>     >>
>     >>
>     >>
>     >> Bret
>     >>
>     >>
>     >
>     >
>     ---------------------------------------------------------------------
>     > To unsubscribe from this mail list, you must leave the OASIS TC
>     that
>     > generates this mail. Follow this link to all your TCs in OASIS at:
>     >
>     https://clicktime.symantec.com/3F9fXMMcYrXabjjNwg7JCCV7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php
>     >
>     >
>
>     ---------------------------------------------------------------------
>     To unsubscribe from this mail list, you must leave the OASIS TC that
>     generates this mail. Follow this link to all your TCs in OASIS at:
>
>     https://clicktime.symantec.com/382JFRioAXKQagJ4tyVvPfX7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php
>     <https://clicktime.symantec.com/312qzDFGiErbUNi5nVaqAuo7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php>
>
>
>
>
>
>

cti message