I believe we have enough information to provide a deterministic sorting token for every type of item a TAXII server returns, assuming weâre limiting the discussion to STIX results. Sometimes, this token must be a compound key, such as the combination of an object ID and itâs modification time. See https://github.com/oasis-tcs/cti-taxii2/issues/50#issuecomment-367505271
I think that stateless approach from a scalability perspective than maintaining cursor state between requests. Cloud APIs such as
AWS, Azure, Cloudant, etc all handle this problem using a system where
the response to your initial request contains at the end of it a continuation
marker/token (UUID), and allow you to fetch the next set of records using
that marker/token.This obviously
requires the implementer of a TAXII server to have some form of server-side
cursor/result set management. But it's a lot more elegant for the consumer
and avoids all these timestamp-related problems.- Jason Keirstead Chief Architect - IBM Security Threat Management www.ibm.com/security
"Would you like me to give you a formula for success? It's quite simple,
really. Double your rate of failure."
- Thomas J. WatsonFrom:
Allan
Thomson <athomson@lookingglasscyber.com>To:
Andras
Iklody <andras.iklody@circl.lu>, Bret Jordan <Bret_Jordan@symantec.com>,
"cti@lists.oasis-open.org" <cti@lists.oasis-open.org>Date:
09/06/2019
11:10 AMSubject:
[EXTERNAL]
Re: [cti] Re: [EXT] Re: [cti] TAXII Pagination Example TextSent
by: <cti@lists.oasis-open.org> If you using the timestamps in Passive
DNS (I know at least one vendor that has stated they donât want or provide
accurate enough timestamps) then you need to use the insertion timestamp
local to the TAXII server not the timestamp in the data itself.
And technically you need to do that on all data because the timestamps
in TAXII for the records are not the timestamps of the data but rather
the timestamp of addition to TAXII.
This is required because you could have older Intel that is months old
being added to a TAXII server today and if someone is sync-ing to that
server then they need to know what was added today.
Allan
ïOn 9/6/19, 4:49 AM, "cti@lists.oasis-open.org on behalf of Andras
Iklody" <cti@lists.oasis-open.org on behalf of andras.iklody@circl.lu>
wrote:
Hello Bret, the problem with this is that TAXII / STIX are meant to work
on the transport layer, not necessarily the native storage behind
it. For example, if we deal with a passiveDNS system behind a TAXII
connector, having microsecond precision would be absolutely meaningless,
hence the second-precision values are converted to "YYYY-MM-DDTHH:MM:SS.ssssssZ" on the fly by padding everything beyond seconds. This would mean that for systems such as this pagination
would not be possible if more values exist than the limit / page. Best regards, Andras On 04.09.19 23:36, Bret Jordan wrote: > Hi Andras, > > In TAXII we define the timestamp to be "YYYY-MM-DDTHH:MM:SS.ssssssZ", > aka microsecond precision. This timestamp is used for
all records as > they are added to the TAXII Server. So under normal
conditions > microsecond precision should give ample amount of space
per second for > new records coming in. > > Now there is a possibility that one may try to bulk
load records and > give every new record the same timestamp. This
would be a less than > ideal design. However, if this is what you have,
and someone requests > more records than you can give, then you would probably
respond with an > error message telling the client that you can not complete
the request > since there are more records with the exact same microsecond
timestamp > than the client requested. > > Bret > > ------------------------------------------------------------------------ > *From:* cti@lists.oasis-open.org <cti@lists.oasis-open.org>
on behalf of > Andras Iklody <andras.iklody@circl.lu> > *Sent:* Wednesday, September 4, 2019 1:01 AM > *To:* cti@lists.oasis-open.org <cti@lists.oasis-open.org> > *Subject:* [EXT] Re: [cti] TAXII Pagination Example
Text > > Hello Bret, > > just curious, how should we deal with more than 100
records that were > added at the same time? > > Best regards, > Andras > > On 03.09.19 21:59, Bret Jordan wrote: >> All, >> >> Here is the text we talked about on the working
call today. Please send >> any changes or suggestions to the list by end of
day next Tuesday the >> 10th. After we get all suggestions and changes,
Drew and I will add >> this to TAXII. >> >> >> TAXII 2.1 supports pagination of large result sets
on certain endpoints. >> These endpoints return results sorted in ascending
order by the date >> they were added to the collection (see section 3.3).
The server may >> limit the number of responses in result to a query,
either as the result >> of a server-specified limit, or in response to a
limit parameter passed >> by the client as part of a query (see section 3.4).
If more records are >> available than are returned, the client may paginate
through the >> remaining records by using the added_after filter
parameter and the >> date/time value from the X-TAXII-Date-Added-Last
header. >> >> Example: >> >> 1. Collection High-Value-Indicators has 1000
records in it. >> 2. The client or server has limited all responses
to 100 records at a time. >> 3. A client will make a request and the server
will respond with the >> first 100 records. >> 4. The server will also populate the two X
headers for TAXII, >> X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last.
These headers >> will contain the date/time value of
when the first and last records >> were added to the TAXII server. >> 5. The server will also set the âmoreâ property
to a value of true on >> the TAXII envelope. >> 6. When a client wants to obtain the next
100 records, the client will >> populate the added_after filter with
the value from the previous >> results X-TAXII-Date-Added-Last header.
This will ensure that the >> client starts requests the records
immediate following the data that >> was returned in their last request. >> >> >> >> Bret >> >> > > --------------------------------------------------------------------- > To unsubscribe from this mail list, you must leave the
OASIS TC that > generates this mail. Follow this link to all your
TCs in OASIS at: > https://clicktime.symantec.com/3F9fXMMcYrXabjjNwg7JCCV7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php > > --------------------------------------------------------------------- To unsubscribe from this mail list, you must leave the OASIS
TC that generates this mail. Follow this link to all your TCs
in OASIS at: https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php
|