Re: [EXT] Re: [cti] TAXII Pagination Example Text

[Andras Iklody <andras.iklody@circl.lu>]

Hello Bret,

the problem with this is that TAXII / STIX are meant to work on the
transport layer, not necessarily the native storage behind it. For
example, if we deal with a passiveDNS system behind a TAXII connector,
having microsecond precision would be absolutely meaningless, hence the
second-precision values are converted to "YYYY-MM-DDTHH:MM:SS.ssssssZ"
on the fly by padding everything beyond seconds.

This would mean that for systems such as this pagination would not be
possible if more values exist than the limit / page.

Best regards,
Andras

On 04.09.19 23:36, Bret Jordan wrote:
> Hi Andras,
> 
> In TAXII we define the timestamp to be "YYYY-MM-DDTHH:MM:SS.ssssssZ",
> aka microsecond precision. This timestamp is used for all records as
> they are added to the TAXII Server.  So under normal conditions
> microsecond precision should give ample amount of space per second for
> new records coming in.
> 
> Now there is a possibility that one may try to bulk load records and
> give every new record the same timestamp.  This would be a less than
> ideal design.  However, if this is what you have, and someone requests
> more records than you can give, then you would probably respond with an
> error message telling the client that you can not complete the request
> since there are more records with the exact same microsecond timestamp
> than the client requested. 
> 
> Bret
> 
> ------------------------------------------------------------------------
> *From:* cti@lists.oasis-open.org <cti@lists.oasis-open.org> on behalf of
> Andras Iklody <andras.iklody@circl.lu>
> *Sent:* Wednesday, September 4, 2019 1:01 AM
> *To:* cti@lists.oasis-open.org <cti@lists.oasis-open.org>
> *Subject:* [EXT] Re: [cti] TAXII Pagination Example Text
>  
> Hello Bret,
> 
> just curious, how should we deal with more than 100 records that were
> added at the same time?
> 
> Best regards,
> Andras
> 
> On 03.09.19 21:59, Bret Jordan wrote:
>> All,
>> 
>> Here is the text we talked about on the working call today.  Please send
>> any changes or suggestions to the list by end of day next Tuesday the
>> 10th.  After we get all suggestions and changes, Drew and I will add
>> this to TAXII.
>> 
>> 
>> TAXII 2.1 supports pagination of large result sets on certain endpoints.
>> These endpoints return results sorted in ascending order by the date
>> they were added to the collection (see section 3.3). The server may
>> limit the number of responses in result to a query, either as the result
>> of a server-specified limit, or in response to a limit parameter passed
>> by the client as part of a query (see section 3.4). If more records are
>> available than are returned, the client may paginate through the
>> remaining records by using the added_after filter parameter and the
>> date/time value from the X-TAXII-Date-Added-Last header.
>> 
>> Example:
>> 
>>  1. Collection High-Value-Indicators has 1000 records in it.
>>  2. The client or server has limited all responses to 100 records at a time.
>>  3. A client will make a request and the server will respond with the
>>     first 100 records.
>>  4. The server will also populate the two X headers for TAXII,
>>     X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last. These headers
>>     will contain the date/time value of when the first and last records
>>     were added to the TAXII server.
>>  5. The server will also set the “more” property to a value of true on
>>     the TAXII envelope.
>>  6. When a client wants to obtain the next 100 records, the client will
>>     populate the added_after filter with the value from the previous
>>     results X-TAXII-Date-Added-Last header. This will ensure that the
>>     client starts requests the records immediate following the data that
>>     was returned in their last request.
>> 
>> 
>> 
>> Bret
>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://clicktime.symantec.com/3F9fXMMcYrXabjjNwg7JCCV7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php
> 
>