RE: [Non-DoD Source] [cti] Re: [EXT] Re: [cti] TAXII Pagination Example Text

["Mates, Jeffrey CIV DC3/TSD" <Jeffrey.Mates@dc3.mil>]

It's also worth noting that microsecond precision is often a lie in multi-server systems because time skew between systems exceeds single microsecond accuracy.  Due to among other things:

1. Unpredictable delays caused by switching and routing NTP packets
2. Every 1,000ft of fiber adds ~1 micro-second of latency, and most of us don't account for this when building out our infrastructure to ensure uniform cable lengths between every device in every data center.
3. NTP packets need to be processed and applied to the CPU's clock which depends on system overhead (but I think it does have higher priority than must due to fun with interrupts)
4. Quartz crystal clocks can have up 5 microseconds of drift per second.
 
Jeffrey Mates, Civ DC3/TSD
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Computer Scientist
Technical Solutions Development
jeffrey.mates@dc3.mil
410-694-4335


-----Original Message-----
From: cti@lists.oasis-open.org <cti@lists.oasis-open.org> On Behalf Of Andras Iklody
Sent: Friday, September 6, 2019 7:49 AM
To: Bret Jordan <Bret_Jordan@symantec.com>; cti@lists.oasis-open.org
Subject: [Non-DoD Source] [cti] Re: [EXT] Re: [cti] TAXII Pagination Example Text

All active links contained in this email were disabled.  Please verify the identity of the sender, and confirm the authenticity of all links contained within the message prior to copying and pasting the address to a Web browser.  




----

Hello Bret,

the problem with this is that TAXII / STIX are meant to work on the transport layer, not necessarily the native storage behind it. For example, if we deal with a passiveDNS system behind a TAXII connector, having microsecond precision would be absolutely meaningless, hence the second-precision values are converted to "YYYY-MM-DDTHH:MM:SS.ssssssZ"
on the fly by padding everything beyond seconds.

This would mean that for systems such as this pagination would not be possible if more values exist than the limit / page.

Best regards,
Andras

On 04.09.19 23:36, Bret Jordan wrote:
> Hi Andras,
> 
> In TAXII we define the timestamp to be "YYYY-MM-DDTHH:MM:SS.ssssssZ", 
> aka microsecond precision. This timestamp is used for all records as 
> they are added to the TAXII Server.  So under normal conditions 
> microsecond precision should give ample amount of space per second for 
> new records coming in.
> 
> Now there is a possibility that one may try to bulk load records and 
> give every new record the same timestamp.  This would be a less than 
> ideal design.  However, if this is what you have, and someone requests 
> more records than you can give, then you would probably respond with 
> an error message telling the client that you can not complete the 
> request since there are more records with the exact same microsecond 
> timestamp than the client requested.
> 
> Bret
> 
> ----------------------------------------------------------------------
> --
> *From:* cti@lists.oasis-open.org <cti@lists.oasis-open.org> on behalf 
> of Andras Iklody <andras.iklody@circl.lu>
> *Sent:* Wednesday, September 4, 2019 1:01 AM
> *To:* cti@lists.oasis-open.org <cti@lists.oasis-open.org>
> *Subject:* [EXT] Re: [cti] TAXII Pagination Example Text
>  
> Hello Bret,
> 
> just curious, how should we deal with more than 100 records that were 
> added at the same time?
> 
> Best regards,
> Andras
> 
> On 03.09.19 21:59, Bret Jordan wrote:
>> All,
>> 
>> Here is the text we talked about on the working call today.  Please 
>> send any changes or suggestions to the list by end of day next 
>> Tuesday the 10th.  After we get all suggestions and changes, Drew and 
>> I will add this to TAXII.
>> 
>> 
>> TAXII 2.1 supports pagination of large result sets on certain endpoints.
>> These endpoints return results sorted in ascending order by the date 
>> they were added to the collection (see section 3.3). The server may 
>> limit the number of responses in result to a query, either as the 
>> result of a server-specified limit, or in response to a limit 
>> parameter passed by the client as part of a query (see section 3.4). 
>> If more records are available than are returned, the client may 
>> paginate through the remaining records by using the added_after 
>> filter parameter and the date/time value from the X-TAXII-Date-Added-Last header.
>> 
>> Example:
>> 
>>  1. Collection High-Value-Indicators has 1000 records in it.
>>  2. The client or server has limited all responses to 100 records at a time.
>>  3. A client will make a request and the server will respond with the
>>     first 100 records.
>>  4. The server will also populate the two X headers for TAXII,
>>     X-TAXII-Date-Added-First and X-TAXII-Date-Added-Last. These 
>>headers
>>     will contain the date/time value of when the first and last 
>>records
>>     were added to the TAXII server.
>>  5. The server will also set the âmoreâ property to a value of true 
>>on
>>     the TAXII envelope.
>>  6. When a client wants to obtain the next 100 records, the client 
>>will
>>     populate the added_after filter with the value from the previous
>>     results X-TAXII-Date-Added-Last header. This will ensure that the
>>     client starts requests the records immediate following the data 
>>that
>>     was returned in their last request.
>> 
>> 
>> 
>> Bret
>> 
>> 
> 
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that 
> generates this mail.  Follow this link to all your TCs in OASIS at:
> Caution-https://clicktime.symantec.com/3F9fXMMcYrXabjjNwg7JCCV7Vc?u=ht
> tps%3A%2F%2FCaution-www.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fport
> al%2Fmy_workgroups.php
> 
> 

---------------------------------------------------------------------
To unsubscribe from this mail list, you must leave the OASIS TC that generates this mail.  Follow this link to all your TCs in OASIS at:
Caution-https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php

Attachment: smime.p7s
Description: S/MIME cryptographic signature