Re: [EXT] [cti] [EXT] Re: [cti] TAXII Pagination Example Text

[Andras Iklody <andras.iklody@circl.lu>]

Hi Bret,

So, if this is the recommended hack to use for TAXII, this advice could
perhaps be in the final text to avoid other implementers falling into
the trap of not being able to fetch subsets of the data as in the
situation I've explained before.

So to reiterate, date_added should be used as the iterator for
pagination, use consecutively incremented micro precision for systems
that emit second precision data.

As for writing a proposal for changes, I think based on our previous
experiences it's a route that we're not willing to go down - fighting
for absolutely minor changes for years is out of scope for small open
source teams. We're just looking for guidance how to implement what has
been agreed upon by the TC at this point.

Best regards,
Andras

On 09.09.19 17:46, Bret Jordan wrote:
> As I stated in a previous email, you should be able to easily do a date_added+.000001 for each record.  You are going to have to store these records on your system, and thus you will need some basic meta-data to store them anyway.  You probably are going to store which device you got them from, and a bunch of other things.  A simple timestamp should not be a significant challenge.
> 
> If you want to write up a proposal for TAXII 2.2, I am sure others, including me, would love to see it.  Please just make sure your proposal does not break existing use-cases.
> 
> Thanks
> Bret
> 
> 
>> On Sep 9, 2019, at 5:14 PM, Andras Iklody <andras.iklody@circl.lu> wrote:
>>
>> The data is collected in bulk from the other sources. We could indeed
>> attach a timestamp to the bulk collection, but that would be the same
>> for all data collected in one shot exceeding any sane pagination limits.
>>
>> I am somewhat confused that we're back at the data storage question
>> though, TAXII/STIX should NOT have inherent needs when it comes to how
>> data is stored as from what I understood it's purely a
>> transport/exchange format / protocol. If TAXII basically requires us to
>> move all the stored data to true micro second precision (instead of
>> padding it on fetch for lower precision tools), then that will kill a
>> lot of interactions with other tools out there.
>>
>> Best regards,
>> Andras
>>
>> On 09.09.19 17:07, Bret Jordan wrote:
>>> So if you are building a streaming collector that gathers data from other sensors, you have control over how that data gets added / stored on that intermediary streaming server.  Since you may need to paginate the data, you must be definition then need to temporarily store that data.  If you need to store data, then you can define what ever data structures you want.  Thus you can define the date added property with microsecond precision.   Keep in mind that the data added is NOT the date that the end / origin sensor used for its timestamp. 
>>>
>>> I just do not see how this is broken or why it would not work. Please help meâ
>>>
>>> Bret
>