Re: [EXT] [cti] [EXT] Re: [cti] TAXII Pagination Example Text

[Andras Iklody <andras.iklody@circl.lu>]

The taxii service would internally query a set of other services and
provide these as a response. There is no centralised database for this
data. Again, TAxII is only there for the transport layer, behind that
we'd have a collection of tools with their own data-sets in various
formats - the only timestamp we have is what they provide.

I am not sure where this data-point should come from. We could create
timestamps on the fly, based on the time when the request hits the
service and add incrementing values in the micro-second part, but that
seems like a dirty hack.

Hope this makes sense, but I have a feeling that the current TaXII
design fits a limited type of tools whilst ignoring other use-cases and
it seems to break the original paradigm of it being purely a transport
layer protocol and actually requires data to be in a static database
behind it (if you want to be able to paginate). Or am I missing
something obvious?

Best regards,
Andras

On 10.09.19 14:26, Bret Jordan wrote:
> You keep talking about systems that emit or use second precision data.
> ÂThe TAXII date_added value is not related to the data or content. ÂIt
> is not related to the STIX data at all and it should not be kept. ÂIt is
> an arbitrary value that does not need to be kept. ÂIt is purely a
> tracking point for the TAXII server itself. ÂTwo TAXII servers that have
> the same object will more than likely not have the same value for the
> date added. Â
> 
> So said another way, if I sync content from you, I will not keep the
> date added values from your server. ÂI will assign my own, because this
> meta data is tied to the TAXII server itself. Â An organize may choose
> to keep it, but it does not need to. ÂIt should be considered a throw
> away value. Â
> 
> So just like you have an auto incrementing index value, you would have
> an auto incrementing date value. ÂSo in your solution you could just
> store the date in a Unix time stamp format with microsecond precision.
> This auto incrementing index could just be translated back and fort has
> between RFC3339 format.
> 
> BretÂ
> 
> Sent from my Commodore 128D
> 
> PGP Fingerprint:Â63B4 FC53 680A 6B7D 1447 ÂF2C0 74F8 ACAE 7415 0050
> 
> On Sep 10, 2019, at 10:43 AM, Andras Iklody <andras.iklody@circl.lu
> <mailto:andras.iklody@circl.lu>> wrote:
> 
>> Hi Bret,
>>
>> So, if this is the recommended hack to use for TAXII, this advice could
>> perhaps be in the final text to avoid other implementers falling into
>> the trap of not being able to fetch subsets of the data as in the
>> situation I've explained before.
>>
>> So to reiterate, date_added should be used as the iterator for
>> pagination, use consecutively incremented micro precision for systems
>> that emit second precision data.
>>
>> As for writing a proposal for changes, I think based on our previous
>> experiences it's a route that we're not willing to go down - fighting
>> for absolutely minor changes for years is out of scope for small open
>> source teams. We're just looking for guidance how to implement what has
>> been agreed upon by the TC at this point.
>>
>> Best regards,
>> Andras
>>
>> On 09.09.19 17:46, Bret Jordan wrote:
>>> As I stated in a previous email, you should be able to easily do a
>>> date_added+.000001 for each record. ÂYou are going to have to store
>>> these records on your system, and thus you will need some basic
>>> meta-data to store them anyway. ÂYou probably are going to store
>>> which device you got them from, and a bunch of other things. ÂA
>>> simple timestamp should not be a significant challenge.
>>>
>>> If you want to write up a proposal for TAXII 2.2, I am sure others,
>>> including me, would love to see it. ÂPlease just make sure your
>>> proposal does not break existing use-cases.
>>>
>>> Thanks
>>> Bret
>>>
>>>
>>>> On Sep 9, 2019, at 5:14 PM, Andras Iklody <andras.iklody@circl.lu
>>>> <mailto:andras.iklody@circl.lu>> wrote:
>>>>
>>>> The data is collected in bulk from the other sources. We could indeed
>>>> attach a timestamp to the bulk collection, but that would be the same
>>>> for all data collected in one shot exceeding any sane pagination limits.
>>>>
>>>> I am somewhat confused that we're back at the data storage question
>>>> though, TAXII/STIX should NOT have inherent needs when it comes to how
>>>> data is stored as from what I understood it's purely a
>>>> transport/exchange format / protocol. If TAXII basically requires us to
>>>> move all the stored data to true micro second precision (instead of
>>>> padding it on fetch for lower precision tools), then that will kill a
>>>> lot of interactions with other tools out there.
>>>>
>>>> Best regards,
>>>> Andras
>>>>
>>>> On 09.09.19 17:07, Bret Jordan wrote:
>>>>> So if you are building a streaming collector that gathers data from
>>>>> other sensors, you have control over how that data gets added /
>>>>> stored on that intermediary streaming server. ÂSince you may need
>>>>> to paginate the data, you must be definition then need to
>>>>> temporarily store that data. ÂIf you need to store data, then you
>>>>> can define what ever data structures you want. ÂThus you can define
>>>>> the date added property with microsecond precision. ÂÂKeep in mind
>>>>> that the data added is NOT the date that the end / origin sensor
>>>>> used for its timestamp.
>>>>>
>>>>> I just do not see how this is broken or why it would not work.
>>>>> Please help meâ
>>>>>
>>>>> Bret
>>>