|
You keep talking about systems that emit or use second precision data. The TAXII date_added value is not related to the data or content. It is not related to the STIX data at all and it should not be kept. It is an arbitrary value that does not need to be
kept. It is purely a tracking point for the TAXII server itself. Two TAXII servers that have the same object will more than likely not have the same value for the date added.
So said another way, if I sync content from you, I will not keep the date added values from your server. I will assign my own, because this meta data is tied to the TAXII server itself. An organize may choose to keep it, but it does not need to. It
should be considered a throw away value.
So just like you have an auto incrementing index value, you would have an auto incrementing date value. So in your solution you could just store the date in a Unix time stamp format with microsecond precision. This auto incrementing index could just be
translated back and fort has between RFC3339 format.
Bret
Sent from my Commodore 128D
PGP
Fingerprint: 63B4 FC53 680A 6B7D 1447 F2C0 74F8 ACAE 7415 0050
Hi Bret,
So, if this is the recommended hack to use for TAXII, this advice could
perhaps be in the final text to avoid other implementers falling into
the trap of not being able to fetch subsets of the data as in the
situation I've explained before.
So to reiterate, date_added should be used as the iterator for
pagination, use consecutively incremented micro precision for systems
that emit second precision data.
As for writing a proposal for changes, I think based on our previous
experiences it's a route that we're not willing to go down - fighting
for absolutely minor changes for years is out of scope for small open
source teams. We're just looking for guidance how to implement what has
been agreed upon by the TC at this point.
Best regards,
Andras
On 09.09.19 17:46, Bret Jordan wrote:
As I stated in a previous email, you should be able to easily do a date_added+.000001 for each record. You are going to have to store these records on your system, and thus you will need some basic meta-data to store them anyway.
You probably are going to store which device you got them from, and a bunch of other things. A simple timestamp should not be a significant challenge.
If you want to write up a proposal for TAXII 2.2, I am sure others, including me, would love to see it. Please just make sure your proposal does not break existing use-cases.
Thanks
Bret
On Sep 9, 2019, at 5:14 PM, Andras Iklody <andras.iklody@circl.lu> wrote:
The data is collected in bulk from the other sources. We could indeed
attach a timestamp to the bulk collection, but that would be the same
for all data collected in one shot exceeding any sane pagination limits.
I am somewhat confused that we're back at the data storage question
though, TAXII/STIX should NOT have inherent needs when it comes to how
data is stored as from what I understood it's purely a
transport/exchange format / protocol. If TAXII basically requires us to
move all the stored data to true micro second precision (instead of
padding it on fetch for lower precision tools), then that will kill a
lot of interactions with other tools out there.
Best regards,
Andras
On 09.09.19 17:07, Bret Jordan wrote:
So if you are building a streaming collector that gathers data from other sensors, you have control over how that data gets added / stored on that intermediary streaming server. Since you may need to paginate the data, you must
be definition then need to temporarily store that data. If you need to store data, then you can define what ever data structures you want. Thus you can define the date added property with microsecond precision. Keep in mind that the data added is NOT the
date that the end / origin sensor used for its timestamp.
I just do not see how this is broken or why it would not work. Please help meâ
Bret
|