Re: [EXT] [cti] [EXT] Re: [cti] TAXII Pagination Example Text

[Bret Jordan <Bret_Jordan@symantec.com>]

So if you are building a streaming collector that gathers data from other sensors, you have control over how that data gets added / stored on that intermediary streaming server.  Since you may need to paginate the data, you must be definition then need to temporarily store that data.  If you need to store data, then you can define what ever data structures you want.  Thus you can define the date added property with microsecond precision.   Keep in mind that the data added is NOT the date that the end / origin sensor used for its timestamp. 

I just do not see how this is broken or why it would not work. Please help meâ

Bret

> On Sep 9, 2019, at 4:58 PM, Andras Iklody <andras.iklody@circl.lu> wrote:
> 
> No, we're building a streaming collector for sensors that passes the
> data on.
> 
> Best regards,
> Andras
> 
> On 09.09.19 16:57, Bret Jordan wrote:
>> But you are going to modify the source code of all of these devices that you do not own to provide a TAXII interface?
>> 
>> Bret
>> 
>> 
>>> On Sep 9, 2019, at 4:55 PM, Andras Iklody <andras.iklody@circl.lu> wrote:
>>> 
>>> The thing is, we don't control the source code of all sensors that we
>>> might want to use and going on a crusade to convince all vendors / open
>>> source projects / abandoned sensor software devs to change it is not one
>>> I personally want to take on ;)
>>> 
>>> Best regards,
>>> Andras
>>> 
>>> On 09.09.19 16:51, Bret Jordan wrote:
>>>> Andras,
>>>> 
>>>> So I think the easy solution would be to have your sensor store the content it gets in microsecond precision.  If you can modify the sensor to add TAXII functionality then you should be able to modify the sensor to add microsecond precision, no?
>>>> 
>>>> Bret
>>>> 
>>>>> On Sep 9, 2019, at 4:09 PM, Andras Iklody <andras.iklody@circl.lu> wrote:
>>>>> 
>>>>> OK, my use-case is as follows.
>>>>> 
>>>>> I have a sensor that ingests large amounts of data (parsing network logs
>>>>> / netflow, passiveDNS, etc). This sensor stores the data in its own
>>>>> format, with the timestamp being accurate to the second. If I were to
>>>>> build an interface that responds to TAXII queries for the collector in
>>>>> front of the sensor, and I were to query this data, I'd be dealing with
>>>>> large data-sets and I'd want to paginate it.
>>>>> 
>>>>> Unless I set a limit that will probably blow through my memory
>>>>> limitations, I have no other TAXII-compliant way to paginate the data in
>>>>> sane chunks without either losing some data (anything beyond the memory
>>>>> limit's envelope for a given second) or without blowing through my
>>>>> memory limits.
>>>>> 
>>>>> Best regards,
>>>>> Andras
>>>>> 
>>>>> On 09.09.19 16:00, Bret Jordan wrote:
>>>>>> Andras,
>>>>>> 
>>>>>> Thanks for the question.   TAXII should work well for this use case.  I
>>>>>> do not see why it would not.   Please keep in mind that the limits we
>>>>>> were talking about are optional.  So a server / sensor may have no limit
>>>>>> which lets you pull all records at once.
>>>>>> 
>>>>>> The sensor can dynamically add / figure out the date-added values how
>>>>>> ever it needs to do so.  So I am not sure why this would not work. Can
>>>>>> you help me understand why you think it will not work?  Or does this
>>>>>> solve your concerns?
>>>>> 
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe from this mail list, you must leave the OASIS TC that 
>>>>> generates this mail.  Follow this link to all your TCs in OASIS at:
>>>>> https://clicktime.symantec.com/3FAG8TQ5AajqsonCufbi7pr7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php 
>>>>> 
>>>> 
>>> 
>>> ---------------------------------------------------------------------
>>> To unsubscribe from this mail list, you must leave the OASIS TC that 
>>> generates this mail.  Follow this link to all your TCs in OASIS at:
>>> https://clicktime.symantec.com/3NzFaS8xJiyBmH9i3guvfjb7Vc?u=https%3A%2F%2Fwww.oasis-open.org%2Fapps%2Forg%2Fworkgroup%2Fportal%2Fmy_workgroups.php 
>>> 
>>