Re: [cti] Re: [EXT] Re: [cti] TAXII Pagination Example Text

[Andras Iklody <andras.iklody@circl.lu>]

This x1000

On 09.09.19 16:26, Jason Keirstead wrote:
> I agree with the problem;
> 
> The problem is rooted in the fact that assuming that a document has an
> "insertion time", is assuming the document lives as-is in a database.
> 
> This all goes back to the "STIX and TAXII are not a database" mantra.
> 
> -
> Jason Keirstead
> Chief Architect - IBM Security Threat Management
> www.ibm.com/security
> 
> "Would you like me to give you a formula for success? It's quite simple,
> really. Double your rate of failure."
> 
> - Thomas J. Watson
> 
> 
> 
> From:        Andras Iklody <andras.iklody@circl.lu>
> To:        Bret Jordan <Bret_Jordan@symantec.com>, Wesley Brown
> <wbrown@lookingglasscyber.com>, "drew.varner@ninefx.com"
> <drew.varner@ninefx.com>, Jason Keirstead <Jason.Keirstead@ca.ibm.com>
> Cc:        Allan Thomson <athomson@lookingglasscyber.com>,
> "cti@lists.oasis-open.org" <cti@lists.oasis-open.org>
> Date:        09/09/2019 11:10 AM
> Subject:        [EXTERNAL] Re: [cti] Re: [EXT] Re: [cti] TAXII
> Pagination Example Text
> Sent by:        <cti@lists.oasis-open.org>
> ------------------------------------------------------------------------
> 
> 
> 
> OK, my use-case is as follows.
> 
> I have a sensor that ingests large amounts of data (parsing network logs
> / netflow, passiveDNS, etc). This sensor stores the data in its own
> format, with the timestamp being accurate to the second. If I were to
> build an interface that responds to TAXII queries for the collector in
> front of the sensor, and I were to query this data, I'd be dealing with
> large data-sets and I'd want to paginate it.
> 
> Unless I set a limit that will probably blow through my memory
> limitations, I have no other TAXII-compliant way to paginate the data in
> sane chunks without either losing some data (anything beyond the memory
> limit's envelope for a given second) or without blowing through my
> memory limits.
> 
> Best regards,
> Andras
> 
> On 09.09.19 16:00, Bret Jordan wrote:
>> Andras,
>> 
>> Thanks for the question.   TAXII should work well for this use case.  I
>> do not see why it would not.   Please keep in mind that the limits we
>> were talking about are optional.  So a server / sensor may have no limit
>> which lets you pull all records at once.
>> 
>> The sensor can dynamically add / figure out the date-added values how
>> ever it needs to do so.  So I am not sure why this would not work. Can
>> you help me understand why you think it will not work?  Or does this
>> solve your concerns?
> 
> ---------------------------------------------------------------------
> To unsubscribe from this mail list, you must leave the OASIS TC that
> generates this mail.  Follow this link to all your TCs in OASIS at:
> https://www.oasis-open.org/apps/org/workgroup/portal/my_workgroups.php ;
> 
> 
> 
> 
>