Re: [cti-taxii] Re: [EXT] Re: [cti-taxii] New Work - Pagination

["Jason Keirstead" <Jason.Keirstead@ca.ibm.com>]

The problems with date based pagination
are well documented from the TAXII 1 days. In addition to the issue below,
you have 

* Time synchronization. Quite often
the two systems won't have their clocks synced, nor any way to do so since
one is owned by a third party. 

* You also then have them running in
different time zones and have their hardware clocks set improperly instead
of just running them on UTC. 

* Finally you have the problem where
you are unsure what date range you should pass, so that you get an amount
of records you can handle, since on 1 hour of data might contain 100 records,
and the next hour might contain 1,000,000

Those reasons (and some others) are
actually what led us to adding item-based pagination as an option.
-
Jason Keirstead
Lead Architect - IBM.Security
www.ibm.com/security

"Things may come to those who wait, but only the things left by those
who hustle." - Unknown 




From:      
 Bret Jordan <Bret_Jordan@symantec.com>
To:      
 Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Cc:      
 "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Date:      
 09/17/2018 01:50 PM
Subject:    
   [cti-taxii]
Re: [EXT] Re: [cti-taxii] New Work - Pagination
Sent by:    
   <cti-taxii@lists.oasis-open.org>

---

I am not suggesting that we change the
filtering by added_after at all.  I personally believe you are correct
in thinking that one could paginate by added_after in most circumstances.
 I also think an added_before would be a nice addition.  

The only problem I have heard about pagination
in regards to using a date added value is you could run into a problem
if large amounts of records were added in a single transaction and that
transaction used the same "date added" for every record in the
transaction. 

One way some systems might address this
is to use microsecond precision for the date added value and then try and
ensure each record gets its own timestamp or that no more than X records
use the same microsecond timestamp.  This should give enough room,
even under heavy load.  But I am sure as soon as I say this, people
will have an example where that will even fall over. 

Bret

---

From: Jason Keirstead <Jason.Keirstead@ca.ibm.com>
Sent: Monday, September 17, 2018 10:29:52 AM
To: Bret Jordan
Cc: cti-taxii@lists.oasis-open.org
Subject: [EXT] Re: [cti-taxii] New Work - Pagination
 
Can we outline the problem sets / challenges
with item-based paging, so everyone is on the same page WRT the discussion?

As well, currently  you can already "page by date" in TAXII
(like you could in TAXII 1) - so in effect we have two mechanisms already.
Are we talking about changing that as well?

-
Jason Keirstead
Lead Architect - IBM.Security
www.ibm.com/security

"Things may come to those who wait, but only the things left by those
who hustle." - Unknown 




From:        Bret
Jordan <Bret_Jordan@symantec.com>
To:        "cti-taxii@lists.oasis-open.org"
<cti-taxii@lists.oasis-open.org>
Date:        09/17/2018
01:11 PM
Subject:        [cti-taxii]
New Work - Pagination
Sent by:        <cti-taxii@lists.oasis-open.org>

---

All,

Now that TAXII 2.1 Working Draft 03 is up for Committee Specification Draft
01 ballot, I would like to start requesting proposals for how we should
address and solve pagination for TAXII 2.1.  As you know, we tried
to use HTTP's native "Item" based pagination in TAXII 2.0, but
after doing additional research and trying to implement this, we discovered
that not only is this a known "do not do for RESTful APIs" but
it is also near impossible to implement in code given the STIX dataset.

I know several of you have expressed ideas for how this could be done/fixed
in TAXII 2.1, so I am asking as one of your sub-committee co-chairs to
please write up your ideas and submit them to the TAXII email list as proposals.

Thanks
Bret